A
New Data Architecture Manifesto
I propose in this
manifesto the review of data architecture in companies, based on a simpler and
more practical focus, which allows their adopters to make full use of their
data in the corporate environment, as an instrument of excellence and
competitiveness, which today is not achieved within the scope of traditional
methodologies.
This is the first
manifesto, which will be followed by a descriptive document of the proposed
methodology, describing the steps to be followed and the recommendations for
deployment in companies of all sizes.
The
Problem
Modern data
architecture today suffers from several technical and social conditions and
limitations, including:
o
Most
organizations make use of heterogeneous platforms, where information is distributed
without proper control, leading to frequent redundancies (and inconsistencies)
of content, the greater consequence of which is the increase in data management
expenses in corporations;
o
The
lack of consensus on what data is of real interest to organizations, leading to
different models and interpretations of corporate data; information towers are
an immediate consequence of this lack of consensus, generating parallel control
structures and thus weakening efforts to unify information at the business level.
o
The
new technologies of management and manipulation of data applied without the
proper criteria in organizations, leading to underutilization of them, plus
increasing operational costs.
o
The
application of agile methodologies in companies without proper preparation for
this, leading to increased cases of error in organizations, often even
compromising their reputation, due to the excessive number of errors exposed to
the market, sacrificing the reliability of the business.
o Aa false belief that IT and data are operational expenses, rather than considering
them as strategic resources, hindering the adoption of good technology
management practices and consequently increasing the spending ceilings of
companies with it. Only manufacturers benefit from this misinterpretation,
because they often offer the market oversized products, making large profits
from this practice.
o
As a consequence of the previous factor, the importance given to data architecture
in organizations is minimal, since managers cannot see its strategic importance
– "data is the new oil" – say some, but few know how to turn this
"oil" into energy.
o
Finally,
an internal factor to IT – professionals in the area usually give little value
to architecture because they understand that only the technical factor can
solve the various issues of information management, making technical
disciplines, such as analysis, design and development, prevail, to the detriment of structural planning – architecture. Due to the pressing need for
speed, architecture is deprecated as a problem-solving tool, although it is the
biggest acceleration factor in structured environments – with architecture
agility is encouraged, leading to results in less time at lower costs and
higher quality of the result. It is not enough just to add more resources to
the environment because these are low-cost commodities – management
and control are needed for consistent solutions at consistent costs for
organizations.
A
new framework of needs
Although there
was never so much need for information by organizations – which are now highly
interactive global mechanisms, producing previously unimaginable volumes of
data – the way this collection is controlled still refers to the beginnings of
information technology, from the second half of the twentieth century.
Information is now an instrument, that the faster it is obtained, the greater
the benefit to the business strategy. The market, although it understood the concept still succumbs to the lack of means to have information with the
necessary speed.
In short, we do
not yet have in the market effective methods of producing information within
the specific needs of a company with the necessary agility, even with new
methods and tools – there is a lack of consensus on what is fundamental, where
it is located and how we can use the information.
We need
strategies and methods to attack the avalanche of information generated inside
and outside companies in an extremely practical and objective way, aiming at
the result in a minimum of time, without, however, leaving aside the principles
and rules of organizations. A new mentality is necessary, which can leave aside
old concepts and adopt objectivity and consistency as the main metrics.
A new data
architecture must arise from these needs that can meet the new world of
the highest volumes of data and the need for rapid decision making.
A new model of data architecture
This new
architecture must be conceived from a mindset of obtaining results, considering
the best of what has existed to date, characterized by:
o
The
importance of consistency – organizations need a consensus in their information
models so that the data is always seen in the same way and ensure its
reliability;
o
Maintaining
heterogeneity – data can be kept in heterogeneous environments if it follows a
model of consistency and interoperability so that the data is always the most
reliable and has the lowest acquisition and processing costs.
o
Importance
of integration – each new system and technology needs to be inserted in the
context of a single model of information, even if it has its own structure and
formats; integration is essential to make the flow of data and operations as
consistent as possible.
o
Adoption
of standards – even in heterogeneous environments, information needs a minimum
of consistency, whether in format, integrity rules, or even nomenclature (when
possible); standards consistency is a key cost reduction factor in the new
economy to maintain competitiveness.
o
Adaptability
– The use of models and standards for corporate information is a guarantee that
data can be used and/or migrated on any chosen platform to ensure zero-impact
technology migration. Formal standards can be replicated in any technology
without any impact on organizations.
o
The
leveling of information - there will always be three levels of information in
organizations:
o
Operational
- the information at its most detailed level (granular);
o
Tactical
– the consolidation of operational information at the intermediate (managerial)
level, providing analytical views of the areas;
o
Strategic
– information at its highest level of consolidation, aimed at senior management
for strategic decision-making
To have the three
levels in perfect harmony, consistent visions and models are necessary, as well
as minimum standards to be followed; the structuring of information in levels
is a major factor in obtaining data quickly for all areas of the organization
and consequently achieving greater agility in the execution of business
operations.
With these few
principles we can build highly effective architectures and enable the creation
of data solutions in record time.
What
we should keep
The importance of
maintaining a company vision – data is meaningless if it does not meet a
purpose – and in this case, we have the effectiveness and continuity of the
business.
Standards –
regardless of structure and format, companies need to value the consistency and the final quality of the information obtained, from the unified vision and quality
standards. Here the principle of governance is established, where the data
maintains a minimum set of conditions to be used and these requirements are
properly controlled within the functions to maintain their usability and
relevance.
An architectural
vision – the model and standards are created from the business and solution
views present today in existing architectural frameworks – only with a significantly higher degree of speed than current molds, whether by the
adoption of new practices, or from new management and development tools.
Planning and
application of integrations – any and all applications in the company need to
comply with the architecture models and data standards and for this the
applications in use in the company need to be integrated with each other, using
effective tools and techniques, through the integration design.
The
new and the old tooling
In the proposal
of a new architecture, the tools are the crucial element - for control,
validation and development of solutions.
Directories and
data catalogs – control the collections of information and products created
from them, such as reports, integration services, and data visualization.
Catalogs start from this new vision to be the main interface between corporate
users, IT areas, and management teams. They must be able to support any and all
data format relevant to companies, as well as provide communication between
applications, as well as additional capabilities such as validation, automatic
code creation and auditing; finally we have artificial intelligence as a new
element of aid in information management with various resources to be added.
New data catalog
programs must also be extended using application-friendly and
application-intensive libraries and API (Application Program Interfaces).
Focus
on organization and productivity, before agility
While agile
methodologies are recognized as effective in troubleshooting when we refer to
the use and processing of data we should only consider these methodologies if
there is the necessary organization that precedes them – applying agile methods
by itself does not solve anything if there is no architectural effort to
support them.
The proposal for
a new data architecture model is based on faster results without the loss of
consistency - pure and simple speed is not a guarantee of quality, but if
preceded by a solid architecture model it can yield up to three times more than
in unorganized environments structured by architectural effort.
The new data
architecture is not 100% formal and is based on speed, quality and consistency to
solve current and future information management problems.
Porto Alegre,
November 28, 2020.
Marco Aurélio
Cavalcante Ribeiro, Business Data Architect and MBA in Administration, Finance
and Value Generation, class of June 2019.
Comments
Post a Comment