Data-centric project requirements?

Several times last week, in different circumstances, I was asked a question containing these three words or their synonyms. That’s not new. It happened previously. But this concentration triggered the write-up that follows. Nothing original and neither is the reason to write it:

Everything that needs to be said has already been said. But since no one was listening, everything must be said again.

— André Gide

Let’s first clarify what is data-centric and then see why it doesn’t go well with project and even less so with requirements.

 

What is data-centric?

The short answer is in these three1Check out all five principles data-centric principles. principles:

  1. Data is self-describing and does not rely on an application for interpretation and meaning.
  2. Data is expressed in open, non-proprietary formats.
  3. Applications are allowed to visit the data, perform their magic and express the results of their process back into the data layer.

I find these three are the most important data-centric principles, but another reason for selecting them is that they are context-independent. The data-centric manifesto they are taken from is currently with an enterprise-only focus2 After the publication of this article, the scope of the manifesto was adapted, and now it includes the other two scales. . Yes, the problem they address is most severely felt – or rather not felt because of ignoring or misattributing the symptoms – in large organizations. Yet, what is behind these principles is equally important for personal information management and on the open web. Let’s go quickly through all three levels then, from big to small, and see what data-centric means for the world wide web, for corporate IT, and then for personal information management.

The web was designed to be a decentralized system where the agreement on a few standards, basically HTTP and HTML, enabled free choice on just about anything else. People were finally free to express themselves and to choose from where and how to get information. They got free to innovate on building new browsers, websites, and whatever web applications and services they can think of. A system like this, with a self-maintained organization, can work well and have a natural tendency for virtuous cycles. In other words, it can amplify goodness and develop its own immune system for whatever threatens its viability. All it needs is to have the right kind of enabling constraints, for example, the standards I mentioned above, and to allow autonomy of all subsystems. This is the balance between autonomy and cohesion. It works for animals, people, tribes, organizations, society, and a socio-technical system like the web.

So the web flourished as a decentralized system, where people were free to choose and create more choices. And then one day the platforms appeared. They offered good and free services. Or at least they looked good and free at first. In reality, they were (and are) neither good nor free. The platforms are not nearly as good information providers as it was the decentralized web before them. What we see is not what we are looking for, but what their algorithms decide to show us. And the services of these platforms are not free. Quite the contrary. We pay with our data, and we pay twice. Once by being their content providers and a second time by giving them our personal data. Importantly, we don’t give them only our current personal data but also future ones, by allowing them to track our online behaviour. Who’s them? I’m talking of course about IT giants like Google, but the best example of extreme centralization and lock-in is Facebook3This has many facets. Facebook can be looked at as a very successful aggregator or as a prime example of a new form of capitalism.. In this way, the web, a decentralized system, shaped by the users, turned into a hyper-centralized system, shaped by a few powerful corporations4The centralization of the web is not only about the content but also about the infrastructure. The convenience of the cloud increased the dependency of both individual users and companies on the strategy and fate of a few powerful providers, namely Amazon, Microsoft, and Google.. It also formed users’ expectations. In 2019 Facebook and Google announced that it was now possible to copy images from Facebook to Google Photos. That’s the new norm for innovation. Only a few people noted the absurdity. As Ruben Verbourgh pointed out, 50 years after being able to send video signals over a distance of 380,000km, we celebrate that we can finally move a photo by 11km (the distance between Facebook and Google headquarters). A bit dystopian, isn’t it?

Yet, the problems with this centralization are not widely understood. For example, very few people realize how platform-based political propaganda works, and that’s why it works so well. Even fewer relate it to the hyper-centralization of the web. Same with fake news and so on. Maybe the least understood of the damages is how it suffocates innovation. It’s easy to illustrate. Even when you use Google for product search, where it should excel after so many years of work, huge investments, massive feedback, and the use of language models with trillion parameters, it’s really lame. Try searching for a bike below a certain price and certain weight. You’ll get results for bikes above that, but okay, then you can fix that using the shopping filter. Currently, that will not allow you to specify the weight even though it’s available in most technical specifications published online. But even if they add it at some point, the final selection will still exclude the majority of the offerings by smaller companies. As a result, you can’t get an answer to this simple question.

A way out is to decouple data and applications.

This was for the web. Now for enterprises.

In enterprises, for decades the applications were built in a way that the data model is separate for each application, trapped inside it, and the interpretation of the data is in the application code. The applications themselves are built based on historical functional requirements. When some change needs to be done, coming from new business needs or changed legislation, it takes months, costs huge amounts of money, and leads to increased complexity and technical debt. The same when two or more application needs to be integrated. This is the application centric-way of building applications. It is dominant to this day. Most big enterprises have thousands of application silos. They try to integrate the data through data warehouses, data lakes, point-to-point interfaces and APIs. All these methods provide partial and temporary solutions and add to the technical debt.

A way to solve this is to decouple data from applications.

A lot more can be said about what data-centric means at the enterprise level. If you have a bit of time to learn about it, I’d recommend watching this video. If you have more time, it’s worth going through The Data-Centric Revolution and if you have even more, read it after its predecessor Software Wasteland to get a better understanding of the size and the nature of the problem. If you have spent more than a couple of years in a big organization, you’ll find many familiar patterns.

We have this problem not only on the open web and in enterprises. We also have it with our personal information management. Our emails are trapped in one application, our documents in another, and then we keep our bookmarks disconnected from them inside the browser. We use one application to search on the web, and another to search our files. Now we can combine it but only if we forfeit our freedom to choose operating system and browser. If we look for something we communicated in writing, we have to remember where we wrote or read it. If we don’t, we have to search our files, our email, Twitter, Facebook, and the web. When we write a Word document we have to open it with MS Word. But when we are in Word we don’t have access to our tasks. For that, we need to go to another application.

A way to solve this is to decouple data from applications.

At all these scales, societal, organizational and personal, when it comes to managing information, we have similar kinds of problems coming from the tight application-data coupling or platform-data coupling. I will focus on organizations from now on to the end of this article, but it’s important to keep the bigger picture in mind.

What does a digital transformation from application to data-centric enterprise look like? In a perfect world it would look something like this:

EKG stands for Enterprise Knowledge Graph. It is something that complies with these design and governance principles.

A more realistic, but still ambitious transformation, will keep the data of the current applications where it is but will have it duplicated (virtualised or streamed) in the enterprise knowledge graph where it will be living an independent life, together with its semantics.

However, all new applications should be built in a way that they “visit the data, perform their magic and express the results of their process back into the data layer”.

While data-centric is all about decoupling applications and data, by itself, as a goal, slogan and buzzword, is problematic. Data-centric has similar problems as the preceding waves of process-centric and service-centric movements. Even worse, the word itself can impede the transformation it promotes. When a data department in a big organization is promoting data-centricity inside the organization, that can be easily misconstrued as just being self-centric. If the idea is sold, there will be more budget for the data department. It’s like a planet faking strong gravity to attract matter so that it gets more mass and consequently stronger actual gravity.

What is actually needed is not data-centrism5It’s way better to put people in the center but wait, wasn’t that what brought the climate change. It seems that just like centralization, any kind of centrism does not solve problems –it just changed the nature of the problems we deal with., just decoupling data from applications. Even not that, loose-coupling would suffice. It’s also easy to explain. You want to run, but your leg is in plaster. Once the plaster is removed you can bend your knee and ankle, you can run, jump, you can walk in one direction and then abruptly change it. But no, loose-coupling is the language of SOA; it’s passé6When microservices became fashionable that was only possible because the original SOA ideas were forgotten and they were exactly about micro-services, only the SOA-in-practice was not.. Nobody would listen. Not that there are many that hear the data-centric cries, but at least they stand the chance to be echoed off the more modern knowledge graphs and F.A.I.R principles.

It took a bit of explanation to cover data-centric. But it was needed, so I can now try to explain why it doesn’t go well with project and even less so with requirements.

 

What’s wrong with project?

Application-centrism is deep in both software engineering, project management, and corporate culture. Take for example the traditional CIO title. Chief Information Officer, the officer responsible for ensuring good management of information. How many do you know that actually do that? They are rather chief application offers, or technology officers or infrastructure officers, anything around information but not the information itself.

Another example is every IT project. It is centred on a certain solution. It is about solving a particular problem, based on collected historical requirements. The application focus of most IT projects ignores space and time. Initially, space (the enterprise) was not represented in the project team at all. This brought Enterprise Architecture to the rescue. But it created its own world of artefacts, often equally far from both business and IT. Time (the future) beyond the project period is not represented as if it will never come. This brought first SOA and later DevOps to the rescue. But DevOps were quickly put in the cloud trenches. Yet, the best that DevOps can do is ensure a smooth deployment, operations, and scalability. What they can’t do is change the core application architecture in response to changes in needs or legislation. For that, a new project will be needed. And it will bring the same kind of problems. IT Projects tend to create local optima and short-term solutions.

If you ask about “data-centric project requirements”, it assumes that data-centrism can be achieved within a project. Well, since it tries to deal with the mess created by project thinking, would it be really successful if managed in the same way? “We can’t solve problems by using the same kind of thinking we used when we created them”7This quote is attributed to Einstein although he never said it, but do we always need a famous person, any authority for that matter, to make something we like and believe in, weigh more?. Not that projects are a bad way of managing the creation of something. But as it was borrowed from the construction business, it came with the full package. Now we talk about “building software”. This is not an innocent metaphor. Even if we imagine that it is and we take it here, a project might be a good way of organizing the building of a house but wouldn’t do for a city. And the horizon of data-centric, literally and metaphorically, in other words in space and time, is way too big to put into a single project. It needs something more strategic, a whole rethinking of the way IT is managed and governed.

 

What’s wrong with requirements?

Requirements are central in software engineering. There are well-established methodologies for elicitating, documenting, and tracking project requirements. There are also methodologies for estimating efforts based on requirements. Requirements allow for a split of responsibilities, and to get acceptance criteria. A whole profession, business analyst, was born and relies on the market demand for requirements. When it was realised that it’s not easy to fix things at the beginning of the project – people need to learn what they really want and what is feasible – some attempts appeared to allow flexibility. First, it was UP, then RUP, the version of Rational (later IBM), and that later evolved into DAD (Disciplined Agile Delivery, which I assumed long dead but just learned is still practised today). Of course, the Agile movement tried to remove the word requirement and replace it with much friendlier8 It’s friendlier in the way it sounds, in the way it’s managed and in the fact that focuses on the experience of the user and not on the functionality of the tool. user story.  And yet both traditional and Agile software engineering focuses on building applications, the features of which correspond to requirements or user stories.

Requirements are viral. They reach even outside software engineering, in disciplines trying to bring coherence to organizations. Take TOGAF, the most popular Enterprise Architecture framework. What you see in the center of its methodology is “Requirement management”.

But it is due to requirement engineering, among other things, that we ended up with silos. Requirements, and especially functional requirements, determine the application boundaries. The fragmentation is not logical, only historical. The way database designers decided what tables to create with what columns and how to associate them is corresponding, if the project was successful, to what was required (and how it was interpreted by the analysts and the person in charge of the database design). And all concrete requirements – mainly functional but even some non-functional – share the same characteristic:

They are all historical.

Requirements are about what is known, from the history of the business case, and by the people involved, by the time they are captured. You have a set of requirements, and you meet them by building an application. Then you have another set, and you build another application. Then it turned out they need to be integrated. So you capture the integration requirements and build an interface. Which is, by the way, yet another application. Then they become many, so you do data warehouse, data lake, virtualization layer or some fancier data integration architecture which in the end is just another silo.

All this is what data-centrism tries to deal with and avoid in the future. Corporate data architecture should not be shaped by known requirements9This doesn’t mean that functional requirements and user stories do not have utility. In a world where data and applications are decoupled, they will not do any harm.. It should be designed for unknown requirements. In the end, if some requirements for a data-centric architecture are needed, these are only non-functional requirements, the -ilities, like interoperability and scalability. But most of them are not project-specific and are already in the principles, like F.A.I.R, EKGF principles, and, of course, the data-centric principles.

 

Related posts

Linked Data uptake

Wikipedia “Knows” more than it “Tells”

SASSY Architecture

Buckets and Balls

 

  • 1
  • 2
    After the publication of this article, the scope of the manifesto was adapted, and now it includes the other two scales.
  • 3
    This has many facets. Facebook can be looked at as a very successful aggregator or as a prime example of a new form of capitalism.
  • 4
    The centralization of the web is not only about the content but also about the infrastructure. The convenience of the cloud increased the dependency of both individual users and companies on the strategy and fate of a few powerful providers, namely Amazon, Microsoft, and Google.
  • 5
    It’s way better to put people in the center but wait, wasn’t that what brought the climate change. It seems that just like centralization, any kind of centrism does not solve problems –it just changed the nature of the problems we deal with.
  • 6
    When microservices became fashionable that was only possible because the original SOA ideas were forgotten and they were exactly about micro-services, only the SOA-in-practice was not.
  • 7
    This quote is attributed to Einstein although he never said it, but do we always need a famous person, any authority for that matter, to make something we like and believe in, weigh more?
  • 8
    It’s friendlier in the way it sounds, in the way it’s managed and in the fact that focuses on the experience of the user and not on the functionality of the tool.
  • 9
    This doesn’t mean that functional requirements and user stories do not have utility. In a world where data and applications are decoupled, they will not do any harm.

7 thoughts on “Data-centric project requirements?”

  1. I do agree to most of what is written above, except ‘Data is self-describing’. No data is a little part of information or a part within a given frame. It is also connected to the other parts in this frame.
    For me it looks like you sit in the IT (computer and software) bus. Come outside and have a look from a business process point of view.
    Any new process or project (an organised change to a process, about an idea of the future use of the process) needs technical aids (e.g.: heating, cooling, electronic gadgets, data storage). Therefore, when IT help is needed, the process describes why data must be collected, which data, how and within which frame. The usage of that data is a given. Then at the end we humans have to understand the result or reports they must be change into information again (we humans can only manage information or pictures with our brain).

    Reply

    • Thank you for your comment. By self-describing, I mean machine-understandable and I use both concepts with the established conventions. They shouldn’t be taken to imply that data has agency or that machines can understand. You wrote “is also connected to the other parts in this frame” and that is an example of self-describing as opposed to requiring a specific application to interpret it.

      This book is a good example that I don’t sit in the “IT bus”, while it’s true that I take that bus from time to time and I enjoy the ride. You can find quite a few posts in this blog criticizing the IT-centrism of the mainstream Business Proces Management and Enterprise Architecture.

      Regarding your last statement, that “we humans can only manage information or pictures with our brain”, I don’t think that is the case and the support of the claim is not coming from computer science but from cognitive science. Simply put, our body and environment contribute a lot to our cognition and our ability to manage information, and the brain alone cannot do all the mental work in most situations that are important for our living.

      Reply

    • “Data is self-describing and does not rely on an application for interpretation and meaning” is the second principles of the data-centric manifesto. You said you disagree but could not understand how the arguments you provided support this. They make sense by themselves but they don’t seem to be supporting the application-centrism and opposing decoupling.

      Reply

  2. Dear Ivo,

    Thank you for sharing your insights into the data centric principles. I have learned to look at it from another perspective.
    I would like to offer a thought as well that might make you rethink some of what you wrote in this articles.
    As you write, it’s about the principles with data centric, fair, … Enterprises need to translate those principles into concrete elements. How they are going to implement the principles. How to make the data loosely/de- coupled with the application. From then on, new application are required to work in a certain way.. We build those apps in projects and now you here me coming.. aren’t those requirements then project requirements to make sure the result is data centric? In shorthand those would then become Data centric project requirement?

    Reply

  3. Interesting,

    I tend to agree strongly with data centricism, and welcome any voice calling that out, particularly the piece about culturally projects are and orgs are application centric. There is of course the rebranding of application as ‘product’, which in theory should also include the data.

    I disagree with pretty much everything else you say, as you can apply data centricism to those things as well, whereas your article seems to suggest you can’t or shouldn’t. If you are saying people don’t do that I agree, but that’s not the same thing.

    For instance as an analyst I mostly work on gathering qualitative and quantitative data , so that I can generate ‘requirements’ when they are needed. The data is indeed historical, but if ‘historical’ is a criticism then surely that applies to data centricity? Largely orgs and customers dont change that much and the ability of LLMs to effectively generate requirements for many enterprise activities (from data) would tend to support that.

    To achieve data centricism, where does the data arise from? I naively assume it’s from applications via people or machines. Even if it’s sourced externally there will be ‘applications’ that need to be built and orchestrated. How do these arise?

    How does an enterprise organise itself to become data centric without some kind of project? You can assign a team to it, but that team will organise themselves into something that smells like a project even if it has a different ‘brand’

    I would suggest that , yes data centric, but to be data centric you need to organise and orchestrate people, initiatives and applications. And actually proposing and encouraging data centric approaches to empower those activities would perhaps be truer to your agenda?

    Reply

    • Thank you for your comment. I’ve just published an essay which wasn’t meant to clarify the points of contention, but might have accidentally done so. At any rate it is another attempt to describe the organisational pathologies brought by application-centric mindset where projects and requirements, which are the focus of this post, have a prominent role.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.