Organisations want to do something with AI. The data is usually there too, but nobody knows exactly what it means. On top of that, it is scattered across departments, network drives and Excel files. The data foundation that should sit underneath it is missing.
Every boardroom is talking about AI by now, and the pressure to show results is growing faster than the data infrastructure underneath. The problem has long since stopped being the models: Claude, Copilot and comparable tools are powerful enough for almost any question. The problem is that those models know very little about your organisation, because your critical data sits in all sorts of places and in different formats.
Waiting gets more expensive by the month. Organisations that put a data foundation in place now will be building AI applications a year from now that get further than a demo. Organisations that do not will keep running pilots that never reach production, while competitors who do take the step move ahead in the meantime.
The difference is not ambition or budget for an AI tool, everyone has that by now, but the foundation that tool has to run on. And once you have that foundation, you do not have to make it ‘AI-ready’ separately. AI runs on the same data you are already getting in order; it is an extension, not a new project.
At LINKIT, our data engineers build that foundation: at a high-tech company where thousands of tables are added every day, at a dairy producer that had to move from loose sensor dumps to a central environment, and at clients who wanted to renew their data platform with Databricks or want to use RAG to make their data easier to search for an LLM. Those projects add up to an approach in four steps. In this first article we take you through steps one and two.
In this article we answer three questions:
- Which question you need to sharpen before you start on a data platform
- How to bring scattered sources together in a data lake or lakehouse
- Why governance and ownership decide whether that platform delivers anything
Before you start: which question do you need to sharpen?
The most common request is not a technical one: an organisation wants AI and has the data for it, but does not know what intelligence can be pulled out of it. “The data is simply available, but very little is done with it”, as Ranith Ranawaka, lead data engineer at LINKIT, sums it up.
At a manufacturing organisation, that came to the table in concrete form. If we put X amount of raw material into the machine, Y comes out, and we want Y to go up. A logical wish, but far too coarse for a data project. The machine could be faulty. The staff might be insufficiently trained. There are dozens of possible causes underneath that single question.
That is why we start with an analysis: which question needs to be answered later, and by whom? What comes out of that is sometimes not AI at all, but a business intelligence dashboard or a forecast, something that was already possible before AI.
In practice that means talking to the people closest to the problem and asking questions that go beyond the first, coarse wish. Where does a process get stuck week after week, which information do you get too late or not at all, and which assumption about the organisation has not been tested for years?
Questions like that cut the problem down to something bounded: a use case with a measurable outcome that shows results within weeks to months. That choice has to be made up front, not after the foundation is already in place. Otherwise you run the risk of building a complete data foundation for data you turn out not to need.
Step 1: from scattered sources to one platform
In this step you bring all the data that is now scattered across machines, systems, spreadsheets and APIs together in one central place you can actually work with. Without that foundation, every following AI initiative stays an isolated experiment instead of something that delivers structural results.
Bringing all data together in a data lake
The place where everything comes together is usually a data lake: raw files, unchanged, on low-cost storage such as AWS S3 or Azure Blob Storage. You dump in what you have and transform nothing yet.
The difference with a data warehouse is the difference between a lake and a built-up area. A lake is a place where you can dump anything. A warehouse is structured, with fixed places for fixed types of data. If you want to store something in a warehouse, you first have to reshape it into the format that fits.
Warehouses date back to the sixties and seventies, to the era of Oracle and IBM Netezza, and almost nobody chooses them now because they are more rigid and often more expensive. What took their place is the lakehouse: a simpler lake where you can still store everything, but which you can also query with SQL. You can use open-source variants such as Delta Lake and Iceberg for this, but commercial platforms like Databricks and Snowflake also offer lakehouses.
The right platform depends on who you have in house
It is not the size of the company that determines the platform choice, but its technical maturity.
If you have engineers in house with domain knowledge, Databricks or Snowflake are a good choice: they require maintenance and expertise, but they give you control and room to expand. If that expertise is missing, a managed service such as Microsoft Fabric makes more sense, because it can be set up plug-and-play. The price is that you are locked into the framework: if you want something extra, you either pay more or build it alongside and bolt it on.
For an organisation with lower technical maturity we specifically do not recommend Databricks or Snowflake. The data is not very complex, the expectations are limited and there is nobody in house who can manage a platform like that. With Fabric, one or two people keep the pipelines up to date, and that matches what they want to know.
Transformation: a separate data pipeline for every source
To transform data you work with an ETL pipeline, which stands for extract, transform, load. For every type of source you build a separate pipeline that transforms the data, because every source arrives differently and has a different structure. Excel files are often structured in tables, but the data still has to be interpreted. External APIs, from government bodies for example, require authentication. Machines on the floor emit a lot of sensor data, which does not always use the same format.
Transforming along the way is less spectacular than the word suggests: it comes down to cleaning up, throwing away what you do not need and adding timestamps, so the data is readable and usable. After that it goes into the platform.
To give a sense of how complex data transformation can get: at a Dutch high-tech company it involves four to five hundred tables a day. One machine there has twelve hundred to two thousand sensors, some of which take twelve thousand measurements per second. A single SAP implementation adds fifteen hundred tables at once, which then all have to be linked together.
The bill is for compute, not for storage
When it comes to the cost model, there is often a surprise here for organisations. Platforms charge almost nothing for storing data. What they do charge for is compute: the processing power needed to actually run your pipelines. So the more inefficiency you have built into your data pipelines, the higher the costs.
That went wrong in an earlier migration from IBM Netezza to Snowflake. Because both run on SQL, a company dragged the existing code across in one move, including all the inefficiencies that were already in it. The bill doubled to tripled.
Read in this case how we cut costs by 98.5% for another high-tech company through more efficient data pipelines.
Batch or streaming: real-time is rarely necessary
Whether you process data through pipelines in batches or continuously depends on three things:
- How current the outcome has to be;
- How often new data becomes available from the source;
- What it is allowed to cost.
Many organisations report once a week or once a month; batch processing is enough for that. Machines that send data every two hours do not call for streaming either. If the business wants to see performance over the last five or ten minutes, the batches sit so close together that streaming becomes the more logical and efficient option. And then costs weigh in, because streaming demands more hardware and is harder to maintain. If it is not necessary, you do it in batches.
AI is an extension of your data foundation
For anyone who has to defend an AI budget, this is perhaps the most useful point. Almost every organisation wants to do something with AI by now, but gets stuck because its own data is not findable or usable enough for a language model. Claude Opus knows nothing about cheese production, so you want to send it the relevant pieces of your own data.
For this you use the same data that is already in your data lake or lakehouse, but you make sure through a RAG database that AI models can read it. That way you inject your own contextual data, as it were, without having to train the model yourself. When an AI model wants to use that data, it does so through a vector search: a search technique that looks for the answer based on the meaning or context behind the query.
“We do not have to tear down our data foundation to use it for AI. It is an extension of what we have already built.”
So anyone with a data foundation in place does not start over to make it ‘AI-ready’.
Step 2: people, governance and ownership
Data that sits in one place is not yet data you can rely on. Step 2 is about two conditions for actually being able to use the platform: who is allowed in, and whether what is in there is correct. Even then, the technology can be in perfect shape: if nobody knows who is responsible for the data, or who understands what the numbers mean, the platform delivers very little. So it is not only about tools, but about roles and ownership, because the blockages are more often in people than in technology.
Access is arranged in the platform
Not everyone is allowed to see all the data, and you arrange that in the platform itself. In Databricks, Unity Catalog does that as a management layer on top of the whole lake. For every request it looks at role and user type, down to column level: out of five columns one person may see three, while a colleague gets all five. In technical terms that is called row masking and column masking.
You run into this when you feed an AI model with internal information too. The rights someone has on an internal document do not automatically carry over when that data is used to feed an AI model. Someone who had no access to a particular document in the source system can still end up seeing it through the chatbot, unless you arrange that separately. In Databricks you solve that, for instance, by letting the same access rules that are already in Unity Catalog carry through into the vector search within the RAG database.
At the high-tech company it is tightly arranged. Every data path has an owner. Even the team that built the pipeline cannot get to production data without approval from that owner, via several layers.
Who owns the data?
According to Ranith the principle is simple: whoever creates the data owns it. That works until the moment several departments produce data together. Who owns it then? There is no ready-made answer; it differs per organisation. At the high-tech company it is assigned to the group lead, the person with an overview of all the machines or of the entire sales domain.
When assigning ownership it also matters that domain experts are involved. At the dairy producer, data came out of the machine that said nothing on its own. Sensor P001 gives values: 2.3, then 2.5, then 2.6. A thousand rows like that, without metadata and without explanation.
Building a data model on top of it required a domain expert, and there was none: one person knew part of the data, someone else knew another part, and that second person was on holiday. The project stood still until they were back. The people who were there were food technologists; SQL and Excel were what they knew. When we showed them what was possible with their own data, they were genuinely surprised that this existed.
What goes wrong is more often people than technology
The most stubborn blockages in data projects are not technical at all.
Patience and budget. Looking at data, designing a model, filling the platform and then getting insight out of it takes months and money. Decision-makers want answers faster and have no proof of the return, because the platform is not there yet. At the dairy producer we are now solving that with a small proof of concept in a Databricks notebook, costing a few thousand euros, paid out of a different budget. If it works, a demonstration goes to leadership and only then does real money become available.
The limits of manual work. Before there was a platform, someone would walk up to the machine with a laptop, collect an XML file there and sit down to look at it in Excel with two or three colleagues.
“You and I can check two hundred rows. Not two thousand or ten thousand. That is impossible, and we would make a lot of mistakes.”
That is exactly what a data foundation is for. Not to make AI possible, but to stop doing work that people cannot do.
Steps 3 and 4: next time
With a platform, monitored quality and access, and clear owners, the foundation is in place. What follows is working out the first use case: choosing something that delivers a demonstrable result within a few weeks to months. That result determines whether, and where, you invest next. How to work out that use case and scale up from there to advanced analytics and AI is what we cover in part 2 of this series.
Curious how solid the foundation under your data really is? Plan a no-obligation session with our data engineers about the first step in your IT modernization.
