Data

Handle more data without a growing cloud bill

The challenge

Every organisation wants to do more with data. Sensor data, transactions, logistics records: volumes grow every year. The cloud looks like the answer, because extra capacity is always a click away. In practice, the bill often grows faster than the value. Data pipelines, the automated steps that collect, convert and store data, often run on oversized machines. The code uses more compute than it needs and nobody knows what a single run actually costs. On top of that, every new data source arrives in its own format, which means another development cycle.

Adding capacity won’t fix that. With software optimisation of your data pipelines, you process the same volume of data, and more, on a fraction of the infrastructure.

The features

Software optimisation focuses on 3 areas:

The application

The approach follows three steps:

Measure

We map what your pipelines actually consume and translate that into cost per run. That shows where the waste is: oversized machines, generous settings or code that asks for more than it needs.

Daily cloud costs of one pipeline: €470 before and €7 after optimisation, €169,000 saved per year
A large server cluster compared with one server processing 56 billion data records a day

Redesign

Based on those measurements, we rewrite the parts of the pipeline that demand the most computing power. The rules for what happens to the data then sit in one fixed, readable place (the configuration) instead of being scattered through the code. You can always trace how a result came about, and the same data always produces the same result.

Safeguard

We test every improvement with realistic data volumes and measure again, so you know the savings last. From then on, you connect a new data source by changing the configuration, without a new software release.

Machines, business systems, files and applications flow through one configuration-driven processing layer into data ready for analysis

The results

For a high-tech manufacturer, the data platform now processes 76 billion data records a day across six pipelines that run around the clock.

One of the most important pipelines used to cost €470 a day. After optimisation, it costs €7. The largest pipeline handles 56 billion records a day on a single cloud server with 6 cores and 10 GB of memory, without a separate server cluster.

New data sources go live with a configuration change. That saves a full build, test and deploy cycle every time.

Read the full case: Data pipeline optimization in high-tech

Impact:

Software optimisation makes data processing cheaper and more predictable, even as volumes keep growing.

98.5%

€169,000

56 bn