From Pipeline Sprawl to Platform Convergence: A Lakebase Intervention

When we first mapped the data architecture behind a cyber risk platform we inherited, the diagram looked more like a transit map than a tech stack.

Azure SQL on one end. Databricks, on the other hand. Between them sat Azure Data Lake Storage (ADLS) staging layers, Azure Data Factory (ADF) pipelines, a Synapse warehouse, and a SQL Data Mart, all stitched together to move data from where it was written to where it was read. Seven systems. Just so a risk advisor could see a score that was already hours old by the time it reached their screen.

The architecture worked. Technically. But every new requirement, whether real-time scoring or an Artificial Intelligence (AI) model on top, meant another integration, another pipeline, another failure point. We weren't building anymore. We were maintaining.

That's when we started asking, what if the transactional and analytical layers didn't need to be separate worlds at all?

The Industry Problem

This wasn't unique to one client. Across engagements, we kept walking into the same architecture: an Online Transaction Processing (OLTP) database handling application reads and writes, a lakehouse running analytics, and a tangle of Extract, Transform, Load (ETL) pipelines holding the two together. Each layer carried its own cost, its own governance model, and its own way of breaking quietly at 2 A.M., which meant teams spent more time maintaining pipelines than building anything meaningful on top of them.

We weren't the only ones noticing. Virtue Market Research found that 74% of Global 2000 enterprises have already begun piloting lakehouse architectures in 2025, a 15% jump from the previous year, because the cost of keeping fragmented stacks alive had outgrown the cost of replacing them.1

Our cyber risk platform became the place where we stopped patching the old model and committed to replacing it.

The Intervention: A Different Way to Build

We didn’t start by looking for a new product. We started by questioning the separation we had accepted for years.

In that cyber risk platform, the disconnect showed up everywhere. An advisor would enter client details into an application backed by Azure SQL, while the data needed to calculate risk, external feeds, modeling outputs, and benchmarks lived elsewhere and were moved in batches.

Databricks processed it. Azure Data Lake Storage (ADLS) staged it. Synapse reshaped it. Azure Data Factory (ADF) pipelines stitched the flow together. By the time everything came full circle, the “real-time” score was already behind.

What bothered us wasn’t just the latency. It was the pattern. Every improvement required another layer. Every integration created a new point of failure. And governance sat in fragments across systems that didn’t speak the same language.

So, we stopped adding layers.

We replaced the application’s Online Transaction Processing layer with Databricks’ Lakebase and pulled the entire transactional workflow into the same environment as the analytics. That one move forced everything else to change. User inputs, workflow states, and configuration metadata no longer had to travel outward before becoming useful. They were already inside the platform, governed, traceable, and immediately available.

From there, we reworked how data moved. Instead of maintaining separate pipelines to push data from one system to another, we implemented a streaming pattern using Delta Live Tables (Lakeflow Declarative Pipelines). Every insert, update, or delete in Lakebase produced an event, and Lakeflow Declarative Pipelines ingested it directly into the lakehouse layers. The flow no longer needed orchestration across tools. It became continuous by design.

The cleanup that followed was almost inevitable. Synapse went away. The SQL Data Mart followed. The Azure Data Factory (ADF) layer, which existed only to shuttle data from point A to point B, became redundant. What remained was simpler to describe and far easier to operate.

Here’s what changed in practical terms:

The impact didn’t show up as a single headline metric. It showed up in how the system behaved. Fewer systems to monitor meant fewer silent failures. Data didn’t need to be reconciled across layers, so trust improved. Most importantly, when we started exploring Artificial Intelligence use cases on the same platform, we didn’t have to redesign anything. The foundation was already there.

That’s when it clicked for us. The value wasn’t just in consolidating tools. It came from collapsing the artificial boundary between how data is created and how it is used.

What This Means Going Forward

A year ago, if someone had told us we'd run transactional workloads and analytics inside the same governed platform, we would have called it ambitious at best. Having done it, the question that remains is whether this approach works. It's why we tolerated the alternative for so long.

The enterprises we work with are heading in the same direction, not because of a single product or vendor, but because the economics of maintaining parallel systems no longer make sense. When your data infrastructure requires three governance models just to serve one dashboard, something fundamental needs to change.

Lakebase gave us the mechanism. But the deeper shift was architectural, and frankly, creative. Rethinking how transactional and analytical data coexist demanded more than engineering. It required the kind of Business Creativity that questions inherited assumptions instead of optimizing around them.

If your team is still bridging that gap with pipelines and middleware, the architecture warrants a revisit. The goal was never to manage more systems better. It was to Outcreate the complexity that enterprises have accepted as the cost of doing business.

We've been through it. Happy to share what we learned.

References

1) Data Lakehouse Platforms Market | Size, Overview, Trends, and Forecast | 2025-2030, Virtue Market Research, February 2026, https://virtuemarketresearch.com/report/data-lakehouse-platforms-market