a digital garden

Data Architectures

25 min read

A deep-dive into the eleven patterns shaping how we store, move, and serve data — and what nobody tells you about choosing between them.

There is a moment in every data team’s life when someone opens a whiteboard, draws a box labeled “Sources” on the left, another box labeled “Analytics” on the right, and connects them with an arrow. That arrow, innocent and thin, is where careers go to die. Because the distance between raw data and reliable insight is not a straight line — it is a landscape of trade-offs, organizational politics, and late-night debugging sessions involving malformed Parquet files.

This is a guide to that landscape. Not a glossary. Not a vendor comparison chart. A guide — the kind you’d get from someone who has migrated a data warehouse at 2 AM, watched a Kafka cluster eat itself during Black Friday, and had to explain to a VP why the dashboard numbers don’t match the spreadsheet they got from finance.

We are going to walk through eleven data architectures that define how modern organizations think about data in 2026. Some are old friends wearing new clothes. Some are genuinely new ideas. And a few are organizational philosophies disguised as technical patterns. The goal is not to tell you which one to pick. The goal is to make sure you understand what you’re signing up for when you do.

The Data Warehouse: Old Guard, Still Standing

Let’s start with the architecture that refuses to die, and for good reason. The data warehouse is a centralized repository for structured, governed analytical workloads. Sources feed into an ETL or ELT pipeline, land in the warehouse, and serve BI tools and dashboards downstream.

The flow is deceptively simple: Sources → ETL/ELT → Warehouse → BI/Analytics.

People have been predicting the death of the data warehouse for over a decade. Today, the warehouse remains the dominant architecture for financial reporting, executive dashboards, and any workload where the CFO needs numbers they can trust. The reason is not technical sophistication — it is trust. A warehouse enforces schema-on-write. Data is cleaned, typed, and validated before it earns a seat at the table. When the auditor calls, you don’t want to explain that your revenue figures come from a directory of semi-structured JSON files in S3.

Here is something worth pausing on, because it connects directly to a misconception that has taken root in the modern data discourse. When people talk about Bronze, Silver, and Gold — the Medallion Architecture popularized by Databricks — they sometimes frame it as a novel invention of the lakehouse era. It is not. Before those labels existed, most warehouse shops already had some rough equivalent: a staging or landing area for inbound source data, a cleaned or production-ready area where transformations happened, and aggregated views, marts, cubes, SSAS models, or semantic layers on top for reporting and consumption. That was the same basic flow. The labels were uglier, the boundaries were fuzzier, and a distressing number of teams buried critical business logic in a rat’s nest of views and stored procedures that no one fully understood — but the bones were the same.

Ralph Kimball’s architecture made this layering explicit and disciplined. His “Enterprise Data Warehouse Bus Architecture” emphasized incremental development — build one business process at a time, not the whole warehouse in a single monolithic effort — and the reuse of conformed dimensions across those processes. A “customer” dimension, defined once in collaboration with data governance, would be shared across sales, marketing, and support data marts. This reduced redundant design work, sped up delivery, and — critically — meant that reports from different business units could be combined in a single drill-across query because the dimensions were guaranteed to align. Bill Inmon’s competing top-down approach pushed for a fully normalized enterprise model built first, with marts derived downstream. Different philosophy, same layered structure beneath it.

So when someone looks at a modern lakehouse pipeline and asks, “Why are we storing the data three times now?” — the honest answer is: in a lot of cases, you always were. You just were not naming it as clearly. Bronze is the staging area with better retention policies. Silver is the cleaned core layer with schema enforcement. Gold is the dimensional mart, the OLAP cube, the semantic model. Databricks gave it a catchy naming convention. The pattern has been there since the 1990s.

That said, warehouses have real limitations. They struggle with unstructured data. They punish you financially for exploratory workloads — spinning up compute to scan terabytes of raw logs in Snowflake or BigQuery gets expensive fast. And they impose a rigidity that frustrates data scientists who want to experiment with raw, messy, real-world data before committing to a schema.

The Data Lake: Freedom Without Guardrails

The data lake was the antidote to warehouse rigidity. Dump everything — structured, semi-structured, unstructured — into cheap object storage at massive scale. Worry about schema later. The promise was irresistible: low-cost storage, infinite flexibility, and the ability to retain every byte of data your organization produces.

Sources → Ingestion → Object Storage → Processing Engine → Analytics.

That pipeline reads like four clean handoffs. Each one hides a world of decisions.

Ingestion is the first place things get interesting — and the first place teams underestimate complexity. Getting data into the lake means dealing with dozens of source systems, each with its own protocols, schemas, rate limits, and failure modes.

The ingestion layer has split into two distinct camps. On one side, you have managed ELT platforms like Fivetran and Airbyte that handle batch extraction from SaaS apps, databases, and APIs with pre-built connectors.

On the other side, you have streaming ingestion via Kafka, Confluent Cloud, Amazon Kinesis, or Redpanda for continuous, low-latency event capture. And sitting between these two worlds is Change Data Capture (CDC) — tools like Debezium that read database transaction logs and emit row-level changes as events, letting you replicate operational databases into the lake in near real-time without hammering the source with full-table scans.

The ingestion layer is not glamorous, but it is the foundation that determines whether your lake has fresh, complete data or stale, partial snapshots that nobody trusts.

Once data lands in object storage, the processing engine is what turns raw files into something useful. This is where the lake’s flexibility becomes both its greatest strength and its biggest trap.

Unlike a warehouse — where the engine and the storage are tightly coupled — a lake separates the two, which means you get to choose your compute. And there are real trade-offs hiding in that choice. Apache Spark remains the workhorse for heavy ETL — terabyte-scale transformations, complex joins, ML feature engineering — with the broadest ecosystem and the best Python experience via PySpark. But Spark is optimized for throughput, not latency. It is a batch engine at heart, and spinning up a Spark cluster for an ad-hoc query feels like starting a freight train to deliver a letter.

Trino (the Presto fork) fills the interactive SQL gap — it is a distributed query engine built for low-latency analytics across heterogeneous sources, and it speaks directly to S3, Iceberg, Delta, HDFS, and relational databases through its connector architecture. Trino won’t replace Spark for heavy transformations, but for an analyst who needs to explore a dataset and get answers in seconds rather than minutes, it is the right tool.

Then there is DuckDB — the in-process, embedded analytical engine that runs on a single machine with zero infrastructure. DuckDB reads Parquet and CSV files directly, executes columnar analytics using vectorized processing, and in many benchmarks outperforms Spark and Trino on datasets that fit in memory. It is the data engineer’s Swiss army knife for local exploration, pipeline testing, and small-to-medium analytics. The catch: it does not distribute. Once your data outgrows a single node, you are back to Spark or Trino.

At the consumption end of the pipeline, the lake’s promise was that different consumers could access the same data through whatever tool fit their workflow. BI platforms — Tableau, Power BI, Looker, Apache Superset — would connect for dashboards and reporting. Data scientists would spin up notebooks on Jupyter or SageMaker for exploratory analysis and model training. ML pipelines would read raw training data directly from object storage. Application APIs would serve curated datasets to downstream microservices.

The vision was democratic access to a single, unified data store. The reality was that without a semantic layer — a consistent definition of what metrics mean, how entities relate, and what access policies apply — every consumer reinvented the wheel. The marketing team’s definition of “active user” diverged from the product team’s. Analysts built dashboards with subtly different filters. ML models trained on data sliced differently than what the business reports showed. The semantic layer — whether implemented through LookML, dbt metrics, Cube, or platform-native features like Snowflake Semantic Views and Databricks Metric Views — emerged precisely to solve this fragmentation, ensuring that every consumer, from a BI dashboard to an AI agent, computes “revenue” the same way.

Without enforced governance, data lakes quickly became data swamps — vast, uncurated collections of files that nobody trusted and fewer people could navigate. Security and access controls were bolted on as afterthoughts. Lineage was a mystery. A team would write data to a path like /raw/marketing/campaigns/2024/, another team would read from it six months later, and nobody could tell you if the schema had changed in between or whether those files were even complete.

Today, the standalone data lake is rarely the answer. It persists as a foundation layer — the cheap, durable storage tier underneath more sophisticated architectures. But if someone tells you their data strategy is “put it all in S3 and we’ll figure it out later,” they are describing a problem, not a solution.

The lake taught us an important lesson: flexibility without trust is chaos. And that lesson is exactly what gave rise to the next architecture.

The Lakehouse: The Default Modern Backbone

The Lakehouse is the architectural idea that dominated the mid-2020s, and for good reason. It unifies the data lake and the data warehouse by layering open table formats — Apache Iceberg, Delta Lake, Apache Hudi — on top of object storage, adding ACID transactions, schema enforcement, and time travel to what was previously a pile of files.

Sources → Ingestion → Object Storage + Iceberg/Delta → Multi-Engine Compute → BI + AI.

This is not a marketing rebranding. It is a genuine architectural shift. The key innovation is transactional metadata: instead of tracking data by directory paths (the old Hive way), lakehouse formats track individual files through snapshots and manifests. Every write is atomic. Every read sees a consistent view. You get schema evolution without breaking downstream queries, partition pruning without users knowing how data is physically organized, and the ability to query data as it existed at any point in the past.

The trade-offs are real, though, and often underestimated. Schema evolution in a growing Lakehouse is still painful. Renaming a column sounds trivial until you realize that three Spark jobs, two dbt models, and a Trino query all reference the old name. Partition evolution — changing how data is physically organized — is supported by Iceberg but requires careful planning to avoid query performance regressions during the transition period. And the sheer number of choices (Iceberg vs. Delta vs. Hudi vs. Paimon? Spark vs. Flink vs. Trino for compute? Which catalog?) creates a decision paralysis that can stall projects for months.

Then there is small file hell. Streaming ingestion writes many tiny files. Without regular compaction — merging small files into larger, optimized ones — query performance degrades steadily. This is not a theoretical concern; it is the single most common operational issue in production lakehouses. Every team running Iceberg or Delta at scale has a compaction job. Many of them were written in a panic after the first performance incident.

Despite all this, the lakehouse is the default modern analytical backbone. It solved the fundamental gap — warehouses offered trust but no flexibility, lakes offered flexibility but no trust — and it did so using open standards that prevent vendor lock-in. If you are starting a new data platform from scratch, this is probably where you begin.

Streaming-First: When Batch Is Too Slow

Not every problem can wait for a nightly batch run. Streaming-first architecture is designed around continuous event flow, enabling workloads where latency is measured in seconds, not hours: real-time personalization, fraud detection, operational monitoring, and AI inference on live data.

Event Sources → Kafka / Stream Bus → Stream Processing → Real-Time Store → APIs/Apps.

The canonical stack is Apache Kafka for durable event transport and Apache Flink for stateful stream processing. Kafka handles ingestion — durable, ordered, partitioned message streams that can be replayed. Flink handles the computation — windowed aggregations, complex event processing, joins across streams, all with exactly-once semantics and event-time watermarks.

This sounds clean on a diagram. In production, it is anything but. Handling late-arriving data in streaming pipelines is one of the most insidious problems in data engineering. An event arrives five minutes after its window closed. Do you recompute? Do you drop it? Do you send it to a dead-letter queue and reconcile later? Every choice has downstream consequences, and Flink’s watermark mechanism — while powerful — requires you to make these decisions explicitly in code.

Then there are the economics. Kafka’s network costs alone can account for 88% of its total operational expense, largely because of the “one write, many reads” pattern where consumers must read entire topics even when they need only a subset. Historical data backfilling is expensive because reading old data forces it through broker nodes, potentially destabilizing live traffic. And Kafka’s lack of native querying capability means you almost always need a secondary OLAP system downstream — adding complexity and cost.

Streaming-first is not an upgrade from batch. It is a different paradigm with different trade-offs. Use it when the business genuinely needs sub-minute freshness. Do not use it because it sounds impressive in an architecture review. The most expensive streaming pipeline is the one nobody actually needs.

Kappa: Everything Is a Stream

The Kappa architecture takes streaming-first to its logical extreme. Instead of maintaining separate batch and streaming pipelines (the Lambda approach), Kappa treats everything as a stream. Historical data? Replay the stream. Batch aggregations? Run them as a stream with a larger window. One codebase, one processing model, one operational surface.

Event Sources → Stream Processing → Materialized Views → Queries.

The appeal is simplicity. Lambda architecture — with its separate batch layer, speed layer, and serving layer — is operationally brutal. You maintain two codebases that must produce identical results, debug discrepancies between batch and real-time views, and manage twice the infrastructure. Kappa eliminates this entirely. Disney, Shopify, Uber, and Twitter have all adopted Kappa-style architectures for exactly this reason.

But Kappa has a hard constraint: it requires replayable streams. If your event log doesn’t retain sufficient history, you cannot reprocess. Kafka’s tiered storage has improved this, but replaying petabytes of events through a stream processor is still orders of magnitude more expensive than scanning the same data in a columnar format on object storage. For deep historical analysis — think training ML models on three years of clickstream data — Kappa is the wrong tool. It excels at operational workloads with bounded lookback windows and falls flat when you need the full weight of your data history.

The honest assessment: Kappa is a simpler operational model than Lambda, but it is not universally simpler than having a well-designed lakehouse with a streaming ingestion layer. The industry has converged toward hybrid approaches where streaming writes directly into transactional lakehouse tables, giving you the best of both worlds without the dogma of “everything must be a stream.”

Data Mesh: An Org Chart Wearing an Architecture Diagram

Here is where things get political. Data Mesh is not really a technology. It is an organizational model that happens to have architectural implications. The core idea: decentralize data ownership so that domain teams own their data as products, supported by a self-service platform and federated governance.

Domain Sources → Domain Data Products → Self-Service Platform → Governance Layer → Consumers.

Zhamak Dehghani’s original framework is intellectually compelling. Centralized data teams become bottlenecks. They don’t understand domain semantics. They create generic pipelines that serve nobody well. By pushing ownership to domains — the people who actually understand the data — you get higher-quality data products, faster iteration, and better alignment between data and business needs.

The implementation reality is far messier. Most Data Mesh initiatives fail, and the reasons are almost never technical. They fail because organizations underestimate the cultural shift required. You are asking domain teams — product engineers, marketing analysts, operations staff — to take on data engineering responsibilities they didn’t sign up for. You need each domain to have the skills, tooling, and incentive to build and maintain data pipelines. That requires investment in a self-service platform sophisticated enough that domain teams can ship data products without becoming infrastructure experts.

Without that platform maturity, Data Mesh degenerates into distributed chaos — every team running their own stack, no interoperability, no consistent governance, and a proliferation of data silos that is somehow worse than the centralized monolith you were trying to escape.

The rule of thumb from James Serra’s analysis is blunt: don’t even think about Data Mesh until you’re at massive scale with major pain points around central IT bottlenecks. For most organizations, it is a destination, not a starting point. And each domain within a mesh can implement its own analytical backbone — a lakehouse, a warehouse, whatever fits — which means you still need the other architectures in this list. Data Mesh is a layer on top, not a replacement for the foundations beneath it.

Data Fabric: The Metadata Control Plane

If Data Mesh is about organizational decentralization, Data Fabric is about intelligent integration across whatever mess you already have. It creates a metadata-driven control plane that provides unified access, governance, and discovery across hybrid and multi-cloud systems.

Distributed Data Sources → Metadata Layer → Policy Engine → Consumers.

The key differentiator is metadata intelligence. A data fabric doesn’t necessarily move data — it creates a semantic layer that knows where data lives, what it means, who owns it, and what policies govern its use. Data virtualization, knowledge graphs, and automated cataloging are core technologies. IBM’s framework positions data fabric as a network of data nodes — warehouses, lakes, IoT devices, transactional databases — all interacting through a unified governance and discovery layer.

In practice, data fabric shines in complex enterprises with decades of accumulated systems. A global bank with Oracle on-premise, Snowflake in AWS, legacy Hadoop clusters, and real-time feeds from trading platforms doesn’t have the luxury of starting fresh. It needs to create coherent data access across all of these systems without rewriting everything. That is the fabric’s sweet spot.

The challenge is implementation complexity. Building a metadata layer sophisticated enough to handle discovery, governance, lineage, and policy enforcement across heterogeneous systems is a multi-year investment. And the category has a vendor-marketing problem: “data fabric” is used to describe everything from simple catalog tools to full integration platforms, making it hard for practitioners to know what they’re actually evaluating.

Data Fabric and Data Mesh are not competing — they are complementary. A fabric’s metadata-driven discovery can serve as the platform layer that makes a mesh viable. The organizations getting this right are the ones that stop treating these as either/or choices.

Event-Driven: The Connective Tissue

Event-driven architecture is not a data platform — it is a communication pattern. Services interact through asynchronous events rather than synchronous API calls. A producer emits an event (say, “order placed”), and any number of consumers can react to it independently.

Producers → Event Broker → Consumers.

This is foundational infrastructure for scalable microservices and, by extension, for streaming data systems. The decoupling is the point: producers don’t know or care who consumes their events. Consumers scale independently. If one downstream service is slow, it doesn’t cascade back up the chain — the broker buffers events until the consumer catches up.

The trade-offs are well-documented but consistently underestimated by teams adopting this pattern for the first time. Debugging asynchronous flows is genuinely hard. A single business transaction might span dozens of events across multiple services. Achieving observability across those flows requires serious investment in distributed tracing. Spotify processes up to 8 million events per second and has built extensive observability tooling just to keep its event pipelines healthy.

Schema evolution is another landmine. One producer changes an event format, and dozens of consumers silently break. ING Bank’s case study on schema registries demonstrated that enforcing backward compatibility across thousands of event types was essential to keeping their payments platform stable. Without a schema registry and strict governance, event-driven systems degrade into a game of whack-a-mole where every deployment is a potential incident.

Event-driven architecture is not optional for modern distributed systems. But it is the hardest pattern on this list to operate well at scale.

AI-Native: Built for Models, Not Just Dashboards

The AI-Native architecture is the newcomer that reflects where the industry is heading. It is purpose-built for machine learning and generative AI workloads, with feature stores, vector databases, model training, and low-latency model serving as first-class citizens.

Sources → Feature Store → Vector DB → Model Training → Model Serving → Applications.

Traditional analytical architectures were designed to answer questions humans ask through dashboards and reports. AI-native architecture is designed to answer questions that models ask at inference time. The requirements are fundamentally different: you need online/offline feature parity (training features must match serving features exactly, or your model’s predictions diverge from reality), sub-millisecond feature retrieval for real-time inference, and the ability to store and search high-dimensional vector embeddings for RAG pipelines.

The vector database market has exploded, with purpose-built solutions like Weaviate, Pinecone, and Milvus competing alongside embedded vector search in established databases like MongoDB Atlas and PostgreSQL (pgvector). The choice depends on scale: if you have tens of millions of embeddings and need hybrid search (combining vector similarity with keyword and metadata filters), a dedicated vector database is justified. For smaller-scale RAG applications, pgvector or SQLite with vector extensions may be sufficient and simpler to operate.

The honest truth about AI-native architectures is that most organizations don’t need a separate one yet. If your AI workload is a handful of LLM-powered features sitting on top of an existing lakehouse, adding a feature store and a vector index to your existing platform is more pragmatic than building a parallel architecture. AI-native becomes essential when ML is a core product capability — recommendation engines, fraud detection systems, autonomous agents — not when you’re prototyping a chatbot.

Composable Stack: LEGO Blocks, Not Monoliths

The Composable Stack philosophy rejects the idea that your data platform should come from a single vendor. Instead, you decouple storage, compute, and serving into independent layers, each running best-of-breed tools connected via open interfaces.

Sources → Storage Layer → Independent Compute Engines → API Layer → Activation Tools.

This is less an architecture and more a design principle: avoid vendor lock-in by ensuring every layer can be swapped without rewriting the layers above and below it. Store data in open formats (Parquet, Iceberg) on commodity object storage. Query it with whatever engine fits the workload — Spark for heavy transformations, Trino for interactive SQL, DuckDB for local analytics, Flink for streaming. Serve it through a semantic or API layer that decouples consumers from the implementation details beneath.

The composable approach has measurable benefits — organizations report 37% shorter time-to-market on average compared to monolithic systems. But it introduces its own complexity: API sprawl, integration testing overhead, and the cognitive load of managing multiple tools. When your storage is S3, your table format is Iceberg, your catalog is Nessie, your transformation engine is dbt on Spark, your ad-hoc query engine is Trino, and your BI layer is Metabase, you have six systems to monitor, upgrade, and debug. Each integration point is a potential failure mode.

The composable stack is the natural evolution of the lakehouse idea. But it requires a team mature enough to manage the operational surface area. If you are a five-person data team, a managed end-to-end platform like Databricks or Snowflake — even with some lock-in — might be the right trade-off. If you are a fifty-person platform engineering org, composability is how you avoid being held hostage by a single vendor’s roadmap.

Reverse ETL: Closing the Loop

Reverse ETL is the architecture that finally acknowledges a dirty secret of the analytics industry: insights that stay in dashboards don’t change anything. The idea is straightforward — push transformed, analytical data back into operational systems like CRMs, marketing platforms, and SaaS tools.

Warehouse/Lakehouse → Transformation → SaaS Tools / CRM / Apps.

The classic data pipeline moves data from operational systems into the warehouse for analysis. Reverse ETL closes the loop, sending enriched segments, lead scores, and behavioral signals back into the tools where sales and marketing teams actually work. A data team builds a churn-risk model in dbt, scores every customer, and Reverse ETL syncs those scores into Salesforce so the account management team can act on them — without ever opening a dashboard.

The market has matured rapidly, with Hightouch processing over 2 trillion records synced in 2024 and Census being acquired by Fivetran in 2025 to create an end-to-end data movement platform. Companies report 15-30% reductions in customer acquisition costs through better targeting enabled by warehouse-native audience segmentation pushed to ad platforms.

But Reverse ETL in production is reliability engineering, not just “moving data around.” You need idempotent writes (re-runs must not create duplicates), identity resolution (matching warehouse records to CRM records is harder than it sounds), and robust monitoring for sync failures, schema drift, and count mismatches between source and destination. The failure mode is insidious: a broken sync doesn’t crash with a loud error — it silently stops updating, and nobody notices until the marketing team asks why their audience segment hasn’t changed in two weeks.

The build-vs-buy calculus here favors managed platforms for most teams. The sync engine, observability, identity mapping, and governance features that tools like Hightouch and Census provide would take months to replicate internally — and the DIY version almost never gets the monitoring and error handling right.

So, Which One Do You Actually Pick?

This is where I’m supposed to give you a decision matrix. I won’t. Decision matrices imply that architecture selection is a rational, context-free process. It is not. It is shaped by your team’s skills, your organization’s maturity, your data volumes, your budget, your existing systems, and — more than anyone admits — the opinions of whoever has the most political capital in the room.

What I will say is this:

Start with the Lakehouse as your analytical backbone unless you have a specific reason not to. It handles 80% of analytical workloads — BI, reporting, ML training, ad-hoc exploration — with open formats and decoupled compute. Build on it.

Add streaming when you have genuine real-time requirements, not perceived ones. If the business can tolerate data that is fifteen minutes old, a micro-batch approach on top of your lakehouse is simpler, cheaper, and easier to debug than a full Kafka + Flink deployment.

Consider Data Mesh only when organizational bottlenecks — not technology bottlenecks — are your primary pain point. And invest in the self-service platform first.

Deploy Reverse ETL early. The ROI is immediate and measurable. Getting warehouse insights into operational tools is where data work translates into business outcomes.

Layer in AI-Native components as your ML workloads mature — feature stores, vector search, model serving — rather than building a parallel architecture from day one.

And above all: resist the urge to architect for problems you don’t have yet. The most successful data teams I’ve seen are the ones that chose the simplest architecture that solved their current pain, built it well, operated it reliably, and evolved it incrementally as requirements changed. The worst ones are still debating whether to use Iceberg or Delta while their business runs on exported CSVs.

The architectures in this guide are not competing philosophies. They are tools. Learn what each one is good at, understand its trade-offs, and deploy it where it fits. That’s not a cop-out. That’s engineering.