Every company that grows past a certain size runs into the same wall: their data warehouse gets too expensive to hold everything, but a plain data lake gets too messy to trust. A lakehouse architecture exists to close that gap -it's what happens when someone finally asks why storage has to be either cheap or reliable, instead of both.
The idea is simple once you see a real example of the problem it solves. So let's start with one.
Netflix's 100 Petabyte Analytics Problem
Netflix stores over 100 petabytes of analytics data - viewing history, A/B test results, streaming quality metrics, recommendation model training data. Their data warehouse bill would be astronomical. But dumping everything into raw S3 files made it impossible to know which data was fresh, correct, or even queryable. They needed both cheap storage AND reliability, essentially a lakehouse architecture built on top of object storage. Apache Iceberg - born at Netflix - was their answer.
That last line is worth sitting with for a second. Iceberg wasn't built by a vendor trying to sell something -it was built by an engineering team that had genuinely hit this wall and needed a way out. That's usually a good sign a pattern is worth learning.
Warehouse vs Lake vs Lakehouse
To understand why Iceberg mattered, it helps to see the three options side by side -a classic Data Lake vs Data Warehouse comparison, plus the lakehouse sitting in between them.
Data Architecture Comparison:
Data Warehouse Data Lake Lakehouse
(Snowflake/BQ) (Raw S3/GCS) (Iceberg/Delta)
────────────── ────────────────── ───────────────── ─────────────────
Storage cost $23/TB/month $0.023/TB/month $0.023/TB/month
(1000x more) (S3 pricing) (S3 pricing)
Query speed Milliseconds Seconds-minutes Seconds
(optimized engine) (scan all files) (metadata skips)
Schema Enforced upfront None (raw dump) Enforced + evolved
ACID txns Yes No Yes
Data formats Proprietary Open (Parquet) Open (Parquet)
Data freshness Batch ETL (hours) Real-time ingest Real-time ingest
Time travel Limited None Full versioning
Vendor lock-in High None None
Lakehouse = Lake storage prices + Warehouse reliabilityThat last line is really the whole pitch. A warehouse gives you speed and structure at a steep price. A lake gives you cheap, open storage but no guarantees about what's actually in it. The lakehouse pattern isn't a third, separate thing -it's a lake that borrowed the warehouse's discipline without borrowing its price tag.
The Modern Data Stack
Once you accept that a lakehouse is the target, the next question is what actually sits between raw data and a dashboard someone can trust. This is where the modern data stack comes in -a fairly standard set of layers that most companies end up converging on.
Modern Data Architecture Flow:
[Data Sources]
MySQL, PostgreSQL, APIs, Events, Logs
|
[Ingestion Layer]
Fivetran / Airbyte / Kafka Connect
|
[Raw Storage - S3/GCS] <-- $0.023/TB/month
Parquet files organized as Iceberg/Delta tables
|
[Table Format Layer] <-- THIS is the lakehouse magic
Delta Lake / Iceberg / Hudi
Adds: ACID, schema, versioning, partition pruning
|
[Transformation - dbt]
SQL models: raw -> staged -> marts
|
[Query Engine]
Spark / Trino / Athena / Presto
|
[Consumption]
Dashboards (Looker) | ML (SageMaker) | ExportsNotice where the actual magic happens. It's not the storage layer, and it's not the query engine — both of those are fairly ordinary. It's the table format layer sitting quietly in the middle, which is the one piece that turns a folder full of files into something a warehouse-grade query can actually trust.
Table Formats Compared
Which brings up the real decision most teams have to make: Apache Iceberg vs Delta Lake, and where Hudi fits in. All three solve the same underlying problem, but they came from different companies solving different pains first.
Open Table Format Comparison:
Feature Delta Lake Apache Iceberg Apache Hudi
──────────────── ────────────── ────────────── ──────────────
Creator Databricks Netflix Uber
Best for Batch analytics Complex schemas Streaming upserts
ACID Yes Yes Yes
Schema evolution Yes Yes (best) Yes
Time travel Yes Yes Yes
Partition evolve Limited Yes (hidden) Limited
Community Largest Fastest growing Niche
Pick Iceberg if: you need schema evolution + partition evolution
Pick Delta if: you are on Databricks
Pick Hudi if: you need streaming upserts (CDC from databases)None of these is objectively "the best" -the right pick depends entirely on what you're already running and what kind of writes you're dealing with. Delta Lake makes the most sense if Databricks is already your platform. Iceberg tends to win when your schemas change often and you don't want to think about partitioning by hand. Hudi is the one to reach for when you're pulling continuous change-data-capture streams out of production databases rather than batch-loading files once a day.
Data Lakehouse in System Design Interviews
This topic shows up more often in interviews than people expect, especially for roles touching analytics, data platforms, or anything ML-adjacent. What separates a strong answer from a weak one usually isn't knowing that Iceberg exists -it's being able to explain why the lakehouse pattern exists at all, and walking through the flow from raw ingestion to a query engine without missing a layer.
Interview Tip
When discussing analytics at scale, say: 'For petabyte-scale analytics, I would use a lakehouse architecture: store all data in Parquet format on S3 at $0.023/TB/monath, organized with Apache Iceberg for ACID transactions and schema evolution. Ingestion via Kafka Connect for real-time and Fivetran for batch sources. Transformations in dbt, queries via Trino or Spark. This gives warehouse reliability at lake storage costs, with no vendor lock-in.'
Key Takeaway
Data lakes are cheap and flexible but become swamps without governance. Data warehouses are structured and fast but expensive and rigid. The lakehouse pattern adds ACID transactions, schema enforcement, and versioning to data lakes via Delta Lake or Iceberg - combining cheapness with reliability.