Lakehouse Tables: Warehouse Features on Object Storage

Three shapes of storage. A data warehouse stores tables in a managed, tightly coupled engine: fast and well governed, but storage and compute are tied together. A data lake dumps raw files into object storage: cheap and flexible, but with no transactions, no schema enforcement and slow, careless queries. A lakehouse keeps the lake's open files and cheap storage and adds the warehouse's table behavior on top.

Columnar files. Formats like Parquet store each column's values together, compressed. A query that needs three of fifty columns reads only those three (column pruning), and similar values compress well. Each file also records statistics, such as the minimum and maximum of each column, which are used to skip files later.

The table is a log, not a folder. Object storage only offers whole-file writes and has no multi-file transactions. A table format (Delta Lake, Apache Iceberg, Apache Hudi) solves this with a small transaction log or manifest: each commit lists the files added and removed. Readers read the log to learn the current file list; a writer commits by atomically appending one log entry. A reader therefore sees the whole commit or none of it, and old entries give you snapshots and time travel.

Partitioning. Lay files out by a column that queries filter on, usually the date (event_date=2026-01-05/). A query with WHERE event_date = '2026-01-05' then opens one folder out of 365 for a year of data, about 0.3% of the files. This partition pruning is the biggest single saving, so choose a partition column that matches the common filters and is not so fine-grained that you create millions of tiny folders.

files = [  # (path, min_date, max_date, rows); the log lists the live files
    ('p1.parquet', '2026-01-01', '2026-01-31', 900_000),
    ('p2.parquet', '2026-02-01', '2026-02-28', 850_000),
    ('p3.parquet', '2026-03-01', '2026-03-31', 920_000),
    ('p4.parquet', '2026-01-15', '2026-03-10', 20_000),   # a late, wide file
]

def prune(files, lo, hi):
    return [f for f in files if f[2] >= lo and f[1] <= hi]   # keep files whose range overlaps

hit = prune(files, '2026-02-10', '2026-02-12')
print([f[0] for f in hit])                                    # ['p2.parquet', 'p4.parquet']
print(sum(f[3] for f in hit), 'of', sum(f[3] for f in files), 'rows opened')   # 870000 of 2690000 rows opened

Data skipping. Inside a partition, the per-file minimum and maximum let the engine skip files that cannot match, as in the sample above. Sorting or clustering data by a frequently filtered column keeps each file's range narrow, so more files are skipped. A wide, overlapping file (like p4) defeats the skip, which is why maintenance matters.

The small files problem. Streaming writes create many tiny files, and each file costs a metadata read and an open. A compaction job rewrites many small files into a few large ones (and the log records the swap atomically). Old files stay until a cleanup job removes those older than the retention window, which also bounds time travel.

Schema evolution and governance. The log stores the table schema, so adding a column is a metadata change, not a rewrite, and readers of older files fill in a default. Engines enforce the schema on write, which is the main thing a plain lake lacks. Because compute is separate from storage, several engines (SQL, Spark, a notebook) can read the same table.