The Standardization Attempt That Failed and What It Taught Us
By the late 1960s, Bachman's network model was winning the enterprise database market. Dozens of vendors were building incompatible systems. IBM had IMS. Honeywell had IDS derivatives. Each vendor's database was a proprietary island. A programmer who learned one system could not transfer their skills to another. Industry groups lobbied for standardization.
The Conference on Data Systems Languages, CODASYL, was the same committee that had standardized COBOL. In 1969 they published the Data Base Task Group report, defining a standard network database model. The spec was thorough, politically negotiated, and described exactly how navigation-based databases should work. It specified record types, set types for relationships, and navigation commands: FIND, GET, STORE, MODIFY, CONNECT, DISCONNECT.
Why CODASYL Failed
CODASYL's problem was not technical. The navigation model it standardized actually worked. The problem was philosophical. Every CODASYL query was a program. You did not say 'give me all orders for customer X.' You said: find customer X, then navigate to the first order in the customer-order set, then get the next order, and repeat until done. The programmer's brain had to hold the physical data structure in mind at all times.
Worse, data independence did not exist. If the business added a new relationship, say tracking which warehouse fulfilled each order, every program that navigated order records had to be rewritten to account for the new pointer structure. A schema change was a codebase change. Development teams were terrified of modifying their data structures because the blast radius was the entire application.
CODASYL crystallized the exact problem Edgar Codd was solving. The network model tied programs to physical storage layout. Every schema change broke existing code. Codd's relational model introduced data independence: the separation of logical structure from physical storage that made SQL possible.
The File System Era
While the CODASYL debate raged in standards committees, most real-world data lived in flat files. COBOL programs processed fixed-width records: name in columns 1-30, address in columns 31-80, balance in columns 81-90. These were not abstractions. They were literal byte positions in a file. A 'join' between two files meant writing a sort-merge program that took days to build and ran overnight.
The flat file era looks primitive from today's perspective, but it encoded a philosophy that never went away. Data lakes, CSV pipelines, log aggregation, and Parquet files in data warehouses are all flat file architectures. Hadoop popularized the idea that raw files were sometimes better than databases for analytical workloads. The format that COBOL programmers used in 1969 and the format that Spark reads in 2024 are spiritually identical: rows of fields, no schema enforcement, no relationships, no transactions.
Understanding when a flat file beats a database is still one of the most valuable instincts a data engineer can have. Write-once, read-many bulk loads? Flat files. Random access with complex queries? Database. The 1960s answer was the same as the 2024 answer.
The Transition to Relational
Edgar Codd was already writing the paper that would make CODASYL irrelevant. His insight was that the navigation model forced programmers to think like the database. The right approach was the reverse: the database should think like the programmer. Describe what you want as a mathematical predicate. Let the system figure out how to retrieve it. The programmer's job ends at specifying the query. The database engine's job begins.
- CODASYL lesson: Standards committees can standardize the wrong abstraction level. Standardizing navigation commands locked in a fundamentally limited model.
- Flat file lesson: Simplicity scales. When you do not need transactions, relationships, or random access, a flat file beats a database on every axis.
- Data independence lesson: Programs should depend on logical data models, not physical storage layouts. Any time you expose physical structure to callers, you pay in brittleness.
What Standardization Committees Teach About Technology Adoption
CODASYL's failure was not that standards committees are bad. COBOL, the language CODASYL standardized first, was a massive success and remains in production on mainframes that process trillions of dollars daily. CODASYL's database standard failed because it standardized a flawed model at the wrong level of abstraction. The lesson is about standardizing interface, not implementation. COBOL standardized a programming language interface that programs could be written in. The DBTG standard specified navigation operations that required programmers to think about physical data structure. That difference in abstraction level determined one standard's longevity and the other's obsolescence.
The flat file lesson went in a different direction. Flat files were never standardized in the 1960s because they were too simple to need standardization. Each program defined its own record format. But that simplicity made them portable, inspectable, and recoverable. When a CODASYL database got corrupted, only the vendor's tools could fix it. When a flat file got corrupted, any text editor could examine it. This auditability and debuggability made flat files survive long after databases became dominant.
Flat Files in Modern Data Engineering
The modern data engineering stack is built on flat files. Apache Parquet is a columnar flat file format used by every major data lake (S3, GCS, ADLS). CSV files are still the universal data exchange format between systems that do not share a database. JSON Lines (JSONL) is the format used for log streaming, event buses, and API responses written to files. The flat file philosophy, simple structure, no proprietary format, inspectable with basic tools, drives the data lake architecture that displaced the enterprise data warehouse.
The key insight from the file system era: data storage formats that can be read without specialized tools outlast those that cannot. Parquet can be read by Python, Spark, Athena, BigQuery, and a dozen other systems. A CODASYL database can only be read by software that understands its specific pointer structure. Format longevity correlates strongly with how many independent tools can read the format. When designing data pipelines, prefer formats with broad ecosystem support over formats with perfect optimization for one workload.
The Transition Pattern That Repeats in Every Technology Generation
- CODASYL navigation model gave way to SQL's declarative queries. The higher abstraction level won.
- Monolithic SQL databases gave way to microservices with independent data stores. Isolation won over shared state.
- SQL databases are giving way to multi-model databases (PostgreSQL with JSONB, pgvector, PostGIS). Extensibility is winning over purity.
- Batch ETL is giving way to streaming pipelines. Real-time access is winning over nightly batch.
- Each transition follows the same pattern: the new model offers a higher abstraction level, more flexibility, or better performance at the cost of the previous model's simplicity or maturity.
What Standardization Committees Teach About Technology Adoption
CODASYL's failure was not that standards committees are bad. COBOL, the language CODASYL standardized first, was a massive success and remains in production on mainframes that process trillions of dollars daily. CODASYL's database standard failed because it standardized a flawed model at the wrong level of abstraction. The lesson is about standardizing interface, not implementation. COBOL standardized a programming language interface that programs could be written in. The DBTG standard specified navigation operations that required programmers to think about physical data structure. That difference in abstraction level determined one standard's longevity and the other's obsolescence.
The flat file lesson went in a different direction. Flat files were never standardized in the 1960s because they were too simple to need standardization. Each program defined its own record format. But that simplicity made them portable, inspectable, and recoverable. When a CODASYL database got corrupted, only the vendor's tools could fix it. When a flat file got corrupted, any text editor could examine it. This auditability and debuggability made flat files survive long after databases became dominant.
Flat Files in Modern Data Engineering
The modern data engineering stack is built on flat files. Apache Parquet is a columnar flat file format used by every major data lake (S3, GCS, ADLS). CSV files are still the universal data exchange format between systems that do not share a database. JSON Lines (JSONL) is the format used for log streaming, event buses, and API responses written to files. The flat file philosophy, simple structure, no proprietary format, inspectable with basic tools, drives the data lake architecture that displaced the enterprise data warehouse.
The key insight from the file system era: data storage formats that can be read without specialized tools outlast those that cannot. Parquet can be read by Python, Spark, Athena, BigQuery, and a dozen other systems. A CODASYL database can only be read by software that understands its specific pointer structure. Format longevity correlates strongly with how many independent tools can read the format. When designing data pipelines, prefer formats with broad ecosystem support over formats with perfect optimization for one workload.
The Transition Pattern That Repeats in Every Technology Generation
- CODASYL navigation model gave way to SQL's declarative queries. The higher abstraction level won.
- Monolithic SQL databases gave way to microservices with independent data stores. Isolation won over shared state.
- SQL databases are giving way to multi-model databases (PostgreSQL with JSONB, pgvector, PostGIS). Extensibility is winning over purity.
- Batch ETL is giving way to streaming pipelines. Real-time access is winning over nightly batch.
- Each transition follows the same pattern: the new model offers a higher abstraction level, more flexibility, or better performance at the cost of the previous model's simplicity or maturity.
