How Systems Learned to Talk

On October 29, 1969, Charley Kline at UCLA typed "LOGIN" to send a message to a machine at Stanford Research Institute. The system crashed after the first two letters - only "LO" made it through. It was the first message ever sent between two computers on a network. But ARPANET's real contribution wasn't just connecting machines. It introduced the Request for Comments (RFC) process - a collaborative, open way to define how systems should communicate. RFC 1 was published in April 1969. Today there are over 9,000 RFCs defining everything from HTTP to WebSockets. The entire internet is built on this culture of proposing, debating, and standardizing interfaces between systems.

FTP was the internet's first real application-layer protocol. Before FTP, transferring a file between machines required writing custom code that understood the other machine's operating system, file format, and storage layout. FTP abstracted all of that behind a standard set of commands: RETR to download, STOR to upload, LIST to browse. It introduced the concept of a "protocol" as a contract between client and server - a shared vocabulary of commands and responses that both sides agree to follow. FTP also introduced the idea of status codes (200 for success, 530 for not logged in) that HTTP would later adopt and expand. The pattern of "verb + resource + status code response" that every REST API uses today started right here.

Tim Berners-Lee didn't set out to create the backbone of the API economy. He wanted physicists at CERN to share research papers. His solution was radically simple: every document gets a unique address (URL), and you retrieve it with a single command (GET). HTTP/0.9 was so minimal it only supported one operation - GET - and could only return HTML. No headers. No status codes. No content negotiation. Just "give me the document at this address." But that simplicity was its genius. Anyone could implement it. Any platform could speak it. Within two years, there were dozens of web servers and browsers, all interoperable because the protocol was too simple to get wrong. Every complex API you'll ever design is a descendant of this one-verb, one-format protocol.

The Common Gateway Interface turned the web from a document library into an application platform. Before CGI, web servers could only serve static files. CGI let a web server hand off a request to any executable program - a Perl script, a C program, a shell script - and return its output as the HTTP response. This was the moment the "request-response" pattern became an "API call." A CGI script that accepted query parameters and returned dynamic HTML was functionally identical to a modern API endpoint. The URL was the endpoint. The query string was the request parameters. The response was the output. CGI scripts were slow (they spawned a new process per request) and messy (they parsed raw environment variables), but they proved that HTTP could do more than serve files. They proved it could be the transport layer for applications.

Dave Winer had a pragmatic idea: what if you could call functions on remote servers using plain HTTP and XML? No CORBA brokers, no IDL compilers, no platform dependencies. Just POST an XML document describing the function name and arguments to a URL, and get back an XML document with the result. XML-RPC was intentionally simple - the entire spec fit on one page. It supported basic types (integers, strings, booleans, dates, arrays, structs) and nothing else. No sessions, no authentication, no service discovery. This constraint was deliberate. Winer believed that simplicity was more important than features, and that developers would build what they needed on top of a simple foundation. XML-RPC proved that HTTP could carry more than web pages. It was the direct ancestor of SOAP - though SOAP would violate every principle of simplicity that XML-RPC stood for.

SOAP started as a natural evolution of XML-RPC: a standard way to call remote procedures using XML over HTTP. But Microsoft and IBM had grander ambitions. They wanted SOAP to be the universal protocol for all enterprise communication. The original spec was reasonable - XML envelopes with headers and bodies, fault handling, and encoding rules. Then came the extensions. WS-Security for authentication. WS-ReliableMessaging for guaranteed delivery. WS-Transaction for distributed transactions. WS-Addressing, WS-Policy, WS-Federation. Each specification added more XML, more complexity, more configuration. The community called it "WS-Deathstar." A simple SOAP request to fetch a user required 30+ lines of XML boilerplate before you got to the actual data. Developers spent more time fighting WSDL files and SOAP faults than building features. SOAP became the cautionary tale that every API designer should study.

In Chapter 5 of his doctoral dissertation, Roy Fielding didn't invent a new technology. He described the architecture that already made the web work - and gave it a name: Representational State Transfer (REST). Fielding was uniquely qualified to do this because he was a principal author of the HTTP/1.1 specification. He understood the web's architecture from the inside. REST defined six constraints: client-server separation, statelessness, cacheability, a uniform interface, layered system, and optional code-on-demand. The key insight was "resources." Every thing in your system gets a unique identifier (a URL). You interact with it using a small, fixed set of operations (HTTP methods). The server sends back a "representation" of the resource's current state. No sessions. No stubs. No IDL. Just URLs, verbs, and representations. While SOAP required pages of XML configuration, REST said: "The web already works. Build APIs the same way."

In 2002, Jeff Bezos sent an email to every team at Amazon. The mandate was simple and absolute: (1) All teams must expose their data and functionality through service interfaces. (2) Teams must communicate with each other through these interfaces. (3) All service interfaces must be designed from the ground up to be externalizable - no exceptions. (4) Anyone who doesn't do this will be fired. This wasn't about public APIs. Bezos understood that if Amazon's internal services had clean, well-defined interfaces, they could be opened to external developers later. That's exactly what happened - Amazon's internal service interfaces became Amazon Web Services. S3, EC2, SQS - these were internal tools first. The Bezos mandate proved that API-first thinking transforms organizations. When every team communicates through APIs, you get clear boundaries, independent deployment, and services that can scale independently.

Before OAuth, if a third-party app wanted to access your Twitter account, you gave it your password. The app stored your actual credentials. If the app got hacked, attackers had your password. If you wanted to revoke access, you had to change your password - which broke every other app that had it. OAuth solved this by introducing delegated authorization. Instead of sharing your password, you authorize the app through Twitter directly, and the app receives a token - a limited, revocable credential that proves you authorized it. OAuth 1.0 used cryptographic signatures (HMAC-SHA1) for every request, making it secure but painful to implement. Developers had to sort parameters, construct signature base strings, and manage nonces. The complexity was real, but the principle - delegated access without sharing credentials - became the foundation of API security.

Jeff Lindsay coined the term "webhooks" to describe a pattern that flipped the API model: instead of the client polling the server repeatedly ("any new data? any new data? any new data?"), the server calls the client when something happens. You register a callback URL, and the server POSTs event data to it when a relevant event occurs. Payment completed? Webhook. New commit pushed? Webhook. Order shipped? Webhook. This was the first widely-adopted pattern for event-driven APIs over HTTP. Webhooks were simple to understand and implement - just handle an incoming POST request. But they introduced new challenges: What if the receiving server is down? How do you verify the webhook is really from the sender and not an attacker? How do you handle duplicate deliveries? Retry policies, HMAC signatures, and idempotency keys all emerged as solutions to webhook reliability problems.

In 2012, Facebook's mobile app was a disaster. It was built as a web app wrapped in a native shell, and it was painfully slow. Mark Zuckerberg called HTML5 on mobile "one of our biggest mistakes." The company rewrote the app natively, but a deeper problem emerged: the REST APIs serving the desktop website were terrible for mobile. A single News Feed screen required dozens of REST calls - one for the post, one for the author, one for the comments, one for the likes, one for the shared content. Each call returned far more data than the mobile app needed (the API was designed for the feature-rich desktop site), and the round trips over slow 3G connections made the app feel broken. This wasn't a REST problem per se - it was a mismatch between REST's resource-oriented model and mobile's need for highly-customized, minimal data payloads. The solution required rethinking the API paradigm entirely.

Lee Byron, Dan Schafer, and Nick Schrock created GraphQL inside Facebook to solve the mobile data fetching problem. The core insight was revolutionary: instead of the server defining fixed response shapes through endpoints, let the client specify exactly what it wants using a query language. A mobile app that only needs the user's name and avatar sends: { user(id: "123") { name, avatarUrl } }. A desktop app that needs the full profile sends a different query to the same endpoint. One endpoint, infinite response shapes, zero over-fetching. GraphQL introduced a type system where every field, argument, and return value has a declared type. The schema is both documentation and validation - the server knows exactly what queries are valid and rejects malformed requests before executing them. This was the type safety that REST APIs never had, without the ceremony of WSDL. Internally at Facebook, GraphQL served over 100 billion API requests per day within two years of adoption.

By 2001, Google was running into a problem that most companies hadn't experienced yet: JSON and XML were too slow. When you're making billions of inter-service calls per day, the CPU time spent parsing text-based formats and the bytes wasted on field names in every message add up to entire data centers worth of compute. Google's solution was Protocol Buffers (protobuf): a binary serialization format where data is encoded as numbered fields with type tags. Instead of {"user_name": "Alice", "age": 30} (31 bytes of text with redundant field names), protobuf encodes the same data in ~10 bytes of binary. You define message structures in .proto files, and the protobuf compiler generates serialization code in your target language. The generated code is zero-copy where possible and avoids reflection entirely. Parsing a protobuf message is 10-100x faster than parsing the equivalent JSON. Google open-sourced Protocol Buffers in 2008, but they'd been running on protobuf internally for seven years by then.

Stubby was Google's internal RPC framework - the system that every Google service used to talk to every other Google service. It was built on top of Protocol Buffers and designed for Google's internal network, where latency between data centers was measured in microseconds and reliability was handled by the infrastructure. Stubby handled load balancing, failover, monitoring, and tracing across Google's entire fleet. By 2010, Google was processing over 10 billion Stubby RPC calls per second across its infrastructure. Stubby proved that RPC at massive scale was not only possible but necessary - the overhead of REST (text parsing, connection management, header processing) was measurable at Google's scale. But Stubby was tied to Google's internal infrastructure, its proprietary transport protocol, and its specific deployment model. Making it work outside Google required rethinking the transport layer from scratch.

JPMorgan Chase spent $200 million per year on proprietary messaging middleware from IBM (MQ Series) and TIBCO. John O'Hara proposed something radical: an open standard for message-oriented middleware. The Advanced Message Queuing Protocol (AMQP) defined a wire protocol for message brokers - exchanges, queues, bindings, and routing rules. Unlike JMS (Java Message Service), which was a Java API without a wire protocol, AMQP was language-agnostic from the start. A Python publisher could send messages through a RabbitMQ broker to a Java consumer. AMQP introduced exchange types that became messaging fundamentals: direct (point-to-point), fanout (broadcast to all), topic (pattern-based routing), and headers (metadata-based routing). RabbitMQ, the most popular AMQP broker, handles billions of messages daily across industries. AMQP proved that asynchronous communication between services was not just possible but often superior to synchronous HTTP calls.

LinkedIn needed to process billions of events per day - page views, profile updates, job applications, connection requests - and traditional message brokers couldn't keep up. Jay Kreps, Jun Rao, and Neha Narkhede built Kafka around a deceptively simple idea: a distributed, append-only commit log. Producers append messages to topic partitions. Consumers read from those partitions at their own pace, maintaining their own position (offset) in the log. Messages aren't deleted after consumption - they're retained for a configurable period (days, weeks, forever). This meant multiple consumers could read the same messages independently, at different speeds, for different purposes. One consumer populates a search index. Another feeds a real-time dashboard. A third trains a machine learning model. All from the same stream of events. Kafka wasn't a queue (messages aren't deleted after reading) or a database (it's append-only). It was a new category: a distributed streaming platform. Today, over 80% of Fortune 100 companies run Kafka in production.

As microservices architectures grew from a handful to hundreds of services, exposing each service directly to clients became unmanageable. API gateways emerged as the "front door" - a single entry point that handles routing, authentication, rate limiting, and protocol translation. Netflix's Zuul handled request routing for their streaming platform. AWS API Gateway made the pattern available as a managed service. Kong, built on NGINX, became the most popular open-source option. The gateway pattern separated cross-cutting concerns from business logic. Individual services didn't need to implement authentication, rate limiting, or request logging - the gateway handled it uniformly. This was the ESB pattern done right: thin middleware that routes and validates, without owning business logic. The gateway also enabled protocol translation - accept REST from external clients, convert to gRPC for internal services, and the client never knows.

Different clients need different APIs. A mobile app needs minimal, aggregated responses to minimize data transfer. A desktop web app needs rich, detailed responses for feature-dense UIs. An admin dashboard needs raw, unprocessed data for analytics. The Backend-for-Frontend (BFF) pattern gives each frontend its own dedicated API layer. The mobile team owns a Mobile BFF that aggregates and slims down responses from microservices. The web team owns a Web BFF that composes data for complex UI components. Each BFF is owned by the frontend team that uses it - they control the API contract and can evolve it at their own pace. SoundCloud pioneered this pattern when their single API couldn't serve both their website and mobile apps efficiently. Phil Calçado documented the pattern, and Sam Newman popularized it in his microservices book. The BFF pattern acknowledges a truth REST tried to ignore: different clients have different needs, and pretending one API fits all is a design compromise.