Protocol Buffers: Binary Serialization
Google's internal infrastructure was processing requests at a scale where JSON's text parsing overhead was measurable. Protocol Buffers (protobuf) was built internally around 2001 and released publicly in 2008. The core idea: define your data schema in a .proto file using an IDL syntax, run the protoc compiler, and get generated classes in any supported language that serialize and deserialize binary data at high speed.
Protobuf serialized data is roughly 3-10x smaller than equivalent JSON. Parsing is significantly faster because there is no string tokenization. Fields are identified by number, not name, so renaming a field does not break serialization compatibility. The schema is required: you cannot deserialize protobuf data without the .proto definition, which is both a constraint and a guarantee of contract enforcement.
Stubby: Internal RPC Before gRPC
Google's Stubby was the internal RPC framework that gRPC descended from. Stubby had been running Google's internal service communication for years before the public release. It used Protocol Buffers for serialization, a custom binary framing protocol for multiplexed connections, and Google's internal service mesh for discovery and load balancing. gRPC was essentially Stubby rebuilt on open standards (HTTP/2 for transport) so the rest of the world could use it.
HTTP/2: Binary Framing at the Transport Layer
HTTP/2 (2015) kept the HTTP semantics (methods, status codes, headers) but replaced the text wire format with binary framing. The key features that matter for API design: multiplexing (multiple requests share one TCP connection, solving HTTP/1.1 head-of-line blocking), server push (the server can send responses the client has not requested yet), header compression (HPACK), and stream prioritization.
gRPC: The Modern RPC Framework
Google open-sourced gRPC in 2016. Define your service in a .proto file (methods, request types, response types), run the compiler, and get fully typed client and server code in Go, Java, Python, C#, Ruby, and more. The generated code handles serialization, connection management, streaming, and error handling. You write the business logic.
gRPC supports four communication patterns that REST cannot express cleanly: unary (single request, single response), server streaming (single request, stream of responses), client streaming (stream of requests, single response), and bidirectional streaming (both sides stream simultaneously). These patterns map directly to real use cases: file uploads, log streaming, sensor data collection, real-time collaboration.
Envoy Proxy: The Service Mesh Foundation
Envoy (2016, Lyft) is the proxy that underpins most service meshes. Critically, it speaks gRPC natively and can transcode between gRPC and REST HTTP/JSON. This means you can build your internal services with gRPC for performance and expose REST endpoints externally through Envoy without maintaining two separate APIs. Istio, AWS App Mesh, and Google Cloud Traffic Director all use Envoy as the data plane.
When to Choose gRPC Over REST
gRPC adds real value in specific scenarios. Internal service-to-service communication: both sides are under your control, both support gRPC, and you need the performance of binary serialization and HTTP/2 multiplexing. Streaming workloads: you need to stream data from client to server, server to client, or bidirectionally. The four gRPC streaming modes (unary, server streaming, client streaming, bidirectional) have no clean REST equivalent. Strongly typed interfaces: when a polyglot microservices environment (Go services calling Python services calling Java services) needs compile-time contract enforcement across languages.
REST is the right choice when: you are designing a public API that external developers will consume (gRPC requires code generation and HTTP/2, which adds friction); your clients are browsers (gRPC is only supported via gRPC-Web, a proxy-based adaptation layer); or your team is not familiar with Protocol Buffers. Mixing gRPC internally and REST externally is common and correct. The API gateway (or Envoy proxy) handles the transcoding.
Protocol Buffer Schema Evolution Rules
Protocol Buffers handle schema evolution through field numbers. Each field has a number that is stable across schema versions. Adding a new field with a new number is always backward compatible: old clients ignore unknown field numbers. Removing a field is backward compatible if you reserve the field number (to prevent reuse). Changing a field's type is not backward compatible and requires a field number change. Renaming a field is fine because field names are not used in the binary encoding.
The evolution rules make protocol buffers safer than JSON for long-lived data. A JSON payload from 2019 stored in a data lake cannot be safely deserialized if the schema changed in 2021 without checking which version of the schema to apply. A protobuf payload carries its field numbers in the binary encoding. As long as you followed the evolution rules, any version of the reader can deserialize any version of the payload.
- Never reuse a field number in a .proto file. Protobuf uses numbers, not names, for binary encoding. Reusing a number causes data corruption.
- Use reserved field numbers and names when removing a field: reserved 5; reserved 'old_field_name';
- Proto3 removed required fields (a source of compatibility bugs in proto2). All fields are optional by default. Use proto3 for new schemas.
- gRPC status codes are separate from HTTP status codes. gRPC has 16 canonical status codes (OK, CANCELLED, UNKNOWN, INVALID_ARGUMENT, etc.). These are mapped to HTTP status codes by grpc-gateway and Envoy.
- Deadlines propagate across gRPC calls. If a client sets a 100ms deadline and the first service takes 30ms, the downstream call automatically has a 70ms deadline. This prevents cascading timeouts.
gRPC in the Service Mesh Era
gRPC integrates naturally with service mesh infrastructure because both operate at the same layer (L7 application protocol). Envoy, Istio, and Linkerd all have native gRPC support: they can load balance gRPC traffic, collect per-RPC telemetry, enforce per-method authorization policies, and apply traffic management rules (canary deployments, circuit breaking, retry policies) at the individual RPC level. This is more granular than REST-level traffic management, which typically operates per-URL.
The gRPC health checking protocol is standardized (grpc.health.v1.Health service) and supported natively by Kubernetes liveness and readiness probes. Load balancers that understand gRPC (as opposed to treating it as opaque TCP) can make per-RPC routing decisions rather than per-connection decisions - critical for server-streaming and bidirectional streaming RPCs that maintain long-lived connections. Switching from a TCP load balancer to an HTTP/2-aware load balancer for gRPC workloads can significantly improve load distribution.
Protocol Buffers code generation also creates a tight coupling between service version and client version. A message field that exists in the server but not in the client proto definition is silently dropped on deserialization. A message field that exists in the client proto but not in the server is ignored by the server. This backward/forward compatibility is by design and allows you to evolve services independently - but only if you follow the rules: never remove fields (mark them reserved instead), never change a field number, never change a field type. Violate these rules and silent data corruption follows. The buf lint tool enforces these rules automatically in CI.
When to Choose gRPC Over REST
gRPC adds real value in specific scenarios. Internal service-to-service communication: both sides are under your control, both support gRPC, and you need the performance of binary serialization and HTTP/2 multiplexing. Streaming workloads: you need to stream data from client to server, server to client, or bidirectionally. The four gRPC streaming modes (unary, server streaming, client streaming, bidirectional) have no clean REST equivalent. Strongly typed interfaces: when a polyglot microservices environment (Go services calling Python services calling Java services) needs compile-time contract enforcement across languages.
REST is the right choice when: you are designing a public API that external developers will consume (gRPC requires code generation and HTTP/2, which adds friction); your clients are browsers (gRPC is only supported via gRPC-Web, a proxy-based adaptation layer); or your team is not familiar with Protocol Buffers. Mixing gRPC internally and REST externally is common and correct. The API gateway (or Envoy proxy) handles the transcoding.
Protocol Buffer Schema Evolution Rules
Protocol Buffers handle schema evolution through field numbers. Each field has a number that is stable across schema versions. Adding a new field with a new number is always backward compatible: old clients ignore unknown field numbers. Removing a field is backward compatible if you reserve the field number (to prevent reuse). Changing a field's type is not backward compatible and requires a field number change. Renaming a field is fine because field names are not used in the binary encoding.
The evolution rules make protocol buffers safer than JSON for long-lived data. A JSON payload from 2019 stored in a data lake cannot be safely deserialized if the schema changed in 2021 without checking which version of the schema to apply. A protobuf payload carries its field numbers in the binary encoding. As long as you followed the evolution rules, any version of the reader can deserialize any version of the payload.
- Never reuse a field number in a .proto file. Protobuf uses numbers, not names, for binary encoding. Reusing a number causes data corruption.
- Use reserved field numbers and names when removing a field: reserved 5; reserved 'old_field_name';
- Proto3 removed required fields (a source of compatibility bugs in proto2). All fields are optional by default. Use proto3 for new schemas.
- gRPC status codes are separate from HTTP status codes. gRPC has 16 canonical status codes (OK, CANCELLED, UNKNOWN, INVALID_ARGUMENT, etc.). These are mapped to HTTP status codes by grpc-gateway and Envoy.
- Deadlines propagate across gRPC calls. If a client sets a 100ms deadline and the first service takes 30ms, the downstream call automatically has a 70ms deadline. This prevents cascading timeouts.
gRPC in the Service Mesh Era
gRPC integrates naturally with service mesh infrastructure because both operate at the same layer (L7 application protocol). Envoy, Istio, and Linkerd all have native gRPC support: they can load balance gRPC traffic, collect per-RPC telemetry, enforce per-method authorization policies, and apply traffic management rules (canary deployments, circuit breaking, retry policies) at the individual RPC level. This is more granular than REST-level traffic management, which typically operates per-URL.
The gRPC health checking protocol is standardized (grpc.health.v1.Health service) and supported natively by Kubernetes liveness and readiness probes. Load balancers that understand gRPC (as opposed to treating it as opaque TCP) can make per-RPC routing decisions rather than per-connection decisions - critical for server-streaming and bidirectional streaming RPCs that maintain long-lived connections. Switching from a TCP load balancer to an HTTP/2-aware load balancer for gRPC workloads can significantly improve load distribution.
Protocol Buffers code generation also creates a tight coupling between service version and client version. A message field that exists in the server but not in the client proto definition is silently dropped on deserialization. A message field that exists in the client proto but not in the server is ignored by the server. This backward/forward compatibility is by design and allows you to evolve services independently - but only if you follow the rules: never remove fields (mark them reserved instead), never change a field number, never change a field type. Violate these rules and silent data corruption follows. The buf lint tool enforces these rules automatically in CI.
When to Choose gRPC Over REST
gRPC adds real value in specific scenarios. Internal service-to-service communication: both sides are under your control, both support gRPC, and you need the performance of binary serialization and HTTP/2 multiplexing. Streaming workloads: you need to stream data from client to server, server to client, or bidirectionally. The four gRPC streaming modes (unary, server streaming, client streaming, bidirectional) have no clean REST equivalent. Strongly typed interfaces: when a polyglot microservices environment (Go services calling Python services calling Java services) needs compile-time contract enforcement across languages.
REST is the right choice when: you are designing a public API that external developers will consume (gRPC requires code generation and HTTP/2, which adds friction); your clients are browsers (gRPC is only supported via gRPC-Web, a proxy-based adaptation layer); or your team is not familiar with Protocol Buffers. Mixing gRPC internally and REST externally is common and correct. The API gateway (or Envoy proxy) handles the transcoding.
Protocol Buffer Schema Evolution Rules
Protocol Buffers handle schema evolution through field numbers. Each field has a number that is stable across schema versions. Adding a new field with a new number is always backward compatible: old clients ignore unknown field numbers. Removing a field is backward compatible if you reserve the field number (to prevent reuse). Changing a field's type is not backward compatible and requires a field number change. Renaming a field is fine because field names are not used in the binary encoding.
The evolution rules make protocol buffers safer than JSON for long-lived data. A JSON payload from 2019 stored in a data lake cannot be safely deserialized if the schema changed in 2021 without checking which version of the schema to apply. A protobuf payload carries its field numbers in the binary encoding. As long as you followed the evolution rules, any version of the reader can deserialize any version of the payload.
- Never reuse a field number in a .proto file. Protobuf uses numbers, not names, for binary encoding. Reusing a number causes data corruption.
- Use reserved field numbers and names when removing a field: reserved 5; reserved 'old_field_name';
- Proto3 removed required fields (a source of compatibility bugs in proto2). All fields are optional by default. Use proto3 for new schemas.
- gRPC status codes are separate from HTTP status codes. gRPC has 16 canonical status codes (OK, CANCELLED, UNKNOWN, INVALID_ARGUMENT, etc.). These are mapped to HTTP status codes by grpc-gateway and Envoy.
- Deadlines propagate across gRPC calls. If a client sets a 100ms deadline and the first service takes 30ms, the downstream call automatically has a 70ms deadline. This prevents cascading timeouts.
gRPC in the Service Mesh Era
gRPC integrates naturally with service mesh infrastructure because both operate at the same layer (L7 application protocol). Envoy, Istio, and Linkerd all have native gRPC support: they can load balance gRPC traffic, collect per-RPC telemetry, enforce per-method authorization policies, and apply traffic management rules (canary deployments, circuit breaking, retry policies) at the individual RPC level. This is more granular than REST-level traffic management, which typically operates per-URL.
The gRPC health checking protocol is standardized (grpc.health.v1.Health service) and supported natively by Kubernetes liveness and readiness probes. Load balancers that understand gRPC (as opposed to treating it as opaque TCP) can make per-RPC routing decisions rather than per-connection decisions - critical for server-streaming and bidirectional streaming RPCs that maintain long-lived connections. Switching from a TCP load balancer to an HTTP/2-aware load balancer for gRPC workloads can significantly improve load distribution.
Protocol Buffers code generation also creates a tight coupling between service version and client version. A message field that exists in the server but not in the client proto definition is silently dropped on deserialization. A message field that exists in the client proto but not in the server is ignored by the server. This backward/forward compatibility is by design and allows you to evolve services independently - but only if you follow the rules: never remove fields (mark them reserved instead), never change a field number, never change a field type. Violate these rules and silent data corruption follows. The buf lint tool enforces these rules automatically in CI.