Three Days of Downtime That Changed an Industry
On August 25, 2008, Netflix's monolithic Oracle database went down. Not for minutes. Not for hours. For three days. No DVDs shipped. No website. The postmortem was brutal: a single corrupted database table took down the entire company because every component depended on the same database. Netflix decided that this could never happen again.
They spent the next seven years migrating from a monolithic Java application running in their own data centers to hundreds of microservices running on AWS. Every piece of the platform was decomposed: the API gateway (Zuul), service discovery (Eureka), circuit breakers (Hystrix), client-side load balancing (Ribbon), configuration management (Archaius), and more. They open-sourced nearly all of it. The Netflix OSS stack became the reference implementation for microservices architecture and is still widely used today.
Netflix did not adopt microservices because it was the fashionable architecture. They adopted it because a monolith nearly killed the company. The best architecture decisions are always driven by specific failures, not abstract principles.
What Netflix OSS Gave the Industry
Each Netflix OSS component solved a specific problem that every microservices architecture faces:
- Eureka (service discovery): services register themselves on startup. Clients look up service addresses dynamically instead of hard-coding IPs. Essential when services scale up and down constantly.
- Zuul (API gateway): a single entry point that routes requests to the appropriate service, handles authentication, rate limiting, and request transformation.
- Hystrix (circuit breakers): stops cascading failures by monitoring downstream calls and short-circuiting requests to failing services before they bring down the caller.
- Ribbon (client-side load balancing): distributes requests across service instances at the client level rather than relying on a centralized load balancer. Works with Eureka's registry to know which instances are healthy.
- Archaius (configuration management): centralized, dynamic configuration that services pull at runtime. Change a feature flag without redeploying.
The Twelve-Factor App (2011)
Adam Wiggins at Heroku distilled years of observing how developers misused cloud platforms into twelve principles for building cloud-native applications. Each factor addresses a specific operational failure mode that Heroku's platform team saw repeatedly. They seem obvious in hindsight. In 2011, most applications violated half of them.
- Codebase: one codebase per service, tracked in version control. Not one big repo for everything.
- Dependencies: explicitly declare all dependencies. Never rely on system-installed packages.
- Config: store configuration in environment variables, not in code. Config changes between environments; code should not.
- Backing services: treat databases, caches, and message queues as attached resources. Swap them without code changes.
- Build/release/run: strictly separate the build stage (create artifact), release stage (combine with config), and run stage (execute). Never change code at runtime.
- Processes: execute the app as one or more stateless processes. Session data belongs in a backing service (Redis), not process memory.
- Port binding: services export their interface via port binding. No app server magic. Just 'listen on port $PORT'.
- Concurrency: scale out via additional processes, not threads within a single process.
- Disposability: fast startup (seconds) and graceful shutdown. Embrace process death as normal.
- Dev/prod parity: keep development and production as similar as possible. Container images eliminate most of this problem.
- Logs: treat logs as event streams. Write to stdout. Let the platform aggregate and ship them.
- Admin processes: run admin tasks (migrations, data cleanup) as one-off processes against the production release.
Why These Principles Still Matter
The Twelve-Factor methodology became the philosophical foundation for Docker (factor III and XI), Kubernetes (factors VI, VIII, IX), and modern CI/CD (factor V). When an interviewer asks 'where does your config live?' or 'how do you handle sessions across multiple instances?' or 'how does your service scale?' they are testing whether your design respects these principles. A service that stores session state in process memory cannot scale horizontally. A service with hardcoded database credentials cannot be deployed to different environments. Twelve-Factor violations do not show up in demos. They show up in production at the worst possible moment.
Further Reading
- 12factor.net, the original document by Adam Wiggins
- Netflix Tech Blog (netflixtechblog.com) for the detailed engineering stories behind each OSS component
- Adrian Cockcroft's talks on Netflix's microservices architecture, particularly 'Migrating to Microservices'
Service Discovery Deep Dive: How Services Find Each Other
In a monolith, a function call is a local operation. In microservices, every call is a network request to a service that could be running on any of dozens of instances, all dynamically assigned IP addresses by Kubernetes or AWS. Hard-coding service addresses is obviously wrong. DNS-based discovery works for stable services but cannot handle the rapid instance churn of auto-scaling environments. Service registries (Eureka, Consul) solve this: services register themselves on startup with their current IP and port, send heartbeats every 30 seconds, and deregister on shutdown. Clients query the registry to discover instances.
The discovery approach determines where load balancing happens. Client-side load balancing (Netflix Ribbon): the client queries the registry, gets a list of instances, and picks one using a local load balancing algorithm (round-robin, least-connections, zone-aware). No central load balancer bottleneck. Server-side load balancing (traditional reverse proxy): the client calls a stable virtual IP, a central load balancer distributes to instances, the client has no awareness of the fleet. Kubernetes Services use this model with kube-proxy handling the virtual IP routing.
- Eureka (Netflix OSS): AP system. During partitions, stale registry data is served rather than refusing lookups. Acceptable for most service discovery.
- Consul: CP system. Strong consistency using Raft consensus. Also provides KV store, health checking, and service mesh capabilities.
- Kubernetes DNS: services get stable DNS names (my-service.namespace.svc.cluster.local). Kubernetes handles instance routing transparently.
- Health checks: services expose /health endpoints. Registries and load balancers probe these and remove unhealthy instances automatically.
- Graceful shutdown: services deregister before stopping, preventing a window where the registry still routes to a stopping instance.
The Strangler Fig Pattern: How to Decompose a Monolith
Netflix did not rewrite their monolith from scratch. A full rewrite is the highest-risk approach (you are rewriting working software while the business continues). The strangler fig pattern incrementally decomposes a monolith: identify a bounded context (authentication, search, recommendations), build it as a standalone service, and route traffic for that feature to the new service. The monolith still handles everything else. Over time, more features migrate to services. The monolith shrinks until it handles nothing and can be retired.
- Step 1: Identify a bounded context with clear data ownership and a high-traffic or high-pain profile.
- Step 2: Create the new service with its own database, completely isolated from the monolith's database.
- Step 3: Deploy the service alongside the monolith. Route a small percentage of traffic to validate correctness.
- Step 4: Gradually shift 100% of traffic for that feature to the service. Maintain the monolith code for rollback.
- Step 5: Remove the feature code from the monolith once the service is stable.
- Anti-pattern: sharing the database between the service and the monolith. This is not a service extraction, it is a distributed monolith.
What Interviewers Test About Microservices
Microservices questions have two failure modes. One: treating every system as if it should be microservices (the interviewer wants to hear when NOT to use microservices). Two: knowing the patterns (circuit breakers, service discovery) by name without understanding what problem they solve. The best answers demonstrate that microservices add operational complexity and are only worth it when the benefits (independent scaling, independent deployment, team autonomy) outweigh that cost.
- Know: the two-pizza team rule and why microservices boundaries should follow organizational boundaries (Conway's Law)
- Know: why distributed transactions are hard and what SAGA pattern (choreography vs orchestration) replaces 2PC with
- Know: the difference between synchronous (REST/gRPC) and asynchronous (event-driven) inter-service communication and when each applies
- Know: how to handle shared data between services (publish events on change, accept eventual consistency for reads)
- Red flag: proposing microservices for a startup or small team without acknowledging the operational overhead
Service Discovery Deep Dive: How Services Find Each Other
In a monolith, a function call is a local operation. In microservices, every call is a network request to a service that could be running on any of dozens of instances, all dynamically assigned IP addresses by Kubernetes or AWS. Hard-coding service addresses is obviously wrong. DNS-based discovery works for stable services but cannot handle the rapid instance churn of auto-scaling environments. Service registries (Eureka, Consul) solve this: services register themselves on startup with their current IP and port, send heartbeats every 30 seconds, and deregister on shutdown. Clients query the registry to discover instances.
The discovery approach determines where load balancing happens. Client-side load balancing (Netflix Ribbon): the client queries the registry, gets a list of instances, and picks one using a local load balancing algorithm (round-robin, least-connections, zone-aware). No central load balancer bottleneck. Server-side load balancing (traditional reverse proxy): the client calls a stable virtual IP, a central load balancer distributes to instances, the client has no awareness of the fleet. Kubernetes Services use this model with kube-proxy handling the virtual IP routing.
- Eureka (Netflix OSS): AP system. During partitions, stale registry data is served rather than refusing lookups. Acceptable for most service discovery.
- Consul: CP system. Strong consistency using Raft consensus. Also provides KV store, health checking, and service mesh capabilities.
- Kubernetes DNS: services get stable DNS names (my-service.namespace.svc.cluster.local). Kubernetes handles instance routing transparently.
- Health checks: services expose /health endpoints. Registries and load balancers probe these and remove unhealthy instances automatically.
- Graceful shutdown: services deregister before stopping, preventing a window where the registry still routes to a stopping instance.
The Strangler Fig Pattern: How to Decompose a Monolith
Netflix did not rewrite their monolith from scratch. A full rewrite is the highest-risk approach (you are rewriting working software while the business continues). The strangler fig pattern incrementally decomposes a monolith: identify a bounded context (authentication, search, recommendations), build it as a standalone service, and route traffic for that feature to the new service. The monolith still handles everything else. Over time, more features migrate to services. The monolith shrinks until it handles nothing and can be retired.
- Step 1: Identify a bounded context with clear data ownership and a high-traffic or high-pain profile.
- Step 2: Create the new service with its own database, completely isolated from the monolith's database.
- Step 3: Deploy the service alongside the monolith. Route a small percentage of traffic to validate correctness.
- Step 4: Gradually shift 100% of traffic for that feature to the service. Maintain the monolith code for rollback.
- Step 5: Remove the feature code from the monolith once the service is stable.
- Anti-pattern: sharing the database between the service and the monolith. This is not a service extraction, it is a distributed monolith.
What Interviewers Test About Microservices
Microservices questions have two failure modes. One: treating every system as if it should be microservices (the interviewer wants to hear when NOT to use microservices). Two: knowing the patterns (circuit breakers, service discovery) by name without understanding what problem they solve. The best answers demonstrate that microservices add operational complexity and are only worth it when the benefits (independent scaling, independent deployment, team autonomy) outweigh that cost.
- Know: the two-pizza team rule and why microservices boundaries should follow organizational boundaries (Conway's Law)
- Know: why distributed transactions are hard and what SAGA pattern (choreography vs orchestration) replaces 2PC with
- Know: the difference between synchronous (REST/gRPC) and asynchronous (event-driven) inter-service communication and when each applies
- Know: how to handle shared data between services (publish events on change, accept eventual consistency for reads)
- Red flag: proposing microservices for a startup or small team without acknowledging the operational overhead