Testing Failure Before It Tests You
Most companies test whether their systems work. Netflix decided to test whether their systems break gracefully. In 2012, they introduced Chaos Monkey: a tool that randomly kills production service instances during business hours. The logic was counterintuitive but compelling. If production failures are inevitable, better to face them on your terms during a business day when engineers are available than at 3 AM on Black Friday.
Chaos Monkey was the start of a discipline called chaos engineering. Chaos Gorilla takes down entire AWS availability zones. Chaos Kong simulates a full regional outage. Latency Monkey introduces artificial delays. The Simian Army (Netflix's full suite) is designed to surface weaknesses before they become crises. The key insight: resilience cannot be proven by architecture diagrams. It can only be proven by testing under real failure conditions.
Chaos engineering is not about breaking things. It is about building confidence that your system handles failure gracefully. The difference between discovering a failure mode in a chaos experiment and discovering it during a customer incident is the difference between a postmortem and a disaster.
Docker: Solving the 'Works on My Machine' Problem (2013)
For decades, software deployment meant reproducing the exact runtime environment your application was developed on. Different OS versions, different library versions, different environment variables. The deployment checklist for a production release was measured in pages. 'Works on my machine' became an industry joke because it was everyone's actual experience.
Docker packaged an application with its entire runtime environment into a container: a lightweight, portable unit that runs identically everywhere. Unlike virtual machines that simulate an entire operating system (gigabytes, minutes to start), containers share the host OS kernel and start in seconds. A Dockerfile replaced pages of deployment documentation. Docker Hub became the app store for server software. The 2013 Docker open-source release triggered a complete rethinking of how software gets built, shipped, and run.
- Immutability: containers are built from an image and never modified. Debug issues by examining the image, not hunting through a live environment.
- Portability: the same container runs on a developer laptop, a CI server, and production. No more environment drift.
- Isolation: containers share the kernel but have isolated filesystems, networking, and process spaces.
- Layering: Docker images are built in layers. Change one layer and only that layer rebuilds. Shared base layers are cached.
Kubernetes: Orchestrating Containers at Scale (2014)
Docker solved packaging. The next problem: how do you run hundreds of containers across a cluster of machines? How do you handle container failures, node failures, scaling, and rolling deployments? Manual container management at scale is a full-time job. Google had been solving this internally for 15 years with a system called Borg that managed billions of containers across their global infrastructure.
Kubernetes is Borg's open-source descendant. Its core value is declarative infrastructure: you describe what you want (3 replicas of this service, each with 2GB memory, accessible on port 8080), and Kubernetes continuously works to make that reality true. If a pod crashes, it is restarted. If a node fails, pods are rescheduled. If traffic spikes, horizontal pod autoscaling adds more replicas automatically.
- Pods: the smallest deployable unit. One or more containers that share network and storage.
- Deployments: manage pod lifecycle. Define desired state; Kubernetes maintains it. Rolling updates replace pods incrementally.
- Services: stable network endpoints for a set of pods. The service IP stays constant even as pods come and go.
- ConfigMaps and Secrets: inject configuration and credentials into pods without baking them into the image.
- Horizontal Pod Autoscaler: scale pod count based on CPU, memory, or custom metrics.
- Ingress: route external HTTP/HTTPS traffic to services based on host and path rules.
The Intersection: Chaos Engineering Meets Containers
Containers and Kubernetes changed chaos engineering. Before containers, killing a process meant real downtime. With Kubernetes, a pod kill triggers automatic rescheduling in seconds. Chaos engineering became safer because the recovery mechanisms are automated. Tools like LitmusChaos and Gremlin run chaos experiments natively against Kubernetes clusters. The combination means you can test failure handling continuously in lower environments and build genuine confidence before production issues arise.
Further Reading
- Basiri et al. (2016): 'Chaos Engineering' by the Netflix team, IEEE Software
- Docker documentation: docs.docker.com
- Kubernetes documentation: kubernetes.io/docs, especially the concepts section
- Brendan Burns et al. (2016): 'Borg, Omega, and Kubernetes', ACM Queue, for the lineage from Google's internal systems
Kubernetes Networking: What Actually Happens When a Pod Makes a Request
Kubernetes networking has four layers, each with its own addressing. Pods get unique IPs within the cluster, assigned by the Container Network Interface (CNI) plugin (Calico, Flannel, Cilium). Containers within a pod share a network namespace and communicate on localhost. Pod-to-pod communication within the cluster is handled by the CNI plugin's overlay network or BGP routing. Services provide stable virtual IPs (ClusterIPs) that never change even as pods come and go. kube-proxy (or eBPF in Cilium) maintains iptables/IPVS rules to load-balance traffic from the ClusterIP to healthy pod IPs. External traffic enters via LoadBalancer services (backed by a cloud load balancer) or Ingress controllers (nginx, Traefik, ALB Ingress).
- ClusterIP: stable virtual IP for internal cluster communication. Not accessible from outside.
- NodePort: exposes a service on each node's IP at a fixed port. Accessible externally but requires knowing node IPs.
- LoadBalancer: provisions a cloud load balancer (AWS ELB, GCP LB) that routes to NodePort. Standard for production external traffic.
- Ingress: HTTP/HTTPS routing rules that route by host/path to ClusterIP services. More flexible and cost-effective than one LoadBalancer per service.
- Network policies: Kubernetes firewall rules. By default, all pod-to-pod traffic is allowed. NetworkPolicy resources restrict this to least-privilege.
Running a Chaos Game Day: From Theory to Practice
Chaos experiments are only valuable if they are systematic and learning is captured. A chaos game day follows a structured process. First, form a hypothesis: 'If we terminate 50% of the checkout service pods, checkout latency will stay below 500ms within 30 seconds as Kubernetes reschedules.' Then define the blast radius: which environment? (never start in production), which services?, what rollback procedure? Run the experiment, observe metrics in real time, and immediately stop if pre-defined thresholds are breached. Document findings, prioritize gaps, and fix them before the next game day.
- Steady-state hypothesis: define the normal behavior metrics before running any experiment.
- Minimal blast radius: start with a single pod kill in staging. Graduate to multi-pod, multi-AZ, and production only after proving resilience.
- Monitoring prerequisites: you cannot run chaos experiments if you cannot observe the system's response. Metrics and dashboards come first.
- Common experiments: pod termination, CPU throttling, memory pressure, network latency injection, dependency unavailability, DNS failure.
- LitmusChaos and Chaos Mesh: Kubernetes-native chaos frameworks that define experiments as CRDs and integrate with CI/CD pipelines.
What Interviewers Test About Containers and Orchestration
Kubernetes questions test whether you understand the operational model, not just the YAML syntax. Interviewers want to hear about resource requests vs limits, horizontal pod autoscaling triggers, and how you handle stateful workloads in a stateless-first orchestrator.
- Know: the difference between resource requests (guaranteed) and limits (maximum). Misconfigured limits cause OOM kills.
- Know: liveness vs readiness vs startup probes and what happens when each fails
- Know: how Horizontal Pod Autoscaler works (CPU/memory metrics, custom metrics) and its lag characteristics
- Know: why StatefulSets exist (stable pod names, stable storage) and when to use them vs Deployments
- Know: rolling update strategy parameters (maxSurge, maxUnavailable) and how to configure zero-downtime deployments
Kubernetes Networking: What Actually Happens When a Pod Makes a Request
Kubernetes networking has four layers, each with its own addressing. Pods get unique IPs within the cluster, assigned by the Container Network Interface (CNI) plugin (Calico, Flannel, Cilium). Containers within a pod share a network namespace and communicate on localhost. Pod-to-pod communication within the cluster is handled by the CNI plugin's overlay network or BGP routing. Services provide stable virtual IPs (ClusterIPs) that never change even as pods come and go. kube-proxy (or eBPF in Cilium) maintains iptables/IPVS rules to load-balance traffic from the ClusterIP to healthy pod IPs. External traffic enters via LoadBalancer services (backed by a cloud load balancer) or Ingress controllers (nginx, Traefik, ALB Ingress).
- ClusterIP: stable virtual IP for internal cluster communication. Not accessible from outside.
- NodePort: exposes a service on each node's IP at a fixed port. Accessible externally but requires knowing node IPs.
- LoadBalancer: provisions a cloud load balancer (AWS ELB, GCP LB) that routes to NodePort. Standard for production external traffic.
- Ingress: HTTP/HTTPS routing rules that route by host/path to ClusterIP services. More flexible and cost-effective than one LoadBalancer per service.
- Network policies: Kubernetes firewall rules. By default, all pod-to-pod traffic is allowed. NetworkPolicy resources restrict this to least-privilege.
Running a Chaos Game Day: From Theory to Practice
Chaos experiments are only valuable if they are systematic and learning is captured. A chaos game day follows a structured process. First, form a hypothesis: 'If we terminate 50% of the checkout service pods, checkout latency will stay below 500ms within 30 seconds as Kubernetes reschedules.' Then define the blast radius: which environment? (never start in production), which services?, what rollback procedure? Run the experiment, observe metrics in real time, and immediately stop if pre-defined thresholds are breached. Document findings, prioritize gaps, and fix them before the next game day.
- Steady-state hypothesis: define the normal behavior metrics before running any experiment.
- Minimal blast radius: start with a single pod kill in staging. Graduate to multi-pod, multi-AZ, and production only after proving resilience.
- Monitoring prerequisites: you cannot run chaos experiments if you cannot observe the system's response. Metrics and dashboards come first.
- Common experiments: pod termination, CPU throttling, memory pressure, network latency injection, dependency unavailability, DNS failure.
- LitmusChaos and Chaos Mesh: Kubernetes-native chaos frameworks that define experiments as CRDs and integrate with CI/CD pipelines.
What Interviewers Test About Containers and Orchestration
Kubernetes questions test whether you understand the operational model, not just the YAML syntax. Interviewers want to hear about resource requests vs limits, horizontal pod autoscaling triggers, and how you handle stateful workloads in a stateless-first orchestrator.
- Know: the difference between resource requests (guaranteed) and limits (maximum). Misconfigured limits cause OOM kills.
- Know: liveness vs readiness vs startup probes and what happens when each fails
- Know: how Horizontal Pod Autoscaler works (CPU/memory metrics, custom metrics) and its lag characteristics
- Know: why StatefulSets exist (stable pod names, stable storage) and when to use them vs Deployments
- Know: rolling update strategy parameters (maxSurge, maxUnavailable) and how to configure zero-downtime deployments