The Papers That Defined How You Operate Systems
Most system design writing focuses on how to build systems. These two works focused on how to operate them. James Hamilton's 2007 paper and Google's 2016 SRE book together define the operational excellence principles that separate systems that look good in architecture diagrams from systems that actually stay up.
On Designing Internet-Scale Services (2007)
James Hamilton wrote one of the most practically useful papers in distributed systems. Not about novel algorithms or theoretical guarantees. About operations. The document reads like a checklist forged in production fires: rules derived from real outages, real 3 AM pages, real customer impact. Hamilton was drawing from his experience operating services at Microsoft scale (Exchange, SQL Server) before moving to Amazon.
- Design for failure: expect hardware failures, network issues, and software bugs. Build self-healing systems that detect and recover automatically.
- Redundancy: single points of failure are technical debt. Audit every dependency for SPOF status.
- Automate everything: manual processes do not scale and introduce human error. If you had to do it manually this week, automate it before next week.
- Zero-downtime deployments: if deploying requires downtime, you will deploy less often. Deploy less often and bugs accumulate. Design for rolling updates from day one.
- Test your backups: an untested backup is not a backup. Regular restore drills surface problems before you need them in production.
- Ship the exact binary you test: if QA tested build 1.2.3, production must run build 1.2.3. Environment-specific builds introduce bugs that only appear in production.
- Simplicity: complex systems are harder to debug at 3 AM. Optimize for debuggability. If an on-call engineer cannot understand the system in 5 minutes, it is too complex.
Operational excellence is not glamorous. But it is the difference between a system that works in demos and one that works at scale. Hamilton's checklist is more valuable than most architectural patterns.
Google's SRE Book (2016)
Google published their internal playbook for keeping the world's largest systems running. Site Reliability Engineering introduced vocabulary and frameworks that became industry standard almost immediately. The key contribution was not any single technique but a philosophy: reliability is an engineering problem that can be measured, tracked, and systematically improved.
- SLIs (Service Level Indicators): the actual measurement. Request success rate, latency percentile, availability percentage.
- SLOs (Service Level Objectives): the target you commit to internally. 99.9% success rate. P99 latency under 200ms.
- SLAs (Service Level Agreements): the external commitment. Legal and commercial. If you miss this, you pay penalties.
- Error budgets: if your SLO is 99.9% availability, you have 43.8 minutes per month of 'allowed' downtime. This budget is shared between incidents and deployments. Spend it wisely.
- Toil: manual, repetitive, automatable work that scales linearly with service growth. Every hour of toil is an hour not spent improving the system. SRE teams target less than 50% toil.
- The on-call runbook: every alert that fires must have a documented response procedure. If it does not, write one. If no one can write one, the alert should not exist.
Error Budgets Change Everything
The error budget concept is the SRE book's most impactful contribution. It shifts the conversation from 'we must have zero downtime' (impossible and paralyzing) to 'we have a budget for failure, and we choose how to spend it.' This has two effects. Development teams can deploy more aggressively as long as the error budget is healthy. Operations teams have an objective measure to pause deployments when the budget is spent. The conversation changes from adversarial ('dev wants to ship, ops wants stability') to collaborative ('we have 20 minutes of budget left this month, how do we spend it wisely?').
Further Reading
- Hamilton (2007): 'On Designing and Deploying Internet-Scale Services', freely available online
- Beyer et al. (2016): 'Site Reliability Engineering', free at sre.google/sre-book
- Beyer et al. (2018): 'The Site Reliability Workbook', practical examples of SRE implementation
- Alex Hidalgo's 'Implementing Service Level Objectives', the most practical SLO book available
Running Blameless Postmortems: How Learning Replaces Punishment
A postmortem that assigns blame teaches the organization one lesson: hide failures. Engineers learn to avoid being associated with incidents rather than to build better systems. Blameless postmortems, popularized by Google SRE and borrowed from aviation safety culture, treat each incident as a system failure rather than an individual failure. The premise: engineers made the decisions that seemed reasonable given the information they had at the time. The system allowed them to make those decisions. Fix the system. The postmortem format documents: what happened (timeline with specific timestamps), contributing factors (not 'root causes,' because complex systems rarely have single root causes), what went well (the mitigations that limited impact), and action items (specific, assigned, time-bounded improvements).
- Five whys technique: ask 'why?' repeatedly to trace a symptom back to contributing factors. Stop when you reach something actionable.
- Contributing factors not root causes: complex systems fail due to multiple interacting factors. A single 'root cause' framing misses systemic issues.
- Action items must be SMART: Specific, Measurable, Assignable, Realistic, Time-bounded. 'Improve monitoring' is not an action item.
- Postmortem review meeting: review with the team within 5 business days while memory is fresh. Include everyone involved.
- Postmortem database: store all postmortems in a searchable internal knowledge base. Same failure patterns recurring across postmortems indicate systemic issues.
On-Call Rotation Design: Making Being On-Call Sustainable
Unsustainable on-call rotations are a reliability risk, not just a people problem. Engineers who are sleep-deprived from overnight pages make mistakes during the day that create the incidents that page them at night. Google's SRE model targets a maximum of 2 incidents per on-call shift and no more than 50% of an engineer's time spent on toil (reactive work). If your on-call rotation exceeds this, you are spending more on fighting fires than preventing them, and your team will burn out and leave.
- Rotation size: at minimum 4 engineers per rotation for 24/7 coverage. 8+ is sustainable long-term with reasonable on-call frequency.
- Follow-the-sun: hand off on-call responsibility between time zones so nobody is paged between midnight and 8 AM local time.
- Alert quality: every alert that fires must be actionable. Alerts that require no action are noise that trains engineers to ignore alerts.
- Escalation policy: define primary, secondary, and management escalation paths. Automated escalation if primary does not acknowledge within 10 minutes.
- On-call compensation: differentiate between 'on-call but not paged' (low burden, lower compensation) and 'paged during sleep hours' (high burden, requires direct compensation or time off in lieu).
What Interviewers Test About Reliability and SRE
SRE questions in senior interviews test whether you think about reliability as an engineering discipline with metrics and trade-offs, not as a vague aspiration. The most impressive answers quantify reliability requirements (99.9% vs 99.99% SLO) and reason about what infrastructure and practices each level requires.
- Know: the math behind SLO percentages. 99.9% = 8.7 hours/year downtime. 99.99% = 52 minutes/year. 99.999% = 5 minutes/year.
- Know: how error budgets change team behavior (development teams deploy aggressively when budget is healthy, cautiously when budget is spent)
- Know: the four golden signals (latency, traffic, errors, saturation) and what each measures
- Know: the difference between MTTR (mean time to recovery) and MTBF (mean time between failures), and which one SRE practice focuses on improving
- Red flag: claiming you can achieve five nines reliability without discussing the infrastructure and organizational investment required
Running Blameless Postmortems: How Learning Replaces Punishment
A postmortem that assigns blame teaches the organization one lesson: hide failures. Engineers learn to avoid being associated with incidents rather than to build better systems. Blameless postmortems, popularized by Google SRE and borrowed from aviation safety culture, treat each incident as a system failure rather than an individual failure. The premise: engineers made the decisions that seemed reasonable given the information they had at the time. The system allowed them to make those decisions. Fix the system. The postmortem format documents: what happened (timeline with specific timestamps), contributing factors (not 'root causes,' because complex systems rarely have single root causes), what went well (the mitigations that limited impact), and action items (specific, assigned, time-bounded improvements).
- Five whys technique: ask 'why?' repeatedly to trace a symptom back to contributing factors. Stop when you reach something actionable.
- Contributing factors not root causes: complex systems fail due to multiple interacting factors. A single 'root cause' framing misses systemic issues.
- Action items must be SMART: Specific, Measurable, Assignable, Realistic, Time-bounded. 'Improve monitoring' is not an action item.
- Postmortem review meeting: review with the team within 5 business days while memory is fresh. Include everyone involved.
- Postmortem database: store all postmortems in a searchable internal knowledge base. Same failure patterns recurring across postmortems indicate systemic issues.
On-Call Rotation Design: Making Being On-Call Sustainable
Unsustainable on-call rotations are a reliability risk, not just a people problem. Engineers who are sleep-deprived from overnight pages make mistakes during the day that create the incidents that page them at night. Google's SRE model targets a maximum of 2 incidents per on-call shift and no more than 50% of an engineer's time spent on toil (reactive work). If your on-call rotation exceeds this, you are spending more on fighting fires than preventing them, and your team will burn out and leave.
- Rotation size: at minimum 4 engineers per rotation for 24/7 coverage. 8+ is sustainable long-term with reasonable on-call frequency.
- Follow-the-sun: hand off on-call responsibility between time zones so nobody is paged between midnight and 8 AM local time.
- Alert quality: every alert that fires must be actionable. Alerts that require no action are noise that trains engineers to ignore alerts.
- Escalation policy: define primary, secondary, and management escalation paths. Automated escalation if primary does not acknowledge within 10 minutes.
- On-call compensation: differentiate between 'on-call but not paged' (low burden, lower compensation) and 'paged during sleep hours' (high burden, requires direct compensation or time off in lieu).
What Interviewers Test About Reliability and SRE
SRE questions in senior interviews test whether you think about reliability as an engineering discipline with metrics and trade-offs, not as a vague aspiration. The most impressive answers quantify reliability requirements (99.9% vs 99.99% SLO) and reason about what infrastructure and practices each level requires.
- Know: the math behind SLO percentages. 99.9% = 8.7 hours/year downtime. 99.99% = 52 minutes/year. 99.999% = 5 minutes/year.
- Know: how error budgets change team behavior (development teams deploy aggressively when budget is healthy, cautiously when budget is spent)
- Know: the four golden signals (latency, traffic, errors, saturation) and what each measures
- Know: the difference between MTTR (mean time to recovery) and MTBF (mean time between failures), and which one SRE practice focuses on improving
- Red flag: claiming you can achieve five nines reliability without discussing the infrastructure and organizational investment required