Running Blameless Postmortems: How Learning Replaces Punishment
Hamilton's Paper and the SRE Book provide important foundations for understanding how to operate systems at scale, especially through practices that turn failures into opportunities for learning. A postmortem that assigns blame teaches the organization one lesson: hide failures. Engineers learn to avoid being associated with incidents rather than to build better systems. Blameless postmortems, popularized by Google SRE and borrowed from aviation safety culture, treat each incident as a system failure rather than an individual failure. The premise: engineers made the decisions that seemed reasonable given the information they had at the time. The system allowed them to make those decisions. Fix the system. The postmortem format documents: what happened (timeline with specific timestamps), contributing factors (not “root causes,” because complex systems rarely have single root causes), what went well (the mitigations that limited impact), and action items (specific, assigned, time-bounded improvements).
Five whys technique: ask “why?” repeatedly to trace a symptom back to contributing factors. Stop when you reach something actionable.
Contributing factors, not root causes: complex systems fail due to multiple interacting factors. A single “root cause” framing misses systemic issues.
Action items must be SMART: Specific, Measurable, Assignable, Realistic, Time-bounded. “Improve monitoring” is not an action item.
Postmortem review meeting: review with the team within 5 business days while memory is fresh. Include everyone involved.
Postmortem database: store all postmortems in a searchable internal knowledge base. Recurring failure patterns indicate systemic issues.
On-Call Rotation Design: Making Being On-Call Sustainable
Unsustainable on-call rotations are a reliability risk, not just a people problem. Engineers who are sleep-deprived from overnight pages make mistakes during the day that create the incidents that page them at night. Google's SRE model targets a maximum of 2 incidents per on-call shift and no more than 50% of an engineer's time spent on toil (reactive work). If your on-call rotation exceeds this, you are spending more on fighting fires than preventing them, and your team will burn out and leave.
Rotation size: at minimum 4 engineers per rotation for 24/7 coverage. 8+ is sustainable long-term with reasonable on-call frequency.
Follow-the-sun: hand off on-call responsibility between time zones so nobody is paged between midnight and 8 AM local time.
Alert quality: every alert that fires must be actionable. Alerts that require no action are noise that trains engineers to ignore alerts.
Escalation policy: define primary, secondary, and management escalation paths. Automated escalation if primary does not acknowledge within 10 minutes.
On-call compensation: differentiate between 'on-call but not paged' (low burden, lower compensation) and 'paged during sleep hours' (high burden, requires direct compensation or time off in lieu).
What Interviewers Test About Reliability and SRE
SRE questions in senior interviews test whether you think about reliability as an engineering discipline with metrics and trade-offs, not as a vague aspiration. The most impressive answers quantify reliability requirements (99.9% vs 99.99% SLO) and reason about what infrastructure and practices each level requires.
Know: the math behind SLO percentages. 99.9% = 8.7 hours/year downtime. 99.99% = 52 minutes/year. 99.999% = 5 minutes/year.
Know: how error budgets change team behavior (development teams deploy aggressively when budget is healthy, cautiously when budget is spent)
Know: the four golden signals (latency, traffic, errors, saturation) and what each measures
Know: the difference between MTTR (mean time to recovery) and MTBF (mean time between failures), and which one SRE practice focuses on improving
Red flag: claiming you can achieve five nines reliability without discussing the infrastructure and organizational investment required
Running Blameless Postmortems: How Learning Replaces Punishment
A postmortem that assigns blame teaches the organization one lesson: hide failures. Engineers learn to avoid being associated with incidents rather than to build better systems. Blameless postmortems, popularized by Google SRE and borrowed from aviation safety culture, treat each incident as a system failure rather than an individual failure. The premise: engineers made the decisions that seemed reasonable given the information they had at the time. The system allowed them to make those decisions. Fix the system. The postmortem format documents: what happened (timeline with specific timestamps), contributing factors (not 'root causes,' because complex systems rarely have single root causes), what went well (the mitigations that limited impact), and action items (specific, assigned, time-bounded improvements).
Five whys technique: ask 'why?' repeatedly to trace a symptom back to contributing factors. Stop when you reach something actionable.
Contributing factors not root causes: complex systems fail due to multiple interacting factors. A single 'root cause' framing misses systemic issues.
Action items must be SMART: Specific, Measurable, Assignable, Realistic, Time-bounded. 'Improve monitoring' is not an action item.
Postmortem review meeting: review with the team within 5 business days while memory is fresh. Include everyone involved.
Postmortem database: store all postmortems in a searchable internal knowledge base. Same failure patterns recurring across postmortems indicate systemic issues.
On-Call Rotation Design: Making Being On-Call Sustainable
Unsustainable on-call rotations are a reliability risk, not just a people problem. Engineers who are sleep-deprived from overnight pages make mistakes during the day that create the incidents that page them at night. Google's SRE model targets a maximum of 2 incidents per on-call shift and no more than 50% of an engineer's time spent on toil (reactive work). If your on-call rotation exceeds this, you are spending more on fighting fires than preventing them, and your team will burn out and leave.
Rotation size: at minimum 4 engineers per rotation for 24/7 coverage. 8+ is sustainable long-term with reasonable on-call frequency.
Follow-the-sun: hand off on-call responsibility between time zones so nobody is paged between midnight and 8 AM local time.
Alert quality: every alert that fires must be actionable. Alerts that require no action are noise that trains engineers to ignore alerts.
Escalation policy: define primary, secondary, and management escalation paths. Automated escalation if primary does not acknowledge within 10 minutes.
On-call compensation: differentiate between 'on-call but not paged' (low burden, lower compensation) and 'paged during sleep hours' (high burden, requires direct compensation or time off in lieu).
What Interviewers Test About Reliability and SRE
SRE questions in senior interviews test whether you think about reliability as an engineering discipline with metrics and trade-offs, not as a vague aspiration. The most impressive answers quantify reliability requirements (99.9% vs 99.99% SLO) and reason about what infrastructure and practices each level requires.
Know: the math behind SLO percentages. 99.9% = 8.7 hours/year downtime. 99.99% = 52 minutes/year. 99.999% = 5 minutes/year.
Know: how error budgets change team behavior (development teams deploy aggressively when budget is healthy, cautiously when budget is spent)
Know: the four golden signals (latency, traffic, errors, saturation) and what each measures
Know: the difference between MTTR (mean time to recovery) and MTBF (mean time between failures), and which one SRE practice focuses on improving
Red flag: claiming you can achieve five nines reliability without discussing the infrastructure and organizational investment required
Running Blameless Postmortems: How Learning Replaces Punishment
A postmortem that assigns blame teaches the organization one lesson: hide failures. Engineers learn to avoid being associated with incidents rather than to build better systems. Blameless postmortems, popularized by Google SRE and borrowed from aviation safety culture, treat each incident as a system failure rather than an individual failure. The premise: engineers made the decisions that seemed reasonable given the information they had at the time. The system allowed them to make those decisions. Fix the system. The postmortem format documents: what happened (timeline with specific timestamps), contributing factors (not 'root causes,' because complex systems rarely have single root causes), what went well (the mitigations that limited impact), and action items (specific, assigned, time-bounded improvements).
- Five whys technique: ask 'why?' repeatedly to trace a symptom back to contributing factors. Stop when you reach something actionable.
- Contributing factors not root causes: complex systems fail due to multiple interacting factors. A single 'root cause' framing misses systemic issues.
- Action items must be SMART: Specific, Measurable, Assignable, Realistic, Time-bounded. 'Improve monitoring' is not an action item.
- Postmortem review meeting: review with the team within 5 business days while memory is fresh. Include everyone involved.
- Postmortem database: store all postmortems in a searchable internal knowledge base. Same failure patterns recurring across postmortems indicate systemic issues.
On-Call Rotation Design: Making Being On-Call Sustainable
Unsustainable on-call rotations are a reliability risk, not just a people problem. Engineers who are sleep-deprived from overnight pages make mistakes during the day that create the incidents that page them at night. Google's SRE model targets a maximum of 2 incidents per on-call shift and no more than 50% of an engineer's time spent on toil (reactive work). If your on-call rotation exceeds this, you are spending more on fighting fires than preventing them, and your team will burn out and leave.
- Rotation size: at minimum 4 engineers per rotation for 24/7 coverage. 8+ is sustainable long-term with reasonable on-call frequency.
- Follow-the-sun: hand off on-call responsibility between time zones so nobody is paged between midnight and 8 AM local time.
- Alert quality: every alert that fires must be actionable. Alerts that require no action are noise that trains engineers to ignore alerts.
- Escalation policy: define primary, secondary, and management escalation paths. Automated escalation if primary does not acknowledge within 10 minutes.
- On-call compensation: differentiate between 'on-call but not paged' (low burden, lower compensation) and 'paged during sleep hours' (high burden, requires direct compensation or time off in lieu).
What Interviewers Test About Reliability and SRE
SRE questions in senior interviews test whether you think about reliability as an engineering discipline with metrics and trade-offs, not as a vague aspiration. The most impressive answers quantify reliability requirements (99.9% vs 99.99% SLO) and reason about what infrastructure and practices each level requires.
- Know: the math behind SLO percentages. 99.9% = 8.7 hours/year downtime. 99.99% = 52 minutes/year. 99.999% = 5 minutes/year.
- Know: how error budgets change team behavior (development teams deploy aggressively when budget is healthy, cautiously when budget is spent)
- Know: the four golden signals (latency, traffic, errors, saturation) and what each measures
- Know: the difference between MTTR (mean time to recovery) and MTBF (mean time between failures), and which one SRE practice focuses on improving
- Red flag: claiming you can achieve five nines reliability without discussing the infrastructure and organizational investment required