SLIs, SLOs & Error Budgets

Everyone agrees a system should be reliable, but how reliable? A vague promise of high availability cannot be measured or traded off against anything else, and it leads to two bad habits: teams that ship nothing because anything could break, or teams that break things without noticing. Site reliability engineering fixes this by making reliability a number, with three terms every designer should know and a budget that follows from them.

Think of an exam. Your actual score, say 87%, is what you measured. You have set yourself a target of 90%. And the school will only let you progress if you score at least 70%. The measured score, your own target and the school's pass mark are three different things, and the gap between the target and the pass mark is what keeps you safe: you aim higher than the line that has real consequences.

• -Your actual score is the SLI (Service Level Indicator): what you measured, such as 99.87% of requests succeeded last month.
• -Your own target is the SLO (Service Level Objective): the internal goal you design and operate toward, such as 99.9% of requests succeed.
• -The school's pass mark is the SLA (Service Level Agreement): an external, often contractual promise with real consequences, such as refunds, if it is missed.
• -The SLO is set stricter than the SLA, so you notice and fix trouble before you owe anyone anything.

A good SLI measures something a user would notice, not something about a machine. CPU at 80% is not an SLI, because users cannot see it. These are:

• -Availability: the share of requests that succeed.
• -Latency: the share of requests faster than a threshold, such as 95% under 300 ms.
• -Correctness or freshness: the share of answers that are right, or data no older than a few minutes.

A complete SLO combines a target with a window, for example: 99.9% of requests succeed, measured over a rolling 30 days.

If the SLO is 99.9%, the system is allowed to fail 0.1% of the time, and that allowance is the error budget. In time, 0.1% of a 30-day month (43,200 minutes) is about 43 minutes of downtime. In requests, if you serve 10 million this month, you may fail 10,000. A 12-minute full outage spends 12 / 43.2, about 28%, of the month's budget in one go.

The budget turns reliability from a feeling into a resource that is spent. While budget remains, the team can ship faster and take risks: risky deployments, experiments, maintenance. When it runs out, new features pause and the team works on reliability until it recovers. This settles the old argument between the people who want to ship and the people who want stability, because both are measured against the same number.

Each extra nine costs roughly ten times more and slows delivery, while users cannot tell the difference: their own phone and network are less reliable than the extra nines would justify. Aiming for 100% also means you could never change anything, since every change carries risk. Choose the lowest SLO that keeps users happy.

Paging someone for every failed request would drown them. Instead, alert on how fast the budget is being used, called the burn rate. If at the current rate the whole month's budget would be gone in two days, page someone now. If it is burning slowly and will last the month, a ticket for the morning is enough. This keeps alerts tied to what users experience and to what you promised.