The Team That Ignored the Real Alert
A team at a major fintech company learned the hard way why alerting strategies matter: they were getting more than 200 alerts a week. Most of them were noise: CPU spikes that resolved on their own, disk usage warnings on volumes that auto-scaled anyway, brief network blips nobody actually needed to touch. Engineers did what engineers do when a system cries wolf often enough, they started ignoring it. Then a real payment processing failure triggered an alert that sat unacknowledged for 47 minutes because it looked exactly like every other false alarm that week. The result was $2.3 million in failed transactions.
Alert fatigue isn't just an annoyance, it's a safety hazard. It's also one of the most common gaps in monitoring and alerting setups that otherwise look well built on paper.
Alert on Symptoms, Not Causes
The natural instinct when setting up monitoring is to alert on anything that looks abnormal: CPU, memory, disk, thread pool utilization. It feels thorough. In practice, it's the fastest way to train an on-call rotation to stop caring.
BAD Alerts (causes - internal metrics):
'CPU > 80% for 5 min' -> Maybe it is a batch job. Fine.
'Memory > 90%' -> Maybe it is cache warming. Fine.
'Disk > 85%' -> Maybe it auto-scales. Fine.
'Thread pool > 80% utilized' -> Maybe it is peak hour. Fine.
These page you for things that MIGHT be problems.
200 alerts/week. 190 are false alarms. Alert fatigue.
GOOD Alerts (symptoms - user-facing impact):
'Error rate > 1% for 5 min' -> Users seeing errors. REAL.
'p99 latency > 2s for 5 min' -> Users waiting too long. REAL.
'Checkout success rate < 98%' -> Revenue impact. REAL.
'Payment processing time > 10s' -> Users abandoning. REAL.
These page you ONLY when users are actually affected.
5-10 alerts/week. All actionable. Trust maintained.The fix is to alert on what the user actually experiences, not on the internal state of a box that may or may not matter. A CPU spike during a nightly batch job is expected behavior. A drop in checkout success rate is not. If you've read our piece on caching strategies and invalidation trade-offs, you've already seen this pattern up close: memory climbing while a cache warms up looks alarming on a dashboard but means nothing to a customer trying to check out. Good alerting strategies draw a hard line between "this metric moved" and "this is actually hurting someone."
SLO-Based Alerting: Burn Rate
Once you've committed to symptom-based alerts, the next question is how sensitive they should be. This is where SLOs and error budgets do the real work. Instead of picking an arbitrary threshold and hoping it's right, you tie the alert to how fast you're burning through the reliability budget you've already committed to.
Burn Rate Alerting:
SLO: 99.9% availability per month
Error budget: 43.2 minutes
Normal burn rate: 1x (budget lasts exactly 30 days)
Burn Rate Budget Consumed Alert Level
---------- ---------------- ----------------
1x Normal pace No alert
2x Budget gone in 15d Warning (Slack)
5x Budget gone in 6d Page during hours
14x Budget gone in 2d PAGE NOW (3 AM!)
100x Budget gone in 7h CRITICAL - all hands
Why this is smarter than threshold alerts:
- A 2-minute spike at 100x burn = 0.5% budget used
-> no alert (short, already resolved)
- A 1-hour period at 5x burn = 7% budget used
-> warning (sustained, investigate)
- Adapts to YOUR reliability targets automaticallyWhat makes burn rate alerting hold up under real traffic is that it accounts for both severity and duration at once. A brief spike that resolves itself doesn't wake anyone up. A sustained problem, even a moderate one, gets flagged before it quietly eats your whole month's error budget. This is the core idea behind most modern SRE alerting: page based on trajectory, not a single noisy data point.
Alert Priority and Routing
Not every real alert deserves the same response. A full outage and a minor performance dip are both "real," but they don't belong in the same escalation path.
Alert Severity Matrix:
Priority User Impact Response Time Channel
-------- ---------------- --------------- ------------------
P1 SEV1 Full outage < 5 min ACK Page on-call NOW
Revenue stopped 15 min response Incident commander
War room
P2 SEV2 Partial degraded < 15 min ACK Page during hours
Some users hit 1 hour response Slack + ticket
P3 SEV3 Minor impact < 4 hours Slack notification
Performance dip Next business day Auto-create ticket
P4 No user impact Best effort Weekly review
Warning/info Sprint planning Dashboard only
Escalation:
5 min: Primary on-call not ACK -> secondary on-call
15 min: Nobody ACK -> engineering manager
30 min: Still no ACK -> VP engineering + status pageUnder the hood, most alert routing setups are just event pipelines: a metric crosses a threshold, that fires an event, and it gets fanned out to Slack, a pager, and a ticketing system. If you're building this kind of pipeline yourself, the ideas in our piece on AMQP, Kafka, and event sourcing apply almost directly, particularly around making sure an alert event can't get silently dropped between the trigger and the page. The same reliability guarantee shows up in our transactional outbox pattern post: you want the alert write and the alert delivery to happen atomically, or you end up with the exact failure mode from the fintech story above, an alert that fired but never actually reached anyone.
The Alert Audit: Reducing Noise
Good alerting strategy isn't a one-time setup, it's maintenance. Alerts that made sense six months ago accumulate cruft as the system changes underneath them, and reducing alert fatigue has to be a recurring habit, not a one-off cleanup.
Monthly Alert Audit Process:
For each alert that fired last month:
1. Was it actionable? (Did someone need to DO something?)
No -> DELETE the alert or convert to dashboard
2. Did it require immediate action?
No -> Downgrade to P3/P4 (no paging)
3. Was it a duplicate of another alert?
Yes -> Consolidate into one alert
4. Was the threshold correct?
Too sensitive -> Increase threshold
Target: < 2 pages per on-call shift per week
Each false alarm erodes trust in the system
Each missed real alert erodes trust in the teamNone of this happens by accident. Deciding what pages a human at 3 AM versus what waits for the next standup is a design decision, the same category of decision as choosing where to place a cache or how to structure a service boundary. If you want to see how these choices fit into the bigger picture of building reliable systems, our guide to low-level design fundamentals is a solid next stop, and our roundup of system design learning platforms is useful if you want structured practice beyond reading.
Interview Tip
When discussing monitoring and alerting, say: "I would implement SLO-based alerting using burn rate. Instead of alerting on CPU or memory thresholds, I alert when the error budget consumption rate suggests we will exhaust our monthly budget prematurely. A 14x burn rate pages immediately. A 5x burn rate creates a ticket during business hours. I would audit alerts monthly, targeting less than 2 pages per on-call shift per week. Every alert must be actionable, if the response is always just wait and see, it should be a dashboard metric, not a page."
Architecture overview
Key Takeaway
Alert on user-facing symptoms (error rate, latency) not internal metrics (CPU, memory). Use burn rate alerts tied to SLOs so alerts are proportional to actual reliability impact. Ruthlessly reduce alert volume to eliminate fatigue, every false alarm erodes trust in the system.