Anomaly Detection in Production Systems: Catching Problems Before Users Notice

How Netflix Detects Problems Before Users Report Them

Netflix streams to 260 million subscribers across every timezone. Their traffic has strong patterns: spikes at 8pm local time, doubles on weekends, surges during new show releases. A static alert like 'error rate > 1%' would fire every Friday evening (normal traffic spike) while missing a 5x increase in buffering events at 3am Tuesday (genuinely broken CDN node). Netflix built an anomaly detection system that learns each metric's unique seasonal pattern and alerts only on deviations that are unusual FOR THAT TIME AND CONTEXT. Result: 60% fewer false alerts and 40% faster incident detection. The same approach powers fraud detection at Stripe, infrastructure monitoring at Datadog, and security intrusion detection at CrowdStrike.

Why Static Thresholds Fail

Static Threshold vs Anomaly Detection:

  Error Rate Over a Week:

  Rate%
  2.0|                              Static threshold: 1%
  1.5|                              ---- ---------- ------
  1.0|............................X.X......................
     |              ___            /\
  0.5|    ____     /   \     _____/  \_____
  0.2|___/    \___/     \___/               \___
  0.0+----+----+----+----+----+----+----+
     Mon  Tue  Wed  Thu  Fri  Sat  Sun

  Problem 1 - False Positives:
    Friday-Sunday: traffic doubles naturally, error rate rises
    to 0.8%. Threshold at 1% fires occasionally during peaks.
    On-call engineer ignores alerts -> alert fatigue.

  Problem 2 - False Negatives:
    Tuesday 3am: error rate jumps from 0.01% to 0.5%.
    50x relative increase! CDN node is failing.
    But 0.5% < 1% threshold -> no alert fires.
    Users notice before monitoring does.

  Anomaly Detection Solution:
  Instead of alerting on absolute value (> 1%),
  alert on deviation from expected baseline.
  Tuesday 3am baseline: 0.01%. Actual: 0.5% = 50x deviation.
  Friday 8pm baseline: 0.7%. Actual: 0.8% = normal variation.

Statistical Approaches

Anomaly Detection Methods (Simple to Advanced):

  1. ROLLING Z-SCORE:
     baseline = rolling_mean(metric, window=1h)
     stddev = rolling_stddev(metric, window=1h)
     z_score = (current_value - baseline) / stddev
     ALERT if abs(z_score) > 3
     Pros: simple, no ML needed
     Cons: ignores seasonality

  2. SEASONAL DECOMPOSITION:
     Build hourly baseline for each day-of-week.
     Tuesday 3am expected: 0.01% +/- 0.005%
     Friday 8pm expected: 0.7% +/- 0.15%
     ALERT if actual > expected + 3 * seasonal_stddev
     Pros: handles daily/weekly patterns
     Cons: needs 2-4 weeks of history

  3. PERCENTILE BANDS:
     For each hourly slot (e.g., Tuesday 3am):
     Compute historical 5th and 95th percentile.
     ALERT if current falls outside the band.
     Pros: robust to outliers in training data
     Cons: static bands, slow to adapt

  Method       Seasonality  Training Data  Compute Cost
  ------------ -----------  -------------- ------------
  Z-score      None         1 hour         Very low
  Seasonal     Daily/Weekly 2-4 weeks      Low
  Percentile   Daily/Weekly 4+ weeks       Low
  Prophet      Multi-level  2+ months      Medium
  LSTM         Complex      6+ months      High

ML-Based Anomaly Detection

When to use ML models:

  Prophet (Facebook):
    - Handles: daily, weekly, yearly, holiday seasonality
    - Input: timestamped metric values (e.g., 2 months of CPU%)
    - Output: predicted value + confidence interval
    - ALERT when actual falls outside confidence interval
    - Best for: business metrics (revenue, signups, orders)
    - pip install prophet; 5 lines of Python to set up

  Isolation Forest:
    - Detects multi-dimensional anomalies
    - 'CPU is normal, memory is normal, but CPU + memory
       + latency TOGETHER are unusual'
    - Isolates anomalies by random splits (outliers need
      fewer splits to isolate)
    - Best for: infrastructure with correlated metrics

  LSTM Neural Networks:
    - Learns complex temporal patterns
    - Predicts next value; deviation from prediction = anomaly
    - Expensive to train, overkill for most use cases
    - Best for: non-seasonal, non-linear patterns

  Production Reality:
    Platform         Built-in Anomaly Detection
    ---------------  ----------------------------
    Datadog          DBSCAN + seasonal decomposition
    New Relic        APM-native anomaly detection
    CloudWatch       Per-metric auto-models
    Grafana Cloud    ML-based anomaly panel
    PagerDuty        Intelligent alert grouping

    Start with built-in tooling. Custom models only when
    built-in cannot handle your specific seasonality.

Production Implementation

• Warm-up period: new metrics need 2-4 weeks of history before anomaly detection is reliable. Use static thresholds during warm-up.
• Multi-signal correlation: a single metric anomaly may be noise. Three correlated metric anomalies is almost always real.
• Alert suppression: during known events (deployments, marketing campaigns), temporarily widen the expected bands.
• Feedback loop: let on-call engineers mark alerts as true/false positive. Use this to tune sensitivity.
• Layered approach: static thresholds for critical hard limits (disk 95% full), anomaly detection for everything else.

Interview Tip

When discussing monitoring and alerting, say: 'I would use a layered approach. Static thresholds for absolute limits - disk at 95%, memory at 90%, error rate above 5% in any 1-minute window. For everything else, I would use anomaly detection that learns seasonal baselines. A 0.5% error rate at 3am Tuesday is a 50x deviation that demands attention, while 0.8% at Friday 8pm is normal weekend traffic. I would start with built-in tools like Datadog anomaly monitors, and add custom Prophet models only for business metrics with complex seasonality (revenue, signups, conversions). Multi-signal correlation reduces false positives - alert when 3+ metrics deviate together, not on individual noise.'

<

/blockquote>

Key Takeaway

Static thresholds miss gradual degradation and fire on normal traffic spikes. Anomaly detection learns seasonal baselines and alerts on relative deviations. Start with built-in tooling (Datadog, CloudWatch). Use a layered approach: static thresholds for hard limits, anomaly detection for behavioral changes, and multi-signal correlation to reduce false positives. The goal is fewer, more meaningful alerts - not more alerts.