The Three Pillars of Observability: Metrics, Logs, and Traces

The 3 AM alert you can't debug by staring at one screen

Your pager goes off at 3 AM: "High error rate on checkout service." You open your laptop. What do you actually look at first?

Without observability, the honest answer is: you SSH into servers and grep logs by hand, one at a time, across 50 instances, hoping the right log line is on the box you happened to pick. With observability, the sequence looks completely different: a dashboard shows you exactly when the error rate started climbing, you click into a trace from that window and see which specific service is timing out, then you pull that one service's logs for that exact time range and find the root cause — all in under five minutes, without touching a single server by hand.

That gap — five minutes versus a three-hour crawl through logs — is the entire argument for observability. It isn't a nice-to-have dashboard; it's the difference between an incident your team barely notices and one that burns an entire on-call shift.

Three different questions, three different tools

The mistake teams make early on is treating "observability" as one thing you buy or install. It's actually three distinct signal types, each answering a different question, each with a different cost profile:

The Three Pillars of Observability:

  METRICS (the numbers)         LOGS (the story)           TRACES (the journey)
  What is happening?            What happened?             Where is the problem?
  ========================     ========================   ==========================
  request_rate: 1500/s         14:32:07 INFO user.login   Trace: abc-123
  error_rate: 2.3%               user_id=42 ip=1.2.3.4     API Gateway:   10ms
  p99_latency: 450ms           14:32:07 ERROR payment       Auth Service:  5ms
  cpu_usage: 72%                 card_declined user=42       User Service:  3ms
  cache_hit_rate: 94%          14:32:08 WARN db.pool          Cache:        1ms
  queue_depth: 1,200             connections_low=true         DB Query:     1985ms <-!
  ========================     ========================   ==========================
  Cheap to store               Expensive at scale         Most expensive
  Great for dashboards         Great for debugging        Great for latency analysis
  Alerts & capacity planning   Root cause analysis        Cross-service diagnosis

The instinct to pick just one and skip the others is understandable — running three separate systems is more operational overhead than running one. But each pillar genuinely can't do what the other two do. Metrics can tell you error rate spiked to 5% at 14:32, but not which specific requests failed or why. Logs can tell you exactly what one request's payment attempt did, but not whether that's 1 failure or 10,000 happening at once across the fleet. Traces can show you a single request crossing 12 services, but you wouldn't want to page through individual traces to answer "is the system healthy right now." You need the numbers to know something's wrong, the journey to know where, and the story to know why.

Metrics: the RED and USE methods

Metrics are cheap to store because they're pre-aggregated numbers, not raw events — which is exactly why they're the right tool for always-on dashboards and alerting, where you're querying constantly and can't afford to scan raw logs every time.

The question most teams get wrong isn't "should I collect metrics," it's "which metrics actually matter." Two frameworks answer that, and they're not interchangeable — they're built for different layers of your stack.

RED Method (for request-driven services):
  R - Rate:     requests per second
  E - Errors:   errors per second (or error %)
  D - Duration: latency distribution (p50, p95, p99)

  These 3 numbers tell you 90% of a service's health.

USE Method (for infrastructure resources):
  U - Utilization: how busy is the resource? (CPU 72%)
  S - Saturation:  how much queuing is there? (queue: 1200)
  E - Errors:      are there error conditions? (disk full)

The USE method — Utilization, Saturation, Errors — came first, developed by performance engineer Brendan Gregg for diagnosing resource-level problems: CPU, disk, memory, network interfaces. For every resource, you ask the same three questions: how busy is it, how much is queuing up waiting for it, and is it throwing errors.

The RED method — Rate, Errors, Duration — is a direct adaptation of that idea for services rather than hardware resources. Tom Wilkie introduced it at a Prometheus meetup in London in 2015, explicitly building on Gregg's USE framework but reframed around what matters for a request-driven microservice: how many requests is it handling, what fraction fail, and how long do they take (as a distribution — p50, p95, p99 — not a single average, since averages hide the slow outliers that actually hurt users).

The practical rule: use RED for anything that serves requests — APIs, gateways, services in a mesh. Use USE for the infrastructure those services run on — CPU, memory, disk, connection pools. A checkout service gets RED metrics; the database connection pool behind it gets USE metrics. Together they cover both "is my service serving traffic well" and "is the machine underneath it running out of room."

Logs: structured is non-negotiable

A log line is a record of one specific event. The difference between a log that helps you at 3 AM and one that wastes your time comes down to one decision: structured versus unstructured.

BAD (unstructured):
  "User john logged in from 1.2.3.4 at 2024-01-15 14:32:07"
  Try querying: "all logins with latency > 1000ms" -> impossible

GOOD (structured JSON):
  {
    "timestamp": "2024-01-15T14:32:07Z",
    "level": "INFO",
    "event": "user.login",
    "user_id": 42,
    "ip": "1.2.3.4",
    "latency_ms": 23,
    "trace_id": "abc-123"     <-- links to traces!
  }

  Now you can query:
  - "all logins where latency_ms > 1000 in the last hour"
  - "all errors for user_id=42"
  - "all events with trace_id=abc-123" -> see full story

A free-text log line is only searchable by string-matching — you can grep for "declined," but you can't ask "show me every decline where latency exceeded a second," because latency isn't a queryable field, it's just characters inside a sentence. A structured log turns every meaningful piece of that event into a real field you can filter, aggregate, and join on.

The trace_id field is the detail that ties this whole system together, and it's worth calling out specifically: when every log line emitted while handling a request includes the same trace ID that request's spans carry, you can pivot directly from "I found this slow trace" to "show me every log line that happened during it" with one query. Without that shared identifier, correlating a trace to its logs means manually matching timestamps and hoping the clocks agree across services — exactly the kind of guesswork observability is supposed to eliminate.

Worth knowing before you set retention policies: logs are typically the largest line item in an observability budget, often 60–80% of total spend, because unlike metrics (small, aggregated numbers) logs are raw, high-volume text. This is why log retention windows tend to be much shorter than metric retention, and why some teams move older logs to cheaper columnar storage (formats like Parquet can run roughly two orders of magnitude cheaper than a general-purpose search index for the same volume of historical logs) once they're past the "actively debugging" window.

Traces: the journey across services

A trace stitches together every service a single request touched, tied together by a shared trace ID, so you can see the full path — API Gateway → Auth → User Service → Cache → Database — with exact timing at each hop. Traces are the most expensive of the three pillars to store in full, because a single busy service can generate a full trace for every request, and at real production volume that's an enormous amount of data, most of which looks identical and uninteresting. That's why production tracing setups almost always sample — keeping all errors and slow requests while thinning out the routine, successful ones — rather than recording every single trace at full fidelity. (If you want the full mechanics of how tracing and sampling work, that's its own deep topic worth a dedicated read.)

What actually drives your observability bill

Three variables set the cost of an observability stack, and they're the same three regardless of which pillar or which vendor: volume (how much raw data you're ingesting), cardinality (how many unique combinations of label values you're generating), and retention (how long you keep it queryable).

Cardinality is the one that catches teams off guard. Every unique combination of label values on a metric creates a separate time series — so a metric tagged with service and status_code might have a few dozen unique series, which is fine. Tag that same metric with a raw user_id or a full request_id, and you've just created a new time series per user or per request, which can turn a cheap metric into an unbounded, extremely expensive one almost overnight. High-cardinality data belongs in logs or trace attributes, where the storage and query model is built for it — not in metric labels.

The unified debug workflow

None of this matters in isolation. The value shows up when the three pillars hand off to each other during an actual incident:

The Observability Investigation Flow:

  1. METRICS alert: "checkout error rate > 5% for 3 minutes"
        |
  2. DASHBOARD: see error spike started at 14:32
        |
  3. TRACES: filter traces from 14:32 with errors
     Find: payment-service spans are timing out (30s)
        |
  4. LOGS: filter payment-service logs from 14:32
     Find: "connection pool exhausted, max=20"
        |
  5. ROOT CAUSE: database migration at 14:30 caused
     slow queries, exhausting the connection pool.
        |
  6. FIX: kill the migration, connections freed,
     errors clear within 30 seconds.

  Metrics -> Traces -> Logs -> Root Cause
  Each pillar answers a different question.

Notice the order isn't arbitrary: metrics are cheap enough to watch continuously, so they're what wakes you up. Traces narrow "something is wrong" down to "this specific service, this specific dependency." Logs then give you the actual error message that explains why. Trying to run this in a different order — grepping logs first with no idea which service or time window to look in — is exactly the 3-hour version of this same investigation.

Common pitfalls

• Treating the three pillars as three unrelated tools. If your logs don't carry a trace ID and your traces don't link back to the metric that fired the alert, you're maintaining three separate systems that happen to describe the same production environment, not one observability practice. The correlation ID is what makes them one system.

• Unstructured logging "for now." Free-text logs feel faster to write in the moment and turn into a permanent tax on every future investigation. Structure it from day one; retrofitting thousands of log call sites later is far more expensive than doing it upfront.

• High-cardinality metric labels. A well-intentioned user_id or order_id label on a metric is one of the most common ways teams accidentally blow up their metrics bill without touching log or trace volume at all.

• Averages instead of percentiles. A p50 (or worse, a mean) latency number can look perfectly healthy while 1% of your users sit through multi-second page loads. RED's "Duration" is meant to be a distribution — p50, p95, p99 — specifically so the slow tail doesn't hide behind a healthy-looking average.

• No retention strategy. Keeping everything at full fidelity forever isn't a strategy, it's a bill waiting to happen. Decide deliberately how long each pillar needs to stay hot and queryable versus archived or discarded.

Beyond three: is there a fourth pillar now?

Some current observability discussion argues metrics, logs, and traces alone aren't sufficient anymore, and proposes continuous profiling — code-level CPU and memory profiling running in production, not just in a debugger — as an emerging fourth signal. The reasoning: traces tell you which service is slow, but not which line of code inside that service is burning the CPU time. Profiling fills exactly that gap. It's not yet as standardized or universally adopted as the original three, but it's worth knowing the term if you see it mentioned — the three pillars are the durable foundation, not necessarily the final word.

Interview framing

A strong answer sounds like: "I'd implement all three pillars, and make sure they're connected rather than three separate tools. Metrics — Prometheus and Grafana — for dashboards and alerting, using the RED method for services and the USE method for infrastructure resources. Structured JSON logs, shipped to something like Loki or the ELK stack, with a trace ID on every log line so I can pivot from a trace straight to its logs. Distributed traces via OpenTelemetry into Jaeger or Tempo for cross-service latency diagnosis. The key point is the handoff: metrics tell me what is wrong and trigger the alert, traces show me where in the call chain, and logs tell me why it happened — in that order, because each one narrows the search space for the next." That sequencing, plus the RED/USE distinction, is what separates a memorized definition from an answer that shows you'd actually debug an incident this way.

Key takeaway

Metrics tell you what is wrong — cheap, aggregated numbers, ideal for dashboards and alerting via the RED method (services) and USE method (infrastructure). Logs tell you why — detailed, structured narratives of individual events, most valuable when a shared trace ID links them back to the request that generated them. Traces tell you where — the path a single request took across every service it touched. None of the three replaces the other two, and the real payoff comes from the correlation between them, not from picking a favorite. Budget accordingly: logs will likely dominate your storage bill, metrics stay cheap unless cardinality gets out of hand, and traces stay affordable only with deliberate sampling.

FAQ

Which pillar should I implement first if I can only do one? Metrics, because they're the cheapest to run and are what actually triggers an alert in the first place — logs and traces are for investigating an incident you already know exists. That said, adding a trace_id field to your logs from day one costs almost nothing and pays off enormously once you add tracing.

What's the difference between logs and traces if they're both event-based? A log is scoped to one event inside one service. A trace is scoped to one request across every service it touched, made up of many spans, and specifically designed to show timing and causality between services — something a pile of individual service logs can't reconstruct on its own.

Why not just use one tool that does all three? Several vendors (Datadog, Grafana stack, Elastic) offer unified platforms that handle all three signals, and correlation is genuinely easier when one vendor owns the trace ID linkage across all three. But "one vendor" and "one pillar" are different claims — even a unified platform still stores and prices metrics, logs, and traces differently under the hood, because they're fundamentally different data shapes.

Is a high cardinality metric always bad? Not inherently — cardinality is a cost and performance trade-off, not a correctness bug. The problem is unbounded cardinality (raw user IDs, full URLs, request IDs) on metrics specifically, since metric backends are built to handle a bounded, relatively stable set of label combinations, not one time series per unique request.

Do I need distributed tracing if I only have a handful of services? It's less urgent with two or three services, where you can often reason about a request's path from memory. It becomes valuable fast as soon as a single request routinely crosses more than four or five services, which is when manually correlating logs across services stops being practical.