Metrics, Logs & Traces: The Three Pillars

Monitoring tells you when something is broken. Observability tells you why. With services split apart and traffic flowing through caches, queues and several regions, why stops being obvious from a single log file. A request may touch ten machines, and the problem may be on any of them. You need three complementary tools, usually called the three pillars: metrics, logs and traces.

A patient in hospital is connected to a monitor showing heart rate, temperature and oxygen level. These are cheap, continuous numbers, and an alarm sounds when one leaves its safe range. The monitor tells nurses something is wrong, but not what. The patient's chart holds detailed notes on every event: medication given at 2:10, a fever at 3:05. And a record of the patient's journey, from emergency room to X-ray to surgery, with times at each stop, shows where the waiting happened.

• -The vital-signs monitor is metrics: numbers over time, good for noticing.
• -The chart notes are logs: detailed events, good for explaining.
• -The journey record is a trace: one request followed through every service, good for locating.

Metrics are numeric time series: counters and gauges such as requests per second, or error rate 2%. They are cheap to store and fast to query, which makes them ideal for dashboards and for alerting the moment a number crosses a threshold. The card below describes the monitoring component as it appears in the lab:

Logs are discrete, timestamped events with full detail: user 4821 got a 500 error at 3:14:02 pm, with a stack trace attached. They answer what exactly happened, at the price of being expensive to store and search at scale without a dedicated aggregation system. Write logs as structured data (JSON, with fields) rather than free text, and put a request id in every line so one request can be followed across services. The card below describes the log aggregator:

A trace follows one specific request across every service it touches, as a set of spans, each recording one step and its duration. It shows exactly where the time went: this request took 800 ms in total, and 600 ms of that was one slow database query in the Orders service. Traces are what make a microservices system debuggable at all.

• -3:14 am: an alert fires because the error rate passed 2%. This is a metric telling you something is wrong.
• -The dashboard shows the latency of checkout, measured at the 99th percentile, jumped from 300 ms to 4 seconds. Another metric, narrowing down what.
• -You open a trace of one slow checkout. It took 4.1 seconds: 50 ms in the gateway, 100 ms in Orders, and 3.9 seconds in a Payments database query. The trace says where.
• -You search the Payments service's logs for that trace id and find lock wait timeout exceeded on table payments. The log says why.

Averages hide pain. Say 980 of 1,000 requests take 100 ms and 20 take 10 seconds. The average is (980 x 100 + 20 x 10,000) / 1,000 = 298 ms, which looks acceptable. But one user in fifty waited ten seconds. A percentile shows it: the 99th percentile (p99) is the time that 99% of requests beat, and here it is 10 seconds. Track p50 for the typical experience, and p95 or p99 for the worst you are willing to allow.

A widely used starting point is the four golden signals: latency (how long requests take, including failed ones), traffic (how much demand), errors (the share of requests that fail), and saturation (how full the system is, such as CPU, memory or queue depth). If you only had four dashboards, these would be them.

• -Logs are bulky: a service writing 1 KB per request at 5,000 requests a second produces 5 MB a second, about 432 GB a day. Keep them for days, not years, and sample the noisy ones.
• -Traces are sampled: keep, say, 1% of requests plus every error, rather than all of them.
• -Metrics must not carry unbounded labels: a label with the user id creates a separate time series for every user and overwhelms the metrics store.