Metrics, Logs & Traces: The Three Pillars
You'll learn to
- -Distinguish what each of the three pillars is best for
Monitoring tells you WHEN something is broken. Observability tells you WHY. With services split apart (previous module) and traffic flowing through caches, queues, and multiple regions, "why" stops being obvious from a single log file: you need three complementary tools.
Metrics
Numeric time series: counters and gauges like "requests per second" or "error rate: 2%." Cheap to store and query, ideal for dashboards and for triggering alerts the instant a number crosses a threshold.
Monitoring is your system's vital signs monitor. The same way a hospital attaches a heart monitor, blood pressure cuff, and oxygen sensor to a patient, you attach monitoring to your servers and services. There are three pillars: - METRICS: Numbers over time (CPU at 73%, 500 requests/second, 12ms average latency, 0.1% error rate) - LOGS: Detailed text records of what happened ('User #42 tried to login, password incorrect, IP: 203.0.113.5') - TRACES: Following a single request across your system ('Request hit API Gateway -> Auth Service -> User Service -> Database -> back, total 240ms') Together, these three pillars answer every question: 'Is the system healthy right now?' (metrics), 'Why did this user get an error?' (logs), 'Why is this endpoint slow?' (traces). Alerting sits on top: 'If error rate > 5% for 3 minutes, page the on-call engineer.' Without monitoring, you're flying blind. You only find out about problems when users complain, by which time thousands have already been impacted.
Examples: Prometheus + Grafana (open-source metrics standard), Datadog (all-in-one SaaS), AWS CloudWatch, ELK Stack (Elasticsearch + Logstash + Kibana for logs), Jaeger/Zipkin (distributed tracing), PagerDuty (alerting/on-call)
Logs
Discrete, timestamped events with full detail: "user 4821 got a 500 error at 3:14:02pm, stack trace attached." Logs answer "what exactly happened," at the cost of being expensive to store and search at scale without a dedicated aggregation system.
Imagine you have 200 servers, each writing its own log file. When a user reports a bug, you need to figure out what happened. Do you SSH into all 200 servers and grep through their logs one by one? That would take hours. A log aggregator collects ALL logs from ALL servers and services, ships them to a central searchable store, and lets you query across everything in seconds: 'Show me every log line containing user_id=42 from the last hour, across all services, sorted by time.' Now you can see the full picture: the request hit the API gateway at 14:23:01, reached the Order Service at 14:23:02, called the Payment Service at 14:23:03, and THAT'S where the error happened. Timeout after 30 seconds connecting to the bank's API. Without centralized logging, you'd never piece this story together. A good log aggregator also supports alerts ('notify me when ERROR log count exceeds 100/minute'), dashboards (error trends over time), and retention policies (keep logs for 30 days, then archive or delete).
Examples: ELK Stack (Elasticsearch + Logstash + Kibana, the classic), Grafana Loki (lightweight, pairs with Prometheus), Splunk (enterprise-grade, expensive), Datadog Logs (SaaS), AWS CloudWatch Logs, Fluentd/Fluent Bit (log shippers)
Traces
A trace follows one specific request across every service it touches, showing exactly where the time went: "this request took 800ms total, and 600ms of that was one slow database query in the Orders service." Traces are what make the microservices world from the last module debuggable at all.
A single request traced across three services: the trace pinpoints which service ate the time, before you ever open a log.
The three pillars work as a team: a metric alerts you that latency spiked, a trace shows you which specific service in the chain is slow, and a log reveals the exact error inside that service.
p99 latency for checkout just spiked from 200ms to 2s. What's the fastest path to root cause?
"Start reading through the logs from around that time to look for errors."
"I'd start with a trace, not logs: grab one of the slow requests and see exactly which service in the checkout call chain is eating the time, since with several services involved, guessing which one to check first from logs alone wastes time. Once the trace points at, say, the Inventory service, then I'd pull that service's logs for the specific error or slow query, and check its metrics to see if this is isolated or affects all of its traffic."