Distributed Tracing with OpenTelemetry: Following Requests Across 20 Services

The 3,000-service question

Uber runs thousands of microservices. When a rider reports that "the app was slow," which service actually caused it — the pricing engine, the ETA service, a Postgres query three hops away, or a Kafka consumer lagging behind? Without tracing, an engineer's only tools are scattered dashboards, per-service logs, and guesswork. That's slow even when you know roughly where to look, and hopeless when you don't.

Uber solved this internally with a tracing system called Jaeger, built in 2015 by Yuri Shkuro's team specifically to find problematic interactions between microservices. Uber open-sourced it in early 2017; by then it was already handling thousands of traces per second across hundreds of internal services. Jaeger joined the Cloud Native Computing Foundation later that year and graduated to a top-level CNCF project in 2019 — the same track Kubernetes and Prometheus took. The pattern it popularized is now standard: paste a request ID into a tracing UI and see a waterfall of every service the request touched, how long each step took, and exactly where time was lost.

This post covers how that works end to end — the data model (traces and spans), how context survives a hop across 20 services, how OpenTelemetry standardizes instrumentation so you're not locked into one vendor, and how teams keep tracing affordable at high request volume using sampling.

Traces and spans: the actual data model

A trace represents one request's complete journey through your system. It's identified by a single trace ID that every service touched by that request shares. A span represents one unit of work within that journey — an HTTP handler, a database query, a cache lookup, a function call you cared enough to measure.

Anatomy of a Distributed Trace:

  Trace ID: abc-123-def-456 (unique per request)

  |-- API Gateway (span-1) ---- 250ms total -------|
  |   |-- Auth Service (span-2) -- 15ms --|        |
  |   |-- User Service (span-3) -------- 210ms --| |
  |       |-- Redis Cache (span-4) - 2ms (HIT) |  | |
  |       |-- PostgreSQL (span-5) -- 195ms ----|  | |
  |   |-- Recommendations (span-6) -- 20ms --|    |
  |-----------------------------------------------|

Reading this waterfall top to bottom tells you the whole story without touching a single log file: the request took 250ms overall, auth and recommendations were cheap, and User Service's 210ms was almost entirely a 195ms Postgres query. That one line is the bottleneck. No dashboard-hopping required.

Every span carries the same core fields, regardless of language or backend:

A trace, then, is just a tree of spans connected by parent-child relationships, all sharing one trace ID. The interesting engineering problem is how that trace ID and parent span ID get from service #1 to service #20 without every team hand-rolling their own header format.

Context propagation: how the trace ID survives 20 hops

Context propagation is the mechanism that turns 20 isolated, disconnected spans into one coherent trace. If a single service in the chain fails to forward the tracing headers — because it's an old service, a serverless function, or a message queue that doesn't preserve headers — the trace breaks into fragments at that point, and you lose exactly the visibility distributed tracing exists to provide. This is the single most common cause of "why does my trace stop halfway through" tickets in production.

The industry converged on one standard for this: W3C Trace Context, carried in an HTTP header called traceparent. Every OpenTelemetry SDK implements it by default, which is why services written in five different languages can all participate in the same trace without agreeing on anything beyond this header format.

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

Broken into its four hyphen-separated fields:

The propagation flow is mechanical: Service A starts a span, generates a trace ID, and attaches a traceparent header to its outbound call to Service B. Service B reads that header, creates its own span as a child of Service A's span (same trace ID, new parent ID pointing at its own span), and attaches an updated traceparent to whatever it calls next. Repeat 18 more times and you have a complete, connected trace — as long as nothing in the chain drops the header. Queues, batch jobs, and pub/sub systems need explicit support for this (OpenTelemetry has semantic conventions for messaging systems specifically because headers don't propagate through Kafka or SQS the way they do through HTTP).

OpenTelemetry: instrument once, send anywhere

Before OpenTelemetry, every tracing vendor (Jaeger, Zipkin, Datadog, New Relic) shipped its own instrumentation SDK. Adopting one meant rewriting instrumentation to switch to another — real vendor lock-in, at the code level. OpenTelemetry, a CNCF project formed by merging OpenTracing and OpenCensus, fixed this by standardizing the instrumentation layer and separating it from the backend that stores and visualizes the data.

OpenTelemetry Architecture:

  [Your Service]
  |  OTel SDK (auto-instrumentation)
  |  Instruments: HTTP, gRPC, DB, Redis, Kafka
  |  Zero code changes for basic tracing!
  |
  v
  [OTel Collector]
  |  Receives, processes, exports telemetry
  |  Can: sample, filter, batch, transform
  |
  +---> [Jaeger]       (open-source, self-hosted)
  +---> [Zipkin]       (open-source, lightweight)
  +---> [Datadog]      (SaaS, full APM)
  +---> [Honeycomb]    (SaaS, high-cardinality)
  +---> [Grafana Tempo] (open-source, S3 backend)

  Vendor lock-in: ZERO
  Switch backends by changing collector config.
  No code changes in your services.

The Collector is the piece most tutorials skip past, but it's what makes this architecture actually operable at scale. It's a standalone binary that sits between your services and your storage backend, and it's built from three pieces you configure in a pipeline:

• Receivers — how data gets in. Usually OTLP (OpenTelemetry's native protocol), but the Collector can also ingest Jaeger-format or Zipkin-format spans, so you can migrate incrementally instead of flipping every service at once.

• Processors — what happens to data in flight. This is where batching happens (grouping spans for efficient network export), where a memory limiter prevents the Collector from OOMing under load, where an attributes processor can strip PII before it ever reaches a third-party SaaS backend, and — critically — where tail-based sampling decisions get made.

• Exporters — where data goes out. You can fan the same trace data out to multiple backends simultaneously, or route traces to Tempo while metrics go to Prometheus.

Because the SDK only ever talks to the Collector over OTLP, switching from Jaeger to Honeycomb to Datadog is a one-line config change in the Collector's exporter section. Nothing in your 20 services needs to be touched or redeployed. That's the actual mechanism behind "zero vendor lock-in," not just a marketing claim.

Auto-instrumentation: tracing without touching your code

The fastest path to a working trace is auto-instrumentation, which patches your language's common libraries (HTTP frameworks, DB drivers, message queue clients) at runtime to emit spans automatically.

# Python auto-instrumentation (zero code changes):
# Install:
#   pip install opentelemetry-distro opentelemetry-exporter-otlp
#   opentelemetry-bootstrap -a install

# Run your existing app with auto-instrumentation:
#   opentelemetry-instrument python app.py

# What gets auto-instrumented:
# - All Flask/FastAPI HTTP endpoints
# - All outbound HTTP requests (requests, httpx)
# - All PostgreSQL/MySQL queries
# - All Redis commands
# - All Kafka produce/consume
# - All gRPC calls

That's genuinely everything needed to get end-to-end traces across a Python service mesh with zero code changes — a reasonable first milestone if you're instrumenting 20 services and don't want to touch all of them by hand. Node.js follows the same pattern with @opentelemetry/auto-instrumentations-node loaded via --require, and Java uses a javaagent attached at JVM startup — no source changes in either case.

Auto-instrumentation covers the "what called what, and how long did it take" question. It won't know that a span represents a $150 payment or which pricing tier a user is on — for that you add manual spans around the business logic you specifically want visibility into:

from opentelemetry import trace
tracer = trace.get_tracer('checkout-service')

with tracer.start_as_current_span('process_payment') as span:
    span.set_attribute('payment.amount', 150.00)
    span.set_attribute('payment.currency', 'USD')
    result = charge_card(order)
    span.set_attribute('payment.status', result.status)

The recommended sequence, and the one most teams get right the first time, is: turn on auto-instrumentation everywhere first to get baseline visibility, then add manual spans only where the automatic ones don't tell you enough. Going straight to hand-instrumenting everything is a common reason "add tracing" projects stall out — it's far more work than it needs to be.

One instrumentation pitfall worth flagging early: attribute cardinality. It's tempting to tag every span with a user.id or full request.url including query strings, but high-cardinality attributes (fields with effectively unbounded unique values) can blow up storage and indexing costs on your backend. Keep attributes bounded and meaningful — http.status_code, db.system, http.route (the templated route, not the raw URL) — and put truly unique identifiers in span events or logs instead, where the cost model is different.

Sampling: not every request needs a permanent record

At meaningful scale, recording a full trace for every single request is neither useful nor affordable. If you're doing 100,000 requests per second, tracing all of them at full fidelity runs roughly $50,000/month in storage on a typical backend — and the overwhelming majority of those traces look identical and boring. Sampling decides which traces are worth keeping.

There are two fundamentally different strategies, and the difference matters more than it first appears:

Head-based sampling decides at the very start of a request — before anyone knows how it will turn out. A common pattern is a flat percentage: flip a weighted coin, trace 1% of requests. It's simple, cheap, and infrastructure-light, which makes it the reasonable default when you're just getting started. Its blind spot is exactly the traces you care about most: a 1% sampler can't specifically catch the request that errored or the one that took 8 seconds, because that information doesn't exist yet at the moment the decision is made. If errors are 0.1% of your traffic and you sample at 1%, you're capturing roughly 1 in 1,000 of your error traces — the incident you actually need to debug may simply not have been recorded.

Tail-based sampling flips the order: the Collector buffers every span for a trace until it completes, then decides whether to keep it based on the full picture — did it error, did it exceed a latency threshold, did it touch a specific service. This means you can write a rule like "keep 100% of errors and everything over 2 seconds, keep 1% of the rest," which is the answer most engineers actually want. The cost is operational: the Collector has to hold every in-flight trace's spans in memory until a decision window closes (commonly 10–30 seconds), and all spans belonging to one trace need to land on the same Collector instance, which usually means adding a load-balancing layer in front of your Collector fleet keyed on trace ID. At 5,000 new traces/second, an 8-span average, 2KB per span, and a 30-second buffer window, that's roughly 2.4GB of memory just for the span buffer — real infrastructure, not a config toggle.

In practice, most teams at scale run both together: a cheap head-based sampler thins the firehose at the edge, and tail-based sampling downstream makes the smarter keep/discard call on what survives. Neither replaces metrics and logs — sampling is specifically a tracing-volume problem, and your error-rate and latency dashboards should still be built on unsampled metrics.

Tracing vs. logs vs. metrics

Distributed tracing is one of the "three pillars of observability," and it's worth being precise about what it's for, because teams sometimes try to make one signal do all three jobs.

Metrics tell you that p99 latency spiked at 14:32. Traces tell you which request path caused it and where the time went. Logs tell you the fine-grained detail once you already know which service and which time window to look at. You need traces specifically to answer "which of these 20 services is the bottleneck for this one request" — no amount of per-service log-grepping answers that question as directly.

Walking through a real trace

Tie it together: a rider opens the app and taps "Request Ride." The API Gateway receives the request, generates a trace ID, and calls Auth (15ms, fine), then User Service. User Service checks Redis first — a 2ms cache hit would normally end the story there — but this time it's a miss, so it falls through to Postgres, and that query takes 195ms. Recommendations runs in parallel and returns in 20ms, well within budget. Total: 250ms, and the waterfall makes it immediately obvious that the entire 250ms request is dominated by one 195ms database call inside User Service, not by the gateway, not by auth, not by recommendations. That's the diagnosis a tracing UI gives you in the time it takes to open one trace — the alternative is manually correlating timestamps across five separate log streams and hoping the clocks agree.

Common mistakes to avoid

A few issues show up repeatedly once teams move tracing into production:

• Broken propagation across async boundaries. Message queues, background jobs, and event-driven handlers don't automatically carry HTTP headers. If a service publishes to Kafka and a consumer picks it up later, the trace context has to be explicitly injected into the message and extracted on the other side, or the trace snaps in two at that boundary.

• Unbounded span attributes. Tagging spans with raw user IDs, full URLs with query strings, or request bodies inflates cardinality and cost fast. Prefer templated routes and bounded enum-like values.

• Treating 100% sampling as the safe default. It feels safer to keep everything, but it's usually the first thing that gets throttled or disabled entirely once a bill arrives — better to design a sampling strategy deliberately than to have one forced on you later.

• Clock skew between services. Trace waterfalls assume roughly synchronized clocks. Servers with meaningfully drifted clocks can show child spans starting before their parent, which is confusing and, at scale, worth fixing with NTP rather than debugging trace-by-trace.

• Instrumenting late. Tracing is most valuable exactly when you don't yet know where a problem is. Teams that wait until an incident to add tracing lose the one dataset that would have made the incident faster to resolve.

Interview framing

If this comes up in a system design interview, a strong answer sounds like this: "For distributed tracing, I'd use OpenTelemetry with auto-instrumentation so every HTTP call, database query, and message produces spans automatically with zero code changes. Traces propagate via the W3C traceparent header, so a trace ID and parent span ID survive every hop between services. I'd deploy an OpenTelemetry Collector as an intermediary — it decouples my services from any specific backend, so I can send data to Jaeger or Grafana Tempo without touching application code. For cost control at scale, I'd use tail-based sampling in the Collector: keep 100% of errors and slow requests, discard a controlled fraction of routine successful traffic. That way I capture every trace worth debugging without paying to store millions of uninteresting ones." That answer covers the data model, the propagation mechanism, the architecture, and the cost trade-off — the four things interviewers are usually listening for.

Key takeaway

Distributed tracing stitches spans from every service a request touches into one connected timeline, linked by a trace ID that's propagated hop-to-hop via the W3C traceparent header. OpenTelemetry standardizes how that instrumentation happens so you're never locked into one vendor's SDK, and the OpenTelemetry Collector decouples your services from wherever that data ends up — Jaeger, Tempo, or a commercial APM. At scale, sampling isn't optional: head-based sampling is a fine starting point, but tail-based sampling — keep all errors and slow requests, thin out the rest — is what lets you catch every request worth debugging without paying to store millions that aren't.

FAQ

What's the difference between a trace and a span? A trace is the full end-to-end record of one request across every service it touched, identified by a single trace ID. A span is one unit of work within that trace — a single HTTP call, database query, or function — and a trace is made up of many spans connected in a parent-child tree.

Is OpenTelemetry free? Yes. OpenTelemetry itself — the SDKs, the Collector, and the specification — is open source and free under the CNCF. You only pay for wherever you choose to send and store the data, whether that's a self-hosted backend like Jaeger or Tempo, or a paid SaaS backend like Datadog or Honeycomb.

Does adding tracing slow down my services? Well-implemented auto-instrumentation adds low, generally sub-millisecond overhead per span, and sampling reduces the volume that gets processed and exported further. The bigger risk to watch is unbounded attribute cardinality driving up backend cost, not request latency.

How long should traces be retained? Most teams keep detailed traces for hours to a couple of weeks, since trace data is high-volume and mainly useful for near-term debugging. Long-term trend analysis is better served by metrics, which are far cheaper to retain for months or years.

Can I use OpenTelemetry with an existing Jaeger or Zipkin setup? Yes. The OpenTelemetry Collector accepts Jaeger- and Zipkin-formatted spans as input, which is specifically meant to let teams migrate service-by-service instead of cutting over all at once.