Skip to content
Dapper, a Large-Scale Distributed Systems Tracing Infrastructure 3 min

Benjamin H. Sigelman, Luiz André Barroso, Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan, Chandan Shanbhag · Google Technical Report 2010

Dapper, a Large-Scale Distributed Systems Tracing Infrastructure

How Google follows one request through hundreds of machines: trace ids, spans, out-of-band collection and sampling, with a waterfall you can slow down.

Read the original paper

Systems & Distributed · White Belt · Pro · 3 min read

Dapper, a Large-Scale Distributed Systems Tracing Infrastructure

When you search the web, one query is handled by a front end, which asks an authentication service, a search service, an ads service, and then hundreds of index servers. If the page is slow, which of those hundreds of machines is to blame? Each machine's own logs only show its own small slice of the story.

Google's answer was Dapper. It follows a single request as it hops from machine to machine and draws the whole journey as a tree. It was designed to run on every production machine all the time, so engineers could ask "why was this slow?" about real traffic. Open-source tracers such as Zipkin were built on the ideas in this paper.

Dapper gives every request a trace id and breaks it into spans (units of work with a parent). It records only a small sample of requests, collects them out of band, and instruments a few shared libraries so that application teams do not have to change code.

Traces and spans

A trace is the full story of one user request. It is made of spans. A span is one named piece of work, usually an RPC call, with a start time, an end time, and some notes.

Every span has its own id and the id of its parent span. The first span has no parent. Because of those ids, you can take thousands of loose log lines from hundreds of machines and rebuild them into one tree.

A trace is a tree of spans. A span is one piece of work, such as an RPC. Because each span records its parent, the tree can be rebuilt afterwards from loose log lines.

A span can also hold annotations: timestamped notes added by the framework or by the programmer, such as "cache miss" or "retry 2." Dapper stores the name of each RPC but not its payload, because logging user data would be a privacy risk.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

2 more sections, 1 more diagram and 3 interactive elements to play with

  • Three design goals
  • Sampling: record only a small share

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.