When you search the web, one query is handled by a front end, which asks an authentication service, a search service, an ads service, and then hundreds of index servers. If the page is slow, which of those hundreds of machines is to blame? Each machine's own logs only show its own small slice of the story.
Google's answer was Dapper. It follows a single request as it hops from machine to machine and draws the whole journey as a tree. It was designed to run on every production machine all the time, so engineers could ask "why was this slow?" about real traffic. Open-source tracers such as Zipkin were built on the ideas in this paper.
Dapper gives every request a trace id and breaks it into spans (units of work with a parent). It records only a small sample of requests, collects them out of band, and instruments a few shared libraries so that application teams do not have to change code.
Traces and spans
A trace is the full story of one user request. It is made of spans. A span is one named piece of work, usually an RPC call, with a start time, an end time, and some notes.
Every span has its own id and the id of its parent span. The first span has no parent. Because of those ids, you can take thousands of loose log lines from hundreds of machines and rebuild them into one tree.
A span can also hold annotations: timestamped notes added by the framework or by the programmer, such as "cache miss" or "retry 2." Dapper stores the name of each RPC but not its payload, because logging user data would be a privacy risk.
