Tracing, Correlation Ids and Sandbox Mode

A customer writes: 'my payment failed at 3:42 and I do not know why.' Behind one API call there may be a gateway, six services and two third-party calls. Without a way to follow that single request through all of them, support has to guess. Observability for APIs starts with one idea: every request carries an identity that every hop records.

A parcel crosses a warehouse, two trucks and a sorting hub. The tracking number printed on the box lets every stop log a scan, and the sender can ask 'where is parcel 123?' and get the whole journey. A correlation id is the tracking number for a request.

The gateway creates an id for each request, or accepts a sensible one from the client, and passes it to every downstream call. Every service puts it in its log lines. The id is returned to the client in a response header such as `X-Request-Id` and in the body of error responses, so a customer can quote it and support can find every log line in seconds.

The W3C Trace Context standard gives this a common format. The `traceparent` header has four parts: a version, a 32-hex-digit trace id that stays the same across the whole request, a 16-hex-digit span id that identifies this hop, and flags (01 means the trace is sampled). Each service keeps the trace id, creates a new span id for its own work, and passes the header on. Press Run.

A tracing system collects the spans, sees that they share a trace id, and draws the request as a timeline: which service was called, in what order, and how long each took. The span ids and the parent links are what let it build that tree.

The pictures show one request through three services, how each hop reports its own span, and how support finds the whole story from the single id the customer quotes.

• -**Logs** say what happened at one place: an error message, a decision. Structured (JSON) logs with the trace id are searchable.
• -**Metrics** say how much and how fast, aggregated over time: request rate, error rate, p99 latency. They are cheap and drive alerts.
• -**Traces** say where the time went across services for one request. They are detailed and expensive, so they are sampled: keep a small share of normal requests and always keep errors and slow requests.

An alert from a metric ('p99 latency doubled'), then a trace of a slow request ('the payments call took 2 seconds'), then the log lines of that service with the same trace id ('connection pool exhausted') is the usual path from symptom to cause.

Logs travel far: to a log store, to dashboards, to support tools, and they live for months. Never write passwords, API keys, tokens, full card numbers or whole request bodies that may hold personal data. Redact known sensitive fields at the logging layer, log identifiers instead of content, and set a retention period. A log that leaks a token is a second place to steal one.

Integrators need to try your API without moving real money or touching real customers. A sandbox is a separate copy of the API for that. Good ones use separate keys (a `test` prefix is common), keep a completely separate data store so test and live data never mix, and mark every response (a `livemode: false` field, for example).

• -Keep the same endpoints, validation and error shapes as production, so code that works in the sandbox works live.
• -Offer deterministic magic inputs: a particular test card number always succeeds, another always declines, a particular amount triggers a timeout, so every error path can be tested.
• -Fire webhooks for sandbox events, and let developers trigger them, because webhook handling is the hardest part to test.
• -Provide a reset, and allow time to be advanced for subscriptions and expiry, so tests run in seconds.
• -Never let the sandbox drift into a different product. A sandbox that behaves differently from production teaches the wrong things.