Logs tell you what one service did. Metrics tell you how a service behaves in aggregate. Neither answers the question you have during an incident: this one checkout took 4 seconds, so where did the time go? The request touched a gateway, an order service, inventory, payments and an email consumer, each with its own logs on its own hosts.
Distributed tracing answers it by giving the whole request one identity and recording each piece of work as a timed span that knows its parent. Put the spans together and you get a tree: who called whom, in what order, and how long each step took. The interview questions are mostly about the mechanics underneath that tree, and about the ways it breaks.
Traces, spans and the header that links them
Three terms, precisely:
- Trace: everything that happened because of one originating request. Identified by a 128-bit trace id.
- Span: one unit of work inside the trace: handling an HTTP request, a database query, consuming a message. It has its own 64-bit span id, a start time, a duration, attributes, and the span id of its parent.
- Context propagation: carrying the trace id and the current span id across process boundaries, so the next service can create children of the right parent.
The W3C Trace Context standard defines the HTTP header that carries this, traceparent:
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
│ │ │ └ flags (01 = sampled)
│ │ └ parent span id (16 hex chars)
│ └ trace id (32 hex chars)
└ versionWhen a service receives a request, it reads traceparent, creates a span with the same trace id and with the incoming span id as its parent, and when it calls another service, it sends a new traceparent containing its own span id. That’s the entire linking mechanism. Everything else (the waterfall views, the service maps) is built from spans that carry these three ids.
Building the tree by hand
To see the mechanism without a tracing library in the way, here’s a small implementation in Node 22. AsyncLocalStorage holds the current span within a process, and traceparent carries it between “services” (plain functions that only communicate through headers):
const format = (s) => `00-${s.traceId}-${s.spanId}-01`;
function parse(header) {
const m = /^00-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$/.exec(header ?? '');
return m ? { traceId: m[1], spanId: m[2] } : null;
}
async function span(service, name, fn, incomingHeader) {
const parent = incomingHeader !== undefined ? parse(incomingHeader) : current.getStore();
const s = { service, name, traceId: parent?.traceId ?? hex(16), spanId: hex(8),
parentId: parent?.spanId ?? null, start: performance.now() };
try { return await current.run(s, fn); }
finally { s.ms = Math.round(performance.now() - s.start); spans.push(s); }
}
const outgoingHeaders = () => ({ traceparent: format(current.getStore()) });If there’s no valid incoming context, span starts a new trace with a fresh trace id. The order service writes its order, calls inventory and payments with outgoingHeaders(), and puts an OrderPlaced message on a queue with the same headers. Then the email consumer processes that message twice: once reading the message’s traceparent, and once ignoring it, the way a consumer without instrumentation would. Printing the spans as trees:
trace bc559d0d16f66e52dd40ee156a574911
gateway POST /checkout 212 ms
orders POST /orders 212 ms
orders INSERT order 9 ms
inventory POST /reservations 21 ms
inventory SELECT stock 15 ms
payments POST /charges 181 ms
payments call card network 181 ms
email consume OrderPlaced 30 ms
trace 5dacf9b1e726bf03c0ce06513f27ba39
email consume OrderPlaced 31 ms(Ids are random, so they differ on every run.)
Reading the first tree is the skill interviewers want to see. The checkout took 212 ms. The order insert and the inventory reservation are small. Payments took 181 ms, nearly all of it in the call to the card network. If this checkout were slow, you’d know immediately which team to talk to and which external call to examine, without grepping five services’ logs.
The second tree is the failure mode. The consumer that ignored the header started a brand-new trace. In a tracing UI you’d see an orphan consume OrderPlaced span with no link to the checkout that caused it. If emails are delayed, you can’t connect a specific delayed email to the order request behind it.
Where propagation breaks in real systems
Auto-instrumentation (the OpenTelemetry Java agent, Micrometer Tracing in Spring Boot 3, OpenTelemetry’s Node SDK) handles the common HTTP clients and servers for you. Traces break at the edges of what it knows about:
- Message queues. Producers must write
traceparentinto message headers and consumers must extract it. Instrumentation exists for Kafka and RabbitMQ clients, but a custom consumer loop or a hand-rolled batch processor often skips it. - Thread pools and async hand-offs. In Java, the current span lives in a thread-local. Submitting work to a plain
ExecutorServiceloses it unless the executor is wrapped (Context.taskWrapping(executor)in OpenTelemetry). In Node,AsyncLocalStoragesurvivesawaitbut not every callback-based library. - Custom HTTP clients and proxies. A gateway or sidecar that rebuilds requests and drops unknown headers severs every trace passing through it.
- Outbox relays and batch jobs. The event is written in one transaction and published later by a separate process. Store the
traceparentin the outbox row so the relay can put it on the message.
For messaging, there’s also a modelling question. Making the consumer a child of the producer’s span (as in the demo) works for one message. When a consumer processes a batch of 100 messages from 100 different traces, one parent isn’t possible, and OpenTelemetry uses span links instead: the consumer span links to each producer context without being a child of any.
Sampling: you won’t keep everything
A busy system generates millions of spans per minute. Storing all of them is expensive, so most setups sample.
Head sampling decides at the start of the trace, typically “keep 5%”. The decision travels in the flags byte of traceparent (01 = sampled), so every downstream service follows the same decision and you get complete traces. It’s cheap, but it’s blind: the rare request that failed or took 9 seconds is just as likely to be dropped as a boring one.
Tail sampling collects all spans briefly (usually in an OpenTelemetry Collector), waits until the trace is complete, then decides: keep every trace with an error, every trace over 2 seconds, and 1% of the rest. You keep the interesting traces, but the collector must buffer spans and see all spans of a trace, which is real infrastructure.
A common mistake is letting each service sample independently. Service A keeps 10%, service B keeps a different 10%, and almost no trace is complete. Respect the incoming sampled flag.
Making traces useful day to day
- Put the trace id in every log line. Then a log search for one trace id returns every log from every service for that request. Spring Boot 3 with Micrometer Tracing adds
traceIdandspanIdto the logging context; you just need them in the log pattern. - Add attributes that answer questions.
order.id,customer.tier,http.route. Avoid unbounded values as metric labels, but on spans they’re fine. - Mind clock skew. Span timestamps come from each host’s clock. A child that appears to start before its parent usually means the hosts’ clocks disagree, not that causality broke. Durations within one span are reliable; cross-host alignment is approximate.
Interview practice: the trace that stops at the gateway
“Your traces show the API gateway span and nothing else. The services behind it are instrumented with OpenTelemetry and their spans do appear, but each as a separate trace. What would you check?”
Answer
The services are creating spans, so instrumentation works. They’re starting new traces, which means they aren’t receiving a valid traceparent from the gateway. I’d check three things in order. First, whether the gateway injects the header at all: some gateways only propagate older formats like B3 (X-B3-TraceId), while the services expect W3C. The propagator configuration must match on both sides, or include both formats during a migration. Second, whether something between them strips headers: a proxy, a sidecar or an allowlist of forwarded headers. Capturing the request as it arrives at a service (or logging the incoming traceparent) settles this quickly. Third, whether the header is malformed: a trace id of the wrong length or all zeros is invalid per the spec, and receivers discard it and start a new trace.
Once fixed, I’d add a check to a synthetic test: send a request with a known traceparent through the gateway and assert that a downstream service logs the same trace id.
Further reading
- Microservices interview questions
- Microservices MCQs
- Sidecar proxies
- Async context propagation in Node
- Specification: W3C Trace Context