Spring Boot Distributed Tracing

When a checkout request touches the gateway, the order service, inventory, payments and a Kafka consumer, a single log file can’t tell you where its 2.3 seconds went. Distributed tracing can. Each service records spans (timed operations with a name, attributes and a parent), and because they all share one trace id, a backend like Jaeger, Zipkin, Grafana Tempo or a commercial APM can assemble them into one timeline.

Spring Boot 3 replaced Spring Cloud Sleuth with Micrometer Tracing, built on Micrometer’s Observation API. Interviewers ask about it to find out whether you’ve debugged across services in practice: how context travels, why traces break, and what sampling costs you.

The Java and configuration here target Spring Boot 3.x on Java 17+, and are illustrative rather than compiled for this article. The header format and the sampling numbers were checked with a Node 22 script, shown below. Boot 4 reorganises the dependencies into starters such as spring-boot-starter-opentelemetry and renames some properties, so check the reference for your version.

Wiring it up

You need Actuator, a bridge from Micrometer Tracing to a tracer library, and an exporter. For OpenTelemetry sending OTLP to a collector:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
  <groupId>io.opentelemetry</groupId>
  <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

spring.application.name=order-service
management.tracing.sampling.probability=0.1
management.otlp.tracing.endpoint=http://otel-collector:4318/v1/traces

The Brave alternative is micrometer-tracing-bridge-brave with zipkin-reporter-brave. Either way, spring.application.name becomes the service name on every span, so set it; otherwise all your services show up under the same default name.

With that in place, Boot instruments inbound HTTP requests in Spring MVC and WebFlux, outbound calls made through the auto-configured client builders, JDBC through additional libraries, and Kafka and RabbitMQ templates and listeners when observation is enabled for them. It also adds the trace and span ids to the logging MDC, so every log line carries a correlation id like [803B448A0489F84084905D3093480352-3425F23BB2432450] (the example the Boot docs use). That one feature is often worth more than the trace UI: paste a trace id into your log search and you get every line from every service for that request.

How context crosses the wire

Between services, trace context travels in an HTTP header. Boot’s default is the W3C Trace Context format, a traceparent header with four dash-separated fields:

import { randomBytes } from 'node:crypto';
const traceId = randomBytes(16).toString('hex'), spanId = randomBytes(8).toString('hex');
const header = `00-${traceId}-${spanId}-01`;
const [version, tid, pid, flags] = header.split('-');
console.log({ header, version, traceId: tid, parentSpanId: pid, sampled: (parseInt(flags, 16) & 1) === 1 });

{
  header: '00-ee2e8de1fb512b2249d354488270d079-1c52fd47fcfb38f4-01',
  version: '00',
  traceId: 'ee2e8de1fb512b2249d354488270d079',
  parentSpanId: '1c52fd47fcfb38f4',
  sampled: true
}

The version is 00. The trace id is 16 random bytes shared by every span in the request. The parent id is the 8-byte span id of the caller’s span. The last byte is flags, where bit 0 means “sampled”. When the order service receives this header, it creates its server span as a child of 1c52fd47fcfb38f4 in trace ee2e8de1..., and when it calls inventory it sends a new traceparent with the same trace id and its own span id. If you need Zipkin’s older X-B3-* headers to talk to legacy services, set management.tracing.propagation.type=b3 (or list both).

Where traces break

Most broken traces come from the same few causes, and interviewers listen for them:

• Clients built by hand. new RestTemplate() or RestClient.create() has no observation registry, so it sends no traceparent, and the downstream service starts a fresh trace. The Boot reference is blunt about it: use the auto-configured RestTemplateBuilder, RestClient.Builder or WebClient.Builder.
• Thread hops. Trace context lives in thread-locals. Work handed to an @Async method, a CompletableFuture.supplyAsync() or a custom executor starts without it. Wrap executors with Spring Framework 6.1’s ContextPropagatingTaskDecorator, or use the Micrometer ContextSnapshot API to carry context across.
• Messaging without observation. For Kafka, set spring.kafka.template.observation-enabled=true and spring.kafka.listener.observation-enabled=true so the producer writes the context into record headers and the consumer continues the trace.
• A proxy or gateway that strips unknown headers. Check that traceparent survives every hop.

Adding your own spans

Automatic spans cover the edges of a service. For the slow part inside it, wrap the operation in an observation:

@Service
public class PricingService {

    private final ObservationRegistry registry;

    public PricingService(ObservationRegistry registry) { this.registry = registry; }

    public Quote price(Cart cart) {
        return Observation.createNotStarted("pricing.quote", registry)
                .lowCardinalityKeyValue("currency", cart.currency())
                .highCardinalityKeyValue("cart.id", cart.id())
                .observe(() -> computeQuote(cart));
    }
}

An observation produces both a span and a timer metric. That’s why the API distinguishes low-cardinality keys (go on the metric and the span; must have few possible values) from high-cardinality keys (span only). Put a cart id in a low-cardinality key and you’ve created a metric series per cart, which is how tracing changes blow up a metrics bill. Annotating a method with @Observed does the same thing declaratively, once you register an ObservedAspect bean.

Sampling: deciding what to keep

Storing every span of every request is expensive at scale, so tracers sample. Boot’s default management.tracing.sampling.probability is 0.1, keeping 10% of traces. Two properties of sampling matter.

The decision must be made once. The root service decides, writes the result into the sampled flag, and every downstream service follows it (parent-based sampling, which is what Micrometer Tracing does when a sampled flag arrives). If each service rolled its own 10% die instead, you’d get fragments. The script simulated a million requests crossing four services:

independent: 82 complete traces, 343797 fragments out of 1,000,000
parent-based: 100024 complete traces, 0 fragments

Independent sampling kept almost no complete traces and over a third of a million useless fragments. That’s what happens if one service in the chain ignores incoming headers, so it’s worth checking.

Head sampling drops errors too. The decision is made when the request starts, before anyone knows it will fail. For a service doing 2,000 requests a second with a 0.2% error rate:

p=0.1: 720,000 traces/h stored, 1440 of 14400 error traces kept
p=0.01: 72,000 traces/h stored, 144 of 14400 error traces kept
tail (all errors + 1% of the rest): 86,256 traces/h stored, 14400 of 14400 error traces kept

At 10%, nine out of ten failing requests have no trace. Lowering the rate to save money makes it worse. Tail sampling, done in an OpenTelemetry Collector with its tail-sampling processor, buffers whole traces for a few seconds and keeps them based on outcome: all errors, all requests slower than a threshold, and a small percentage of the rest. In this example it stores about an eighth of the 10% head-sampled volume and keeps every error trace. The cost is a collector tier that has to hold traces in memory and route all spans of a trace to the same instance.

In practice, many teams send 100% from services to a nearby collector and let the collector do tail sampling, or keep head sampling at a modest rate and rely on log correlation (which works for every request, sampled or not) for the rest.

Interview exercise: the trace that stops at the order service

In Grafana Tempo, checkout traces show the gateway and the order service, then end. Inventory and payments appear as separate traces with their own ids. Logs in inventory have trace ids, but they don’t match the order service’s. All services are on Spring Boot 3.2 with the OTel bridge and 10% sampling. Where do you look?

Reasoned answer

Downstream services are producing traces, so tracing works in each of them; what’s missing is propagation from the order service. Since inventory’s log trace ids differ, inventory isn’t receiving a traceparent header and is starting new root traces.

The first suspect is how the order service builds its HTTP clients. If it uses RestClient.create() or new RestTemplate(), or builds a RestClient.Builder with RestClient.builder() rather than injecting Boot’s, no observation is attached and no header is sent. Check the code, then confirm on the wire: log request headers in inventory, or point the client at a request-echo endpoint. If the calls happen inside @Async methods or a custom executor, the context may be lost before the call, so check for a ContextPropagatingTaskDecorator. Also rule out a gateway or service mesh stripping the header, and mixed propagation formats (one service on b3 while the others expect w3c).

There’s a second, quieter consequence: with no incoming header, inventory makes its own 10% sampling decision, so even once propagation is fixed you should confirm traces are complete across all four services, not just more frequent.

Keep exploring

• Spring Boot interview questions
• Spring Boot multiple-choice practice
• Distributed tracing across microservices
• Keeping metric cardinality under control
• Tracing in the Spring Boot 3.5 reference