System Design Observability and Failure Questions

An observability design that starts with “we’ll use Prometheus, Grafana and Jaeger” has chosen tools before deciding what they’re for. The result is familiar: hundreds of dashboards, alerts that fire on every blip, and, during a real incident, no way to answer the one question that matters.

A better approach starts from questions. What will the on-call engineer need to know at 3 a.m.? What will a customer support agent need to look up? Each question points to a specific signal, and the design follows from that.

Questions first, signals second

Take a concrete system: an export service. A user requests a CSV export through an API, the request goes onto a queue, a worker builds the file and uploads it to object storage, and the user downloads it from a link. Here are the questions people will ask about it:

The three classic signals fall out of this naturally. Metrics are cheap aggregate numbers over time, good for “how is the whole population doing?” and for alerting. Traces follow one request across services and show where its time went. Logs record discrete events with detail, good for “what exactly happened to this one?” A common mistake is expecting one signal to do all three jobs: logs are a poor way to compute a p99, and metrics can’t tell you why one specific export failed.

Alerting on the error budget

Suppose the service promises that 99.9% of export API requests succeed over 30 days, at about 2,000 requests per second. The arithmetic for what that allows:

SLO 99.9% over 30 days at 2000 req/s: 5.18 B requests, error budget 5.18 M failed requests
  equivalent full outage: 43.2 minutes
  error rate  0.1%: burn rate   1.0x, whole month's budget gone in  720.0 hours
  error rate  0.5%: burn rate   5.0x, whole month's budget gone in  144.0 hours
  error rate  2.0%: burn rate  20.0x, whole month's budget gone in   36.0 hours
  error rate 10.0%: burn rate 100.0x, whole month's budget gone in    7.2 hours

The burn rate is the current error rate divided by the rate the SLO allows. A burn rate of 1 uses the budget exactly over the month; 20 uses it in a day and a half.

That framing fixes the most common alerting mistake, which is alerting on raw thresholds like “error rate above 1% for 5 minutes”. A one-minute 5% spike fires that alert and wakes someone up, yet barely touches a monthly budget. Meanwhile a steady 0.4% error rate never crosses the threshold and quietly burns most of the budget.

The Google SRE workbook recommends multiwindow burn-rate alerts instead. The thresholds come from how much budget a sustained burn would consume:

multiwindow alert thresholds (fraction of the monthly budget consumed):
  burn rate 14.4x sustained for  1 h consumes 2% of the budget
  burn rate    6x sustained for  6 h consumes 5% of the budget
  burn rate    1x sustained for 72 h consumes 10% of the budget

Page someone when the burn rate exceeds 14.4 over the last hour and also over the last 5 minutes (the short window makes the alert stop soon after the problem does). Open a ticket, not a page, for the slower burns. The result is fewer alerts, each tied to something users actually feel.

The cardinality bill

Metrics are cheap until labels multiply. A time series database stores one series per unique combination of label values, and the combinations multiply:

http_requests_total with method x route x status x instance: 100,000 possible series
  ... plus a customer_id label (200,000 customers): 20,000,000,000
  ... with the histogram's 12 buckets as well:      1,200,000 (before customer_id)

Five methods, 40 routes, 10 status codes and 50 instances is already 100,000 possible series. Not all combinations occur, but adding a customer_id label to “see per-customer errors” multiplies the possibilities by 200,000. Prometheus keeps active series in memory, so this kind of change can take down the monitoring system itself at the worst possible moment.

The rule: metric labels should have small, bounded sets of values (route templates like /v1/exports/{id}, never the raw path with the id in it). High-cardinality detail, like customer ids, export ids and user agents, belongs in logs and trace attributes, which are indexed for lookup rather than aggregated. If you genuinely need per-customer SLOs, compute them from logs or a dedicated pipeline for your top customers, not from a label on every metric.

Tracing through the queue

A trace connects spans by passing context along with each call. Over HTTP, the W3C traceparent header carries the trace id and parent span id. The export service breaks this at the queue: the API publishes a message and returns, and hours of work happen in a worker that has no idea which request started it. The trace splits in two, and the slowest part of the system, the queue wait plus build, sits in a trace nobody can find.

The fix is to put the trace context in the message itself. OpenTelemetry’s messaging instrumentation does this: the producer writes traceparent into the message headers, and the consumer creates its span either as a child of that context or with a span link to it (links suit batch consumers that process many messages in one span). Record the enqueue time in the message too, so the consumer can report queue wait as its own number.

Then make one identifier available everywhere. The export id exp_7Qm2 should appear as a trace attribute and in every log line about that export, alongside the trace id. When support asks why that export failed, an engineer searches logs for the export id, finds the trace id, and sees the whole path: API accepted at 09:14:03, queued for 6 minutes, build failed after 40 seconds with range_too_large.

Sampling without losing the failures

Tracing every request at 2,000 per second is expensive, so systems sample. Head sampling decides at the start of a request, randomly:

head sampling 1.0%: 20 traces/s kept; failed requests traced: 2.4 per minute out of 240
head sampling 0.1%: 2 traces/s kept; failed requests traced: 0.2 per minute out of 240

At a 0.2% error rate there are 240 failed requests per minute. Sampling 1% at the head keeps about 2 of them. Sampling decided before the request knows it will fail throws away exactly the traces you need. Tail sampling, typically in the OpenTelemetry Collector, buffers the spans of each trace and decides after it completes: keep every trace with an error or with latency above the p99, plus a small random share of normal ones. It costs a buffering tier, and it’s the only way to keep all the interesting traces at a reasonable volume.

Interview practice: the export that’s ‘stuck’

“Users say exports sometimes take hours. The API’s dashboards show p99 latency of 120 ms and an error rate of 0.01%. Nothing has alerted. What’s wrong with the observability, and what would you add?”

Answer

The dashboards measure the wrong thing. The API only enqueues the export, so a fast, error-free API says nothing about whether exports complete. The user’s experience is end-to-end time from request to downloadable file, and nothing measures it, which is why nothing alerted.

I’d add an SLI for that journey: the share of exports that reach succeeded within 10 minutes of creation, computed from the operation records, with a burn-rate alert on it. For diagnosis, I’d add queue metrics (depth and age of the oldest message, since age is what users feel), and per-step histograms for queue wait, build and upload. I’d propagate traceparent through the queue message so each export is one trace from the API through the worker, with tail sampling that keeps all failed and slow exports. Every log line would carry the export id and trace id, so support can go from a customer complaint to the exact trace. With that in place, “exports take hours” turns into a specific answer, such as the oldest message being 3 hours old because workers are stuck retrying one poison message.

Further reading

• System design interview questions
• System design MCQs
• Tracing and request causality in microservices
• Queues and backlog management
• Google SRE Workbook: Alerting on SLOs