A time-series metrics system stores numbers indexed by name, a set of labels, and time: http_requests_total{method="GET", route="/orders", status="200", instance="api-7"} was 1,204,511 at 14:03:15. Prometheus, InfluxDB, VictoriaMetrics, Datadog and CloudWatch all work on this model. In a design interview you might be asked to build one, or to design the monitoring for some other system, and the same handful of ideas decide whether the design survives contact with real traffic.
The central one surprises people: the number of data points is rarely the problem. The number of distinct series is.
A series is every combination of label values
Each unique combination of metric name and label values is its own series, with its own entry in the index, its own in-memory buffer for recent samples, and its own chunk files on disk. So the series count for a metric is the product of the distinct values of each label.
Take a request counter with four labels. A short Node 22 script makes the arithmetic concrete:
const labels = { method: 5, route: 40, status: 8, instance: 200 };
const series = Object.values(labels).reduce((a, b) => a * b, 1);
console.log('series for http_requests_total:', series.toLocaleString('en-US'));
console.log('...with a user_id label (50,000 active users):', (series * 50000).toLocaleString('en-US'));
const perDay = series * (86400 / 15);
console.log('samples/day:', perDay.toLocaleString('en-US'), ' ~GB/day:', (perDay * 1.37 / 1e9).toFixed(2));series for http_requests_total: 320,000
...with a user_id label (50,000 active users): 16,000,000,000
samples/day: 1,843,200,000 ~GB/day: 2.53320,000 series is an upper bound (not every route returns every status), and it’s manageable. Add a user_id label and the bound becomes 16 billion. Even if only a fraction of those combinations ever appear, the index explodes, memory usage climbs until the server is killed, and queries that touch the metric time out. Prometheus’s documentation says plainly not to put high-cardinality values like user ids or email addresses in labels. Request ids, full URLs with query strings, raw SQL text and timestamps are the other usual offenders.
The second line also shows why samples aren’t the bottleneck. Facebook’s Gorilla paper reported about 1.37 bytes per sample using delta-of-delta timestamps and XOR-compressed values, and modern TSDBs are in that range. Nearly two billion samples a day comes to 2.5 GB. That’s a disk you can buy. Sixteen billion index entries is not something you can fix with hardware.
So the design rule is: labels describe a bounded set of things you’d want to group or filter by (service, region, route template, status class). Per-entity detail like “which user saw this error” belongs in logs or traces, which are built for high-cardinality lookups. In code, that means recording route="/orders/{id}", the template, rather than /orders/98123.
Ingestion and storage, briefly
For the write path, there are two models. Prometheus pulls: it scrapes each target’s /metrics endpoint every 15 or 30 seconds. That gives it a built-in health signal (a failed scrape is itself a metric, up == 0) and lets the monitoring system control load. Push-based systems (StatsD, OpenTelemetry’s OTLP export, CloudWatch’s PutMetricData) suit short-lived jobs and clients behind NAT. At large scale both end up behind a write path that shards series across ingesters by a hash of the series labels, batches recent samples in memory, and writes compressed blocks every couple of hours, with a write-ahead log so a crash doesn’t lose the in-memory window.
Queries mostly ask for recent data over a time range for a few series, aggregated: “error rate by route, last six hours”. Storage is therefore organised by series and by time block, with an inverted index from each label pair to the series that carry it, so {route="/orders"} finds its series without scanning everything.
Retention tiers and rollups
Nobody needs 15-second resolution for last year’s traffic. The usual design keeps several tiers:
raw 15s 15 days samples: 2.76e+10
5m rollup 90 days samples: 8.29e+9
1h rollup 730 days samples: 5.61e+9Those sample counts, for the same 320,000 series, come from the same script. Two years of hourly data costs less than two weeks of raw data. A background compactor reads expired raw blocks, aggregates each series into 5-minute and 1-hour buckets, and deletes the raw blocks once the rollups are written. Thanos and Cortex do this for Prometheus data; InfluxDB uses downsampling tasks; Graphite has retention rules per metric pattern.
What a rollup stores matters. One averaged value per bucket loses information you’ll want. Store sum, count, min and max per bucket (so four values, which roughly quadruples the rollup sample counts above, still small). From those you can rebuild the average for any coarser window correctly, sum / count, and still see the spike that max captured. Averaging averages gives the wrong answer whenever buckets have different counts.
Counters need one more step. A counter like http_requests_total only goes up, except when a process restarts and it drops to zero. Rates are computed from differences between samples, with resets detected and corrected (Prometheus’s rate() does this). Rolling up a counter by averaging its raw value is meaningless; roll up the increase within each bucket instead.
The percentile trap
Latency is where rollups and aggregation go wrong most often. Suppose four API instances report their own p99 latency, and one is struggling. The script simulates 9,000 requests on each healthy instance around 40 ms and 1,000 requests on a slow one around 400 ms:
const p = (arr, q) => { const s = [...arr].sort((a, b) => a - b); return s[Math.floor(q * (s.length - 1))]; };
const perInstanceP99 = inst.map((a) => p(a, 0.99));
const avgOfP99 = perInstanceP99.reduce((a, b) => a + b) / perInstanceP99.length;
const trueP99 = p(inst.flat(), 0.99);per-instance p99 (ms): 60, 60, 60, 596
average of p99s: 194 ms true fleet p99: 489 msThe dashboard that averages per-instance p99s says 194 ms. The real p99 across all requests is 489 ms, two and a half times worse. Percentiles don’t compose: you can’t average them across instances, and you can’t average them across time buckets either, so a “p99 by hour” rollup built from per-minute p99s is just as wrong.
The fix is to store distributions, not percentiles. A histogram records counts per latency bucket (le="50", le="100", le="250" and so on). Counts add up, so merging four instances or sixty minutes is just summing bucket counts, and you compute the percentile at query time:
bucket counts {"le25":3376,"le50":16847,"le100":6777,"le250":129,"le500":624,"le1000":247} -> p99 falls in bucket le 500The merged histogram puts p99 in the 250 to 500 ms bucket, consistent with the true 489 ms. Prometheus’s histogram_quantile() interpolates within that bucket, so the precision depends on where you put the boundaries; choose them around your SLO thresholds. Newer formats (Prometheus native histograms, OpenTelemetry exponential histograms) pick bucket boundaries automatically with bounded relative error.
Interview exercise: the metrics bill that tripled
Your team’s hosted metrics bill tripled this month and the Prometheus server is restarting with out-of-memory errors. Nobody added new services. Where do you look, and what do you change?
Answer and reasoning
Memory and cost tracking series count, with no new services, points at a cardinality explosion: some label has started carrying unbounded values. Find it first. Prometheus exposes its own numbers: prometheus_tsdb_head_series shows the active series trend, and the TSDB status page (/tsdb-status in the UI, or /api/v1/status/tsdb) lists the metrics and label names with the most series. A query like topk(10, count by (__name__)({__name__=~".+"})) finds the worst metrics, then count by (label) on that metric finds the offending label.
The usual culprits are a recent change that put a raw path (/orders/98123) instead of a route template into a route label, a user or tenant id added “for debugging”, or an error label carrying full exception messages that include ids. Fix the instrumentation so the label carries a bounded value, and stop the bleeding immediately with a relabel rule at scrape time (metric_relabel_configs with labeldrop or a replace that normalises the value). Then put guardrails in place: a per-target sample_limit so one bad deploy can’t take down the server, an alert on head series growth, and a review checklist item for new labels. The data that the label was trying to capture, which user hit which error, should go to logs or traces with a trace id, where high cardinality is expected.