“You added a metric and the monitoring bill tripled. What happened?” Interviewers ask about metric cardinality to see whether you understand how time-series databases store data. Candidates often know how to create a Micrometer counter but not that tagging it with a customer ID creates one new series per customer. In production this ends with Prometheus running out of memory, slow dashboards, or a vendor invoice that gets escalated to your director.
Before you start
You should know what a counter, a timer and a gauge are, how Spring Boot Actuator exposes /actuator/prometheus, and roughly how Prometheus scrapes and stores metrics. Examples use Spring Boot 3.x with Micrometer and micrometer-registry-prometheus.
The short answer
A metric’s cardinality is the number of distinct tag-value combinations, and each combination is stored as its own time series. Tags must come from small, bounded sets: HTTP method, status class, URI template, outcome, region. Values such as user IDs, order IDs, raw request paths, query strings and exception messages are unbounded and will create series without limit. Spring Boot’s HTTP server metrics use the URI template (/orders/{id}) for exactly this reason. Guard against mistakes with a MeterFilter that caps tag values, and put per-request detail in logs and traces instead.
How it works
A Micrometer meter is identified by its name plus its tags. Registering a counter with a new tag value creates a new meter, which the Prometheus registry exports as a new series.
@Service
class PaymentMetrics {
private final MeterRegistry registry;
PaymentMetrics(MeterRegistry registry) { this.registry = registry; }
void recordPayment(String method, boolean success) {
registry.counter("payments.processed",
"method", method, // card, paypal, bank: bounded
"outcome", success ? "success" : "failure") // two values: bounded
.increment();
}
}This produces at most methods × 2 series. Now compare:
registry.counter("payments.processed", "customer", customerId).increment(); // one series per customerWith a million customers, that is a million series for one counter. Timers are worse: a Prometheus timer with histogram buckets exports one series per bucket per tag combination, often 20 to 70 series each. Multiply that by every unbounded tag and memory in Prometheus, and in your own JVM, grows without limit. Each meter is held in the MeterRegistry forever unless removed, so the application heap grows too.
Spring Boot’s http.server.requests timer is tagged with method, uri, status, outcome and exception. The uri tag is the matched route pattern, not the path. A request to /orders/9481 is recorded as uri="/orders/{id}". Requests that match no handler are recorded as NOT_FOUND or REDIRECTION, so scanners probing random paths cannot inflate it.
Step-by-step walkthrough
Step 1: List the questions the metric must answer
“Is the payment success rate dropping, and for which method?” needs method and outcome. It does not need customer. Write the dashboard queries first, then add only the tags those queries group by.
Step 2: Estimate the series count before shipping
Multiply the cardinality of each tag. method (4) × outcome (2) × region (3) = 24 series. If the meter is a timer with percentile histograms enabled, multiply by the bucket count. Anything that estimates to “depends on traffic” is unbounded.
Step 3: Normalize values that come from input
When a tag value must come from a request, map it into a fixed set first:
String plan = switch (account.plan()) {
case FREE, TRIAL -> "free";
case PRO -> "pro";
case ENTERPRISE -> "enterprise";
};
registry.counter("signups", "plan", plan).increment();For outbound HTTP clients, always pass a URI template so the client metric records the template:
restClient.get().uri("/customers/{id}/orders", customerId).retrieve().body(Orders.class);
// tagged uri="/customers/{id}/orders", not the expanded pathBuilding the URL by concatenation ("/customers/" + customerId + "/orders") loses the template and every customer becomes a tag value.
Step 4: Install a guard rail
A MeterFilter can cap the distinct values of a tag for a metric name prefix and deny further meters after the cap:
@Configuration
class MetricsGuardRails {
@Bean
MeterFilter capHttpClientUris() {
return MeterFilter.maximumAllowableTags(
"http.client.requests", "uri", 100,
MeterFilter.deny()); // drop new meters beyond 100 distinct uri values
}
}Spring Boot also offers management.metrics.web.client.max-uri-tags and management.metrics.web.server.max-uri-tags (default 100), which apply this same filter for HTTP metrics and log a warning once the cap is reached.
Worked scenario
A team adds a timer to understand slow searches:
Timer.builder("search.latency")
.tag("query", request.query()) // raw user search text
.tag("tenant", request.tenantId()) // 40,000 tenants
.publishPercentileHistogram()
.register(registry)
.record(elapsed);Within a week, Prometheus memory grows from 6 GB to 40 GB and the server is restarted by its OOM killer twice. The application’s heap also climbs because the registry holds every unique timer. Grafana panels using search_latency_seconds_bucket time out. The query tag alone has hundreds of thousands of values, and each one carries dozens of histogram buckets.
The fix keeps the question (“which kinds of searches are slow?”) and drops the unbounded detail:
Timer.builder("search.latency")
.tag("type", request.hasFilters() ? "filtered" : "plain")
.tag("tier", tenantTier(request.tenantId())) // free, standard, premium
.publishPercentileHistogram()
.register(registry)
.record(elapsed);
log.info("search completed tenant={} ms={} query_length={}", request.tenantId(), elapsed.toMillis(), request.query().length());Six series combinations replace hundreds of thousands. Per-tenant investigation moves to logs and traces, where a trace ID links the slow request to its query. Prometheus memory returns to normal after the old series age out of retention.
Common mistake
- “Tags are free, add everything for debugging.” Each distinct combination is stored and indexed separately.
- Tagging with exception messages. Messages contain IDs and timestamps. Tag the exception class name, which Boot already does.
- Using
request.getRequestURI()as a tag in a custom filter instead of the matched pattern. - Assuming a bounded tag stays bounded. “Country” is bounded; “city” from user input is not.
- Enabling percentile histograms on every timer. Histograms multiply series; enable them on the few SLO-critical timers.
- Fixing only the dashboard. Series still exist in storage until retention expires and in the JVM until restart.
Verify the behavior
Test that the meter count stays flat as input varies:
class SearchMetricsTest {
@Test
void cardinalityDoesNotGrowWithTenants() {
SimpleMeterRegistry registry = new SimpleMeterRegistry();
SearchMetrics metrics = new SearchMetrics(registry);
for (int tenant = 0; tenant < 1_000; tenant++) {
metrics.record(new SearchRequest("tenant-" + tenant, "shoes " + tenant), Duration.ofMillis(20));
}
assertThat(registry.find("search.latency").timers()).hasSizeLessThanOrEqualTo(6);
}
}In a running service, count series directly in Prometheus with count by (__name__)({__name__=~"search_latency.*"}), or check the /api/v1/status/tsdb endpoint, which lists the metrics and labels with the highest series counts.
Follow-up questions
What is the difference between a metric and a log for this purpose? Metrics aggregate cheaply over bounded dimensions; logs and traces carry per-event detail. Use each for what it is good at.
How do exemplars help? Prometheus exemplars attach a trace ID to a histogram sample, so you can jump from a slow bucket to a specific trace without adding a high-cardinality tag.
How do you add common tags such as application and region? management.metrics.tags.application=checkout adds a constant tag to every meter, which is bounded and useful.
How would you remove a bad meter at runtime? registry.remove(meter) removes it from Micrometer, but you should fix the code; Prometheus will still hold existing series until retention.
Interview exercise
Product asks for a dashboard of API errors per customer for 20,000 business customers. Someone proposes tagging http.server.requests with customer_id. What do you propose instead?
Answer and reasoning
Tagging the server timer with customer ID multiplies every existing combination of method, URI, status and histogram bucket by 20,000, which is likely millions of series. Instead, keep metrics for fleet-wide health (error rate by URI template and status), and answer per-customer questions from logs or traces, where customer ID is just a field: a structured log line per failed request feeds a log analytics query or a per-customer report. If a small number of strategic customers need live alerting, tag a separate low-volume counter with a bounded customer_segment or an allow-list of named accounts, defaulting everything else to other. The reasoning: match the data to the tool by cardinality, and estimate series counts before adding any tag.