Kubernetes Autoscaling and Meaningful Metrics

An autoscaler is a feedback loop. It reads a number, compares it with a target, and changes the replica count. Everything interesting about autoscaling is in the choice of that number. If the metric tracks the thing that actually limits your throughput, adding pods helps. If it tracks something incidental, the autoscaler will add pods that make no difference, or worse, add pods that make the real bottleneck slower.

This guide covers the HorizontalPodAutoscaler (HPA, autoscaling/v2) on Kubernetes 1.30 and later. The manifests below are illustrative and weren’t applied to a live cluster; the arithmetic was checked by hand.

What the HPA actually computes

The HPA controller runs every 15 seconds by default. For each metric it computes a desired replica count with one formula:

desiredReplicas = ceil( currentReplicas × currentMetricValue / desiredMetricValue )

If you run 4 pods at an average 90% CPU utilisation with a 60% target, you get ceil(4 × 90 / 60) = 6. If you configure several metrics, it computes a number for each and takes the largest. There’s a default tolerance of 10%: if the ratio current / desired is between 0.9 and 1.1, nothing happens, which stops the replica count twitching on noise.

Two defaults in behavior shape how fast it moves. Scale-up has no stabilisation window, so a spike is answered within one or two loops. Scale-down uses a 300-second stabilisation window: the controller picks the highest recommendation from the last five minutes, so a brief dip doesn’t remove pods you’ll need again in a minute. Interviewers like asking why scale-down is slower than scale-up, and the honest answer is that removing capacity too early hurts users while keeping it a few extra minutes only costs money.

The formula is dumb on purpose. It assumes that doubling replicas halves the per-replica metric. That assumption holds for CPU-bound stateless work with even load balancing. It breaks the moment the work is limited by something outside the pod.

Why CPU is the default and where it misleads

CPU is the default because metrics-server provides it with no extra setup, and for a stateless HTTP API that does real computation per request, it’s a fair proxy for load. There’s one detail people miss: averageUtilization is a percentage of the container’s CPU request, not of the node or the limit. A pod requesting 200m and using 300m reports 150%. If the request is set far too low, every pod looks overloaded and the HPA pins at maxReplicas; set it far too high and the HPA never scales. The scaling signal is only as good as your requests (see requests and limits).

CPU misleads in three common cases:

• I/O-bound services. A service that spends most of its time waiting on a database or an upstream API can have 20% CPU while requests queue up and latency triples.
• Queue consumers. A worker that pulls one message, calls a slow partner API and sleeps on the response shows low CPU regardless of how far behind it is.
• Memory-bound or JVM services. Garbage collection can burn CPU without reflecting user load, and memory doesn’t fall when you add replicas, so memory is rarely a good HPA target either.

The question to ask before picking a metric is: “If I double the pods, does this number halve?” For CPU on a stateless API, roughly yes. For worker CPU while the database is saturated, no.

A worked example: the order-export workers

Picture an order-export Deployment that reads messages from a queue, loads each order from Postgres and writes a file to object storage. Each worker processes about 10 messages a second when the database is healthy. The current HPA scales on CPU at 70%.

On a Monday morning the backlog climbs to 9,000 messages. Worker CPU sits at 25%, because each worker is mostly waiting on network calls. The HPA sees 25% against a 70% target and keeps the minimum of 3 replicas. Users wait about five minutes for exports (9,000 messages at 30 per second is 300 seconds), and the autoscaler does nothing. CPU was the wrong signal.

Backlog per replica is the better signal. If you want the queue drained within about 30 seconds, and each worker handles 10 messages per second, each worker should own at most about 300 messages. With a metrics adapter (prometheus-adapter or KEDA) exposing the queue depth as an external metric, the HPA looks like this:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-export
  namespace: exports
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-export
  minReplicas: 3
  maxReplicas: 28
  metrics:
    - type: External
      external:
        metric:
          name: queue_messages_ready
          selector:
            matchLabels:
              queue: order-export
        target:
          type: AverageValue
          averageValue: "300"
  behavior:
    scaleUp:
      policies:
        - type: Pods
          value: 8
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300

With AverageValue, the HPA divides the external metric by the current replica count. At 9,000 messages and 3 replicas, each replica “owns” 3,000, and the formula gives ceil(3 × 3000 / 300) = 30. That’s above maxReplicas, so the HPA asks for 28, and the scale-up policy means it gets there in steps of at most 8 pods per minute.

Where maxReplicas comes from

The number 28 isn’t arbitrary. Each worker opens a pool of 5 Postgres connections. The database allows 200 connections, and the checkout API needs 60 of them at peak. That leaves 140 for exports, and 140 / 5 = 28 workers. Above 28, new workers fail to get connections, or they get them by starving the checkout API.

This is the failure the original version of this article warned about. When the shared dependency is the bottleneck, more replicas increase contention. Lock waits grow, each query slows down, throughput per worker drops, and the backlog can grow faster than before you scaled. A scaling signal that only looks at your own pods can’t see this. Two guards help:

1. Derive maxReplicas from the dependency’s budget, as above, and write the arithmetic in a comment or runbook so the next person doesn’t raise it casually.
2. Watch a dependency signal next to the scaling signal: database CPU, connection count, p95 query time. If those saturate while replicas climb, the right fix is in the database (an index, a read replica, batching) rather than in the HPA.

Gotcha

An autoscaler that hits maxReplicas and stays there isn’t working; it’s telling you the real limit is somewhere else. Alert on “pinned at max for 15 minutes” rather than raising the ceiling.

Signals for different workloads

There isn’t one correct metric, but there are reasonable defaults:

Latency is tempting as a scaling metric, but use it carefully. Latency rises both when you need more pods and when a dependency is slow; scaling on it will add pods during a database incident, which is exactly when you don’t want them. Use it to verify that scaling helped, not as the trigger.

Also remember the layers around the HPA. The HPA adds pods; if there’s no room for them on existing nodes they sit Pending until the Cluster Autoscaler or Karpenter adds nodes, which can take a minute or more. Scale-from-zero isn’t possible with a plain HPA (minReplicas must be at least 1 unless an alpha feature gate is on); KEDA handles that case.

Try it yourself: an interview question

“Your payment-reconciliation workers scale on CPU at 60%. During month-end the queue reaches 50,000 messages, CPU stays around 30%, and replicas stay at the minimum of 4. Someone suggests lowering the CPU target to 20%. What would you do instead?”

Answer

Lowering the target to 20% would make the HPA scale, but for the wrong reason: it would add pods whenever CPU drifts up, including when the queue is empty, and it still wouldn’t track the backlog. The low CPU tells you the workers are waiting on I/O, so CPU isn’t measuring how far behind you are.

I’d switch the HPA to an external metric for backlog per replica (or the age of the oldest message, if the SLA is about time). I’d pick the per-replica target from measured throughput: if one worker reconciles 20 messages a second and month-end needs the queue cleared within 10 minutes, each worker can own about 12,000 messages, so 50,000 messages needs at least 5 workers, and a smaller target gives headroom. Then I’d set maxReplicas from whatever the workers share: database connections, the bank’s API rate limit, or both. Finally I’d watch the dependency’s latency and error rate during the next month-end; if throughput per worker drops as replicas rise, the dependency is the real ceiling and the fix belongs there.

Further reading

• Kubernetes interview questions
• Kubernetes MCQs
• Horizontal Pod Autoscaling in detail
• Requests, limits and scheduling
• Official docs: Horizontal Pod Autoscaling