AWS Auto Scaling and Dependency Limits

Auto Scaling is easy to switch on and easy to misunderstand. The pitch is that the fleet grows with demand, so you stop worrying about capacity. What actually happens is that the fleet grows with a metric, after a delay, and the new instances send more traffic to everything behind them. Each of those three parts can turn a traffic spike into an outage, and senior interviews tend to probe all three.

What a scaling policy really does

An EC2 Auto Scaling group keeps a number of instances (its desired capacity) between a minimum and a maximum. Policies change the desired capacity:

• Target tracking keeps a metric near a target, like a thermostat: average CPU at 50%, or ALBRequestCountPerTarget at 1,000. AWS creates and manages the CloudWatch alarms for you. It’s the right default.
• Step scaling adds or removes set amounts at thresholds you define, useful when you want aggressive steps for large breaches.
• Scheduled scaling changes capacity at known times (9:00 on weekdays, before a sale).
• Predictive scaling forecasts load from history and scales ahead of it, which helps with daily patterns.

Target tracking deliberately scales out faster than it scales in. It reacts to a few minutes of high readings, but waits for a longer period of low readings before removing capacity, to avoid flapping.

Scaling is slower than the spike

Count the steps between a surge starting and a new instance serving traffic:

1. The metric has to reflect the surge. Standard EC2 metrics arrive at one-minute granularity with detailed monitoring, five without.
2. The alarm needs several data points above target before it fires (target tracking’s scale-out alarm evaluates a few minutes of data).
3. The group launches instances. Booting, pulling a container image, starting the JVM and warming caches can take several minutes.
4. The load balancer’s health checks must pass before it routes traffic there.
5. The instance warmup setting tells Auto Scaling to ignore a new instance’s metrics until it’s ready, so a half-started instance doesn’t drag the average down and stop further scaling.

That’s often 5 to 10 minutes from spike to relief. Traffic that doubles in 30 seconds isn’t something reactive scaling can catch. The tools for that gap are headroom (run the target at 50% CPU, not 80%), scheduled or predictive scaling for known events, faster boot (pre-baked AMIs, smaller images), warm pools of pre-initialised instances, and a queue in front of work that doesn’t need an immediate answer.

Choosing the metric that measures the bottleneck

CPU is the default because it’s always there, but it’s only the right signal when the instances are CPU-bound. Some common mismatches:

• A service that mostly waits on a database or HTTP calls shows low CPU while requests pile up. It never scales.
• A service whose bottleneck is the database shows high CPU from retries and serialisation work. It scales out, sends more queries, and the database gets slower.
• A queue worker’s CPU depends on message mix, not on how far behind it is.

For web traffic behind an Application Load Balancer, ALBRequestCountPerTarget is usually closer to the truth: you load test one instance, find the request rate it handles at acceptable latency, and target a fraction of that.

For queue workers, the AWS-recommended signal is backlog per instance: visible messages divided by in-service instances. You pick the target from how long a message may wait. Here’s the arithmetic for a payment worker in Python, including the dependency limit that the next section is about:

import math
# Queue workers: each message takes 0.5 s, so one instance with 4 worker threads does 8 msg/s.
per_instance_rate = 4 / 0.5
acceptable_latency_s = 60
backlog_target = per_instance_rate * acceptable_latency_s
print(f"backlog-per-instance target: {backlog_target:.0f} messages")

# The downstream payment API allows 400 requests/s for our account.
api_limit = 400
max_useful = math.floor(api_limit / per_instance_rate)
print(f"instances the payment API can feed: {max_useful}")

# What target tracking would ask for at different backlogs.
for backlog in (2_000, 20_000, 80_000):
    desired = math.ceil(backlog / backlog_target)
    capped = min(desired, max_useful)
    drain_min = backlog / (capped * per_instance_rate) / 60
    print(f"backlog {backlog:>6}: wants {desired:>3} instances, cap {capped:>2}, drains in ~{drain_min:.1f} min")

Output from python3 scaling.py:

backlog-per-instance target: 480 messages
instances the payment API can feed: 50
backlog   2000: wants   5 instances, cap  5, drains in ~0.8 min
backlog  20000: wants 42 instances, cap 42, drains in ~1.0 min
backlog  80000: wants 167 instances, cap 50, drains in ~3.3 min

(The drain times ignore messages that keep arriving, so treat them as lower bounds.) The target tracking configuration uses metric math to compute backlog per instance. This is illustrative, with made-up names, and wasn’t applied to an account:

{
  "TargetValue": 480,
  "CustomizedMetricSpecification": {
    "Metrics": [
      { "Id": "m1", "ReturnData": false,
        "MetricStat": { "Stat": "Sum", "Metric": { "Namespace": "AWS/SQS", "MetricName": "ApproximateNumberOfMessagesVisible",
          "Dimensions": [{ "Name": "QueueName", "Value": "payments" }] } } },
      { "Id": "m2", "ReturnData": false,
        "MetricStat": { "Stat": "Average", "Metric": { "Namespace": "AWS/AutoScaling", "MetricName": "GroupInServiceInstances",
          "Dimensions": [{ "Name": "AutoScalingGroupName", "Value": "payment-workers" }] } } },
      { "Id": "e1", "Expression": "m1 / m2", "Label": "Backlog per instance", "ReturnData": true }
    ]
  }
}

GroupInServiceInstances only exists if group metrics collection is turned on for the Auto Scaling group, which is a common reason this policy silently does nothing.

Every instance you add is load somewhere else

The last row of that output is the important one. At an 80,000 message backlog, the policy wants 167 instances, but the payment API allows 400 requests per second for the account, which 50 instances already saturate. Instances 51 to 167 add nothing except 429 Too Many Requests responses, retries that make the rate limit worse, and cost.

Every scaling tier has dependencies with their own ceilings:

• Databases: connection limits and CPU. More app instances means more pooled connections (instances times pool size), which is often the first limit hit.
• Third-party APIs: rate limits per account or per key.
• Other AWS quotas: Lambda concurrency shared by the account, DynamoDB partition throughput on a hot key, NAT gateway connections to a single destination, KMS request rates.
• Your own downstream services, which might not autoscale as fast, or at all.

So the group’s maximum size isn’t a cost knob only. It’s a statement about how much load the dependencies can take, and it should be computed from them, like the max_useful = 50 line above. When the backlog exceeds what the capped fleet can drain, the queue absorbs it, which is fine for payments that can take three minutes. For synchronous traffic you need a different answer at the edge: rate limiting, load shedding with 503 and Retry-After, or degrading optional features.

Scaling in, health checks and other sharp edges

Scale-in is where requests get dropped. Turn on connection draining (the target group’s deregistration delay) so in-flight requests finish, and for queue workers use lifecycle hooks or scale-in protection so an instance isn’t terminated halfway through a job.

The health check grace period needs to be longer than boot time. If it’s shorter, Auto Scaling marks new instances unhealthy before they’ve started, replaces them, and the group loops through launches without ever adding capacity. Using ELB health checks (not just EC2 status checks) means instances that are running but not serving get replaced, but make the health endpoint check the process itself, not its dependencies. A health check that queries the database marks every instance unhealthy during a database blip, and Auto Scaling replaces the whole fleet at the worst moment.

Interview practice: scaling made it worse

An API on EC2 behind an ALB scales on average CPU at 60%. During a sale, latency rises, CPU rises, the group scales from 12 to 48 instances, and latency gets worse. The RDS instance shows CPU at 98% and connections near the maximum. What happened, and what would you change?

Answer

The bottleneck was the database, not the web tier. As database queries slowed, app instances spent more CPU on retries, timeouts and request queuing, so the app’s CPU metric rose and target tracking added instances. Each new instance opened its own connection pool and sent more queries, pushing the database further into saturation. The scaling policy was measuring a symptom and feeding the cause.

Short term, I’d cap the group’s maximum at what the database can serve (from connection limits and load test results) so scaling can’t amplify the problem, and shed load at the edge with rate limiting so the requests we do accept succeed. Then fix the bottleneck: look at the slow queries, add read replicas or caching for read-heavy endpoints, reduce pool sizes per instance so the fleet’s total connections fit, or add RDS Proxy.

Longer term, scale on a metric closer to real capacity, like ALBRequestCountPerTarget with a target from load testing, alarm on database CPU and connections directly, and run a load test that includes the database so the next limit is found in a test rather than during a sale.

Keep going

• AWS interview questions
• AWS MCQs
• AWS RDS connections and database capacity
• AWS SQS delivery and visibility timeouts
• Target tracking scaling policies (AWS docs)