Teams new to CloudWatch tend to end up in one of two places: no alarms at all, or a hundred alarms that fire so often everyone mutes the channel. Both come from the same gap. An alarm isn’t “CPU is high”. It’s a specific question, asked on a schedule, about a specific number, and the answer depends on details like the statistic and what happens when no data arrives. Knowing those details lets you build alarms that mean something when they fire.
What a metric datapoint actually is
A CloudWatch metric is identified by a namespace (AWS/ApplicationELB), a name (TargetResponseTime) and dimensions (LoadBalancer=app/shop/50dc6c495c0c9188). Every distinct combination of dimensions is a separate metric. A metric published with an InstanceId dimension can’t be queried “across all instances” unless it’s also published without that dimension or you aggregate with metric math or a search expression.
When you graph or alarm on a metric, you choose a period (60 seconds, 5 minutes) and a statistic that summarises all the raw values in each period:
Average,Minimum,Maximum,SumandSampleCount.- Percentiles like
p50,p95,p99, and trimmed means.
The statistic matters more than people expect. Suppose 1,000 requests in a minute: 990 take 80 ms and 10 take 6 seconds. The average is about 140 ms, which looks healthy. The p99 is 6 seconds, which means one in a hundred users is waiting long enough to give up. An alarm on average latency would never fire for this. Alarm on p99 (or p95) for latency.
Counts have the opposite trap. HTTPCode_Target_5XX_Count with Sum going from 5 to 50 might be a disaster at 1,000 requests a minute or noise at 100,000. Alarm on a rate using metric math, such as 100 * errors / requests, so the threshold means the same thing at 3 a.m. and at peak.
How an alarm decides
An alarm compares a statistic to a threshold every period. Three settings decide when it changes state:
- Period: the length of each datapoint.
- EvaluationPeriods (N): how many recent periods to look at.
- DatapointsToAlarm (M): how many of those N must breach to go into
ALARM.
“3 out of 5 one-minute periods above 5%” fires on a sustained problem but not a single spike. “1 out of 1” fires on any spike and flaps. Alarm states are OK, ALARM and INSUFFICIENT_DATA.
The fourth setting is TreatMissingData, which says what to do with periods where no datapoint arrived: missing (the default: leave it out of the evaluation), notBreaching (count it as good), breaching (count it as bad), or ignore (keep the current state).
Here’s a simplified model of that evaluation in Python, run against a minute-by-minute 5xx error rate for a low-traffic internal API. The last few minutes have no data because nobody called the API. Real CloudWatch has extra nuances (for instance, it can look back further for datapoints when some are missing), so treat this as a sketch of the logic, not the exact algorithm:
# Simplified CloudWatch alarm evaluation: "DatapointsToAlarm of EvaluationPeriods" over one-minute
# datapoints, with a TreatMissingData setting. None means no datapoint was published that minute.
def evaluate(points, threshold, m, n, missing):
states = []
for i in range(n, len(points) + 1):
window = points[i - n:i]
breaching = 0
known = 0
for p in window:
if p is None:
if missing == "breaching": breaching += 1; known += 1
elif missing == "notBreaching": known += 1
continue # "missing"/"ignore": skip it
known += 1
breaching += p > threshold
if breaching >= m: states.append("ALARM")
elif known == 0: states.append("INSUFFICIENT_DATA")
else: states.append("OK")
return states
# 5xx error rate (%) per minute for a low-traffic internal API
error_rate = [0.2, 9.0, 0.1, 0.3, 6.5, 7.2, 8.1, None, None, None, 0.4]
print("1 of 1 :", evaluate(error_rate, 5, 1, 1, "missing"))
print("3 of 5 :", evaluate(error_rate, 5, 3, 5, "missing"))
print("3 of 5, missing=breaching :", evaluate(error_rate, 5, 3, 5, "breaching")[-4:])
print("3 of 5, missing=notBreaching:", evaluate(error_rate, 5, 3, 5, "notBreaching")[-4:])Output from python3 alarm.py:
1 of 1 : ['OK', 'ALARM', 'OK', 'OK', 'ALARM', 'ALARM', 'ALARM', 'INSUFFICIENT_DATA', 'INSUFFICIENT_DATA', 'INSUFFICIENT_DATA', 'OK']
3 of 5 : ['OK', 'ALARM', 'ALARM', 'ALARM', 'ALARM', 'OK', 'OK']
3 of 5, missing=breaching : ['ALARM', 'ALARM', 'ALARM', 'ALARM']
3 of 5, missing=notBreaching: ['ALARM', 'ALARM', 'OK', 'OK']Read it line by line. “1 of 1” pages for the lone 9% spike in minute two, which was a single failed request out of a dozen; that’s the alarm people learn to ignore. It also drops into INSUFFICIENT_DATA whenever traffic stops, which often triggers its own notifications. “3 of 5” ignores the spike and fires for the sustained run of errors in minutes five to seven, which is the real incident.
The last two lines show why TreatMissingData is a decision, not a default. With breaching, the quiet period keeps the alarm firing after the errors stopped, because silence is counted as failure. With notBreaching, it recovers once the bad minutes leave the window.
Which is right depends on what silence means:
- For an error rate on a low-traffic API, no data means no requests, so
notBreachingis right. - For a heartbeat metric that a batch job publishes every run, no data means the job didn’t run, so
breachingis right. Otherwise a dead job never alarms. - For CPU on an instance that was terminated,
missingorignoreavoids alarms for things that no longer exist.
Symptoms page, causes explain
The alarms that should wake someone up describe what users experience: error rate above a threshold, p99 latency above the SLO, queue age (ApproximateAgeOfOldestMessage) above what the business tolerates, a heartbeat missing. These are symptoms. They’re few, stable and meaningful.
Cause metrics (CPU, memory, connection counts, disk queue depth) are for dashboards and lower-urgency tickets. High CPU with healthy latency isn’t an incident; it might be good utilisation. If you page on causes, you get pages that don’t need action and miss problems that don’t move them.
Composite alarms help reduce noise: an alarm whose rule combines others, like ALARM(error-rate) AND ALARM(latency-p99), or one that suppresses child alarms during a known maintenance window.
Logs: where the explanation lives
Metrics tell you that something is wrong; logs tell you which request and why. Two habits make CloudWatch Logs useful:
Log structured JSON, with the request ID, route, status and duration as fields. Then CloudWatch Logs Insights can query them directly. To find which routes produced 5xx responses in the last hour (illustrative query; field names depend on your logs):
fields @timestamp, route, status, durationMs, requestId
| filter status >= 500
| stats count(*) as errors, pct(durationMs, 99) as p99 by route
| sort errors desc
| limit 10Publish metrics from logs where it helps. The Embedded Metric Format lets an application write a JSON log line that CloudWatch extracts as a metric, with no extra API calls. Metric filters do the same for existing log patterns, like counting lines with "level":"ERROR". Be careful with dimensions: each unique dimension combination is a billed custom metric, so a dimension like userId can create millions of metrics and a large bill.
Also set a retention period on every log group. The default is to keep logs forever, which is rarely what anyone wants to pay for.
Interview practice: the alarm nobody trusts
A team has a CloudWatch alarm on average TargetResponseTime above 1 second, 1 out of 1 one-minute periods, with TreatMissingData left at its default. It fires several times a night, always resolves on its own, and was muted months ago. Last week, customers complained about timeouts for two hours and the alarm didn’t fire at all. Redesign it.
Answer
The alarm measures the wrong statistic with the wrong sensitivity. “1 of 1” means any single slow minute fires it, and at night, with few requests, one slow request can push the average over a second, which explains the noisy pages. During the real incident, a fraction of requests were timing out while most stayed fast, so the average stayed under a second. Averages hide tails.
I’d replace it with a p99 latency alarm at a threshold tied to the service’s latency objective, say p99 above 1.5 seconds for 3 out of 5 one-minute periods. Alongside it, an error-rate alarm using metric math (100 * HTTPCode_Target_5XX_Count / RequestCount, plus ELB 5xx if the load balancer itself returns 504s on timeouts) for 3 out of 5 periods. For both, TreatMissingData should be notBreaching, since no traffic at night isn’t a failure. If night traffic is so low that p99 is meaningless, I’d add a minimum request count condition with metric math, or a composite alarm requiring both latency and errors.
Finally, every alarm that pages should link to a dashboard and a runbook, and the team should review alarms that fire without action, because a muted alarm is worse than none: it gives false confidence.