A canary release sends a small slice of real traffic to a new version while the rest stays on the current one. If the new version misbehaves, only that slice is affected and you roll it back. Most candidates can explain that much. The interview question that follows is harder: “How do you decide the canary is healthy?” Answers like “we watch the dashboards” or “we check there are no errors” are where the conversation usually turns.
The deployment mechanics are the easy part. Load balancers, service meshes and tools like Argo Rollouts or Flagger can split traffic by percentage. The hard part is the decision: which signals you compare, against what, for how long, and what threshold means stop.
A rollout plan you can actually evaluate
A typical plan for a stateless HTTP service:
- Deploy version 1.5 alongside 1.4. Route 5% of traffic to it.
- Hold for a fixed period, comparing 1.5 with 1.4 on agreed metrics.
- If it passes, go to 25%, hold, compare again. Then 50%, then 100%.
- If any check fails at any step, route all traffic back to 1.4 automatically and alert a human.
The important word is “agreed.” Write the pass and fail rules down before the rollout begins, ideally as configuration the deployment tool evaluates. If people decide while looking at a graph, a 20% latency regression becomes “probably noise” when the release is late, and a single spike becomes “let’s roll back to be safe” when it isn’t.
Choosing the metrics
Start with what users experience, then add what predicts trouble:
- Error rate: the fraction of requests that fail, typically 5xx responses, and for some services certain 4xx codes that indicate a bug (a sudden rise in 400s often means the new version rejects valid input).
- Latency at high percentiles: p95 and p99, not the mean. A version that’s fast for most requests and slow for 2% of them barely moves the average.
- Saturation: CPU, memory, connection pool usage, queue depth. A memory leak shows up here long before it causes errors.
- A business signal: checkouts completed, messages sent, searches with results. Some bugs return HTTP 200 with the wrong answer, and only this catches them.
Each metric needs a comparison and a threshold, for example “canary p99 latency no more than 10% above baseline p99” or “canary error rate not significantly higher than baseline.”
A trap interviewers look for: aggregating percentiles. If you have three canary pods, averaging their three p99 values doesn’t give you the canary’s p99. Compute the percentile from the combined histogram, which Prometheus does with histogram_quantile over summed buckets.
Compare with a live baseline, not with yesterday
Comparing the canary’s error rate with last Tuesday’s is tempting and misleading. Traffic mix, time of day, a partner’s outage and a dozen other things change between then and now. The fair comparison is against the current version during the same minutes.
Even that has a bias. The canary pods are freshly started: cold caches, JIT compilers that haven’t warmed up, connection pools still filling. Comparing them with long-running production pods can make a good release look slow. Netflix’s Kayenta, which Spinnaker uses for automated canary analysis, handles this by starting a fresh baseline group running the old version at the same size as the canary, and comparing the two new groups with each other. If you can’t do that, at least skip the first few minutes after the canary starts before you begin measuring.
Enough traffic to tell the difference
Suppose the baseline’s error rate is 0.20% and the canary’s is 0.30%. Is that a regression? It depends on how many requests you’ve seen. A small one-sided two-proportion test makes this concrete:
from math import sqrt, erf
def compare(canary_errors, canary_total, base_errors, base_total):
p1 = canary_errors / canary_total
p2 = base_errors / base_total
pooled = (canary_errors + base_errors) / (canary_total + base_total)
se = sqrt(pooled * (1 - pooled) * (1 / canary_total + 1 / base_total))
z = (p1 - p2) / se
p_value = 1 - 0.5 * (1 + erf(z / sqrt(2))) # one-sided: is the canary worse?
return f"canary {p1:.2%} baseline {p2:.2%} z={z:.2f} p={p_value:.3f}"
print("10 min:", compare(6, 2_000, 76, 38_000))
print("60 min:", compare(36, 12_000, 456, 228_000))10 min: canary 0.30% baseline 0.20% z=0.96 p=0.168
60 min: canary 0.30% baseline 0.20% z=2.36 p=0.009A service handling 4,000 requests a minute sends the canary about 200 a minute at 5%, so ten minutes gives it 2,000 requests and six errors. A difference that size happens by chance about one time in six, so you can’t call it. After an hour, the same rates are very unlikely to be chance, and the canary is genuinely worse.
So the hold time per step should come from traffic volume, not habit. A busy API might get a clear answer in five minutes. An internal service with a few requests a minute may never collect enough at 5%, and for that service a canary by percentage is mostly theatre: a larger slice, a longer hold or synthetic traffic would serve it better. Automated analysis tools apply statistical tests like this one (Kayenta uses a Mann-Whitney U test) for exactly this reason.
Routing: who ends up on the canary
Random per-request routing is simplest, but it means one user can bounce between 1.4 and 1.5 within a page load. If the versions render different HTML or use a changed API contract, that causes errors that belong to neither version. Sticky routing, keyed by user ID or a cookie, keeps each user on one version. It also makes the user-facing impact clearer: 5% of users had the new version, rather than 5% of every user’s requests.
Some teams route internal staff or a low-risk region first. That’s useful as an early step, but staff traffic isn’t representative, so don’t let it replace a stage with real customers.
What a canary can’t catch
Canaries test code paths that a slice of real traffic exercises within the hold period. Several things slip through:
- Shared state changes. A database migration runs once, for everybody. If 1.5’s migration drops a column 1.4 still reads, routing traffic back to 1.4 doesn’t help. Schema changes need the expand and contract pattern, covered in the rollback guide linked below.
- Rare paths. A bug in the yearly-invoice job won’t appear in a 30-minute canary.
- Slow problems. A leak that takes six hours to exhaust memory passes every stage of a two-hour rollout. Keep watching saturation after you reach 100%.
- Background workers and consumers. Queue consumers don’t get traffic from the load balancer, so a percentage split doesn’t apply. Run a small number of new-version consumers instead, and watch for poison messages.
Interview practice
Your team’s canary process is: deploy to 10% of pods, wait 15 minutes, and if the error-rate dashboard “looks normal”, go to 100%. Last month a release passed and then caused a 40% increase in p99 latency at full traffic, which took two hours to notice. Propose a better process and explain each change.
Answer
The current process has no stated metric threshold, measures only errors, decides by eye and stops observing at the moment risk is highest.
First, write the rules down and automate them. For this service: error rate not significantly above baseline, p99 latency within 10% of baseline p99 computed from merged histograms, and memory growth within an agreed bound. The rollout tool evaluates these at each step and rolls back on failure without waiting for a person.
Second, compare against a baseline running the old version during the same window, ideally freshly started pods of the same count, so cold-start effects cancel out. Skip the first few minutes of each step.
Third, add steps: 5%, 25%, 50%, 100%. The latency regression was probably load-dependent (perhaps a lock or a connection pool that’s fine at low concurrency). A 50% step at real load would likely have exposed it, and only half of traffic would have been affected.
Fourth, base hold times on traffic volume so each comparison has enough samples to mean something. Finally, keep the same checks running for an hour after 100% with automatic alerting, because leaks and load effects appear late. With those in place, the two hours to notice becomes minutes, and the rollback doesn’t depend on someone watching.