Microservice Configuration and Safe Rollouts

Plenty of large outages weren’t caused by a code deploy. Someone changed a timeout, a routing rule, a feature flag or a rate limit, and that change went to every instance in every region within seconds. Code changes usually go through review, tests, a canary and a gradual rollout. Config changes, historically, went through a text box.

The interview version of this topic is “how do you roll out a risky change safely”, and the strongest answers treat configuration exactly like code: versioned, validated, staged and reversible. The rest of this note works through how.

Why config is riskier than it looks

Three properties make config dangerous:

• It’s fast. A dynamic config system can push a value to 2,000 instances in under a minute. That’s great for a kill switch and terrible for a typo.
• It skips the safety net. Unit tests run against code. Nobody’s test suite runs against the value someone types into the flag dashboard at 4pm on Friday.
• It’s global by default. A code deploy rolls through instances one batch at a time. A config change often lands everywhere at once unless you design otherwise.

So the goal is to put back the protections code has: something that checks the change, something that limits how much of the system sees it at first, and something that reverts it quickly.

Two ways to stage a change

There are two different things people mean by “gradual rollout”, and it’s worth separating them in an interview.

Staging by infrastructure. Apply the new config to one instance, then one availability zone, then one region, then everywhere. This is the right approach for operational settings: connection pool sizes, timeouts, JVM flags, routing weights. The blast radius is “the traffic that hits these instances”, which is random with respect to users.

Staging by user cohort. Turn a feature on for 1% of users, then 5%, then 25%. This is the right approach for product behaviour: a new pricing algorithm, a new checkout flow. Here each user should get a consistent experience, and you want to compare the cohort’s metrics against everyone else’s.

Cohort rollouts need a stable assignment. The standard technique is to hash the user id together with the flag name and map the result to a bucket from 0 to 9,999:

import { createHash } from 'node:crypto';

const bucket = (flag, userId) =>
  createHash('sha256').update(`${flag}:${userId}`).digest().readUInt32BE(0) % 10_000;
const enabled = (flag, userId, percent) => bucket(flag, userId) < percent * 100;

Running this over 100,000 synthetic users, plus a comparison with the naive approach (Math.random() < 0.05 on every request), printed:

5%: 4976 users, 20%: 19933 users, 5% cohort still on at 20%: 4976/4976
overlap between two 5% flags: 239 users (independent would be ~250)
Math.random per request: 95/1000 users saw the flag change between two page loads

(The last number varies from run to run; around 9.5% is expected, since two independent 5% draws differ with probability 2 × 0.05 × 0.95.)

Each line answers a specific design question:

1. Is the percentage accurate? 4,976 out of 100,000 is close to 5%. Good enough.
2. Do early users stay in when you widen the rollout? All 4,976 users from the 5% stage are still enabled at 20%. That property comes from comparing the bucket against a growing threshold. A user who saw the new checkout yesterday sees it today.
3. Do different flags pick different users? Including the flag name in the hash means the 5% for new-pricing and the 5% for new-search overlap about as much as chance predicts. Without the flag name, the same 5% of users would get every experiment at once, and one unlucky cohort would absorb every bug.
4. What goes wrong with random assignment? Nearly 1 user in 10 would see the feature appear and disappear between page loads. That’s a confusing experience, and it makes the metrics useless because nobody belongs to a stable group.

Validate before you apply, and keep the last good version

A config loader should refuse bad input instead of trusting it. Here’s the shape, applied to a payment timeout:

function validate(cfg) {
  const errors = [];
  if (!Number.isInteger(cfg.paymentTimeoutMs) || cfg.paymentTimeoutMs < 100 || cfg.paymentTimeoutMs > 10_000)
    errors.push(`paymentTimeoutMs must be an integer 100..10000, got ${JSON.stringify(cfg.paymentTimeoutMs)}`);
  if (!(cfg.rolloutPercent >= 0 && cfg.rolloutPercent <= 100))
    errors.push(`rolloutPercent must be 0..100, got ${JSON.stringify(cfg.rolloutPercent)}`);
  return errors;
}
let active = { version: 41, paymentTimeoutMs: 800, rolloutPercent: 5 };
function apply(candidate) {
  const errors = validate(candidate);
  if (errors.length) { console.log(`rejected v${candidate.version}: ${errors.join('; ')}; keeping v${active.version}`); return; }
  active = candidate; console.log(`applied v${candidate.version}`);
}

rejected v42: paymentTimeoutMs must be an integer 100..10000, got "800ms"; keeping v41
applied v43

Version 42 had "800ms" as a string. Without validation, setTimeout(fn, "800ms") in Node coerces it to NaN and fires after about 1 ms, so every payment call would time out instantly. With validation, the instance logs a rejection and keeps running on version 41.

That “keep running on the last known good” behaviour matters as much as the check. The worst design is one where an invalid config makes the service crash on reload, and then crash again on restart because the bad config is still there. Validate in CI too (schema checks on the config repository), so most mistakes never reach an instance.

Dynamic reload has its own hazards

Spring Cloud Config with @RefreshScope, or any watch-based system, lets you change values without a restart. That’s useful, but not every setting is safe to change live:

@RefreshScope
@ConfigurationProperties(prefix = "payments")
public class PaymentProperties {
    private Duration timeout = Duration.ofMillis(800);
    // getters and setters
}

A timeout read on each call is fine to refresh. A connection pool size, a thread count or a Kafka consumer’s group id is not: the object was built at startup, and refreshing the property either does nothing (confusing) or rebuilds the bean under load (risky). Be explicit about which properties are dynamic and which require a rolling restart.

There’s also a compatibility problem that mirrors database migrations. If version 2 of the code reads a new config key, and you roll back the code to version 1 while the config still uses the new shape, does version 1 still start? Add new keys before the code that needs them, keep old keys until no running code reads them, and remove them last.

Watching the rollout and stopping it

A staged rollout is only safer if someone (or something) looks at each stage. Decide before starting:

• Which metrics gate the next stage. Error rate, p99 latency, checkout conversion, payment declines, compared between the enabled cohort and the rest.
• How long each stage bakes. Long enough to see the traffic patterns that matter, often including a daily peak.
• What triggers an automatic halt. “If the cohort’s error rate exceeds the control’s by more than 0.5 percentage points for 10 minutes, set the flag to 0%.”

Rollback has to be one action that doesn’t depend on a build pipeline. For flags, that’s setting the percentage to zero. For infrastructure config, it’s re-applying the previous version, which is trivial if configs are versioned in Git and painful if they were edited in place.

Finally, delete flags after the rollout finishes. A codebase with 300 stale flags has 300 code paths nobody tests, and eventually someone flips an old one by accident.

Interview practice: the timeout change that took out every region

“An engineer lowered the database query timeout from 5 seconds to 500 ms in the shared config service to reduce tail latency. It applied everywhere in about 30 seconds. A few minutes later, a nightly batch endpoint in every region started failing because its queries take 2 to 3 seconds. How would you have prevented this, and how do you recover?”

Answer

Recovery first: revert the config to the previous version, globally, which should be a single command if configs are versioned. Then confirm the batch endpoint recovers and check whether any batch runs need replaying.

Prevention has several layers. The change should have been staged by infrastructure, not applied globally: one canary instance or one zone first, with a bake period that includes the batch window, or at least an explicit check of which endpoints exceed the new limit (the slow query logs would have shown 2 to 3 second queries). A single global timeout is also too blunt; interactive endpoints and batch endpoints have different budgets, so the timeout should be per operation or per workload. And the config system should require the change to go through review with the same evidence a code change would have: which callers are affected, which metric would show a problem, and what the rollback is. Validation can’t catch this one (500 ms is a valid number), so staging and observation are the real defences here.

Further reading

• Microservices interview questions
• Microservices MCQs
• Kubernetes ConfigMap updates
• Deadlines across call chains
• Martin Fowler’s site: Feature Toggles (aka Feature Flags)