In a data centre of fixed servers, a service’s address could live in a config file for years. In a container platform it can’t. Instances start and stop with every deploy, autoscaling adds and removes them by the minute, and a rescheduled pod comes back on a different IP. The caller needs a way to turn “inventory” into “one of 10.0.1.11, 10.0.1.12 or 10.0.1.13, whichever are healthy right now”.
That’s service discovery. The mechanics are simple. The interview questions are about the gap between what the registry believes and what’s actually running, because that gap is where requests fail.
The moving parts
Every discovery setup has the same pieces, whatever the product:
- A registry holding the current instances of each service. Eureka, Consul and ZooKeeper are standalone registries; in Kubernetes, the API server’s EndpointSlices play this role.
- Registration. Instances add themselves on startup (self-registration, as with a Eureka client), or the platform registers them (Kubernetes adds a pod’s IP once its readiness probe passes).
- Health. Instances send heartbeats, or the registry probes them. An instance that misses heartbeats for longer than a TTL is removed.
- Lookup. Callers ask the registry for instances. Since asking on every request would overload the registry and add latency, callers cache the answer and refresh it periodically.
Two architectures follow from where the lookup happens.
Client-side discovery: the calling service queries the registry, caches the instance list, and picks an instance itself (round robin, least requests). Spring Cloud’s LoadBalancer with Eureka works this way: a lb://inventory URI is resolved inside the caller. It’s efficient (no extra hop) and allows smart balancing, but every language needs a client library that does it correctly.
Server-side discovery: the caller sends to a stable address, and something else picks the instance. A Kubernetes ClusterIP Service is the everyday example: the caller connects to inventory.shop.svc.cluster.local, which resolves to one virtual IP, and kube-proxy rules forward each new connection to a ready pod. AWS load balancers and service mesh sidecars do the same. Callers stay simple, at the cost of a component in the path.
Simulating a crashed instance
To see the gap between registry and reality, here’s a simulation of one minute with three instances. Instances heartbeat every 5 seconds, the registry expires an entry 15 seconds after its last heartbeat, and the client refreshes its cached list every 10 seconds. The client sends 10 requests per second, round robin. At t=20s, instance 10.0.1.12 dies.
const TTL = 15, HEARTBEAT = 5, CLIENT_REFRESH = 10;
function simulate(label, { crashAt = null, gracefulAt = null, retryOtherInstance = false }) {
const registry = new Map(); // address -> last heartbeat time
const alive = new Set(['10.0.1.11:8080', '10.0.1.12:8080', '10.0.1.13:8080']);
const victim = '10.0.1.12:8080';
let cache = [], cachedAt = -Infinity, rr = 0, failed = 0, lastFail = null;
for (let t = 0; t < 60; t++) {
if (t === crashAt) alive.delete(victim); // kill -9: no deregistration
if (t === gracefulAt) registry.delete(victim); // deregister first...
if (gracefulAt !== null && t === gracefulAt + CLIENT_REFRESH) alive.delete(victim); // ...stop after clients refresh
for (const a of alive) if (t % HEARTBEAT === 0 && !(gracefulAt !== null && t >= gracefulAt && a === victim)) registry.set(a, t);
for (const [a, hb] of registry) if (t - hb > TTL) registry.delete(a);
if (t - cachedAt >= CLIENT_REFRESH) { cache = [...registry.keys()]; cachedAt = t; }
for (let r = 0; r < 10; r++) { // 10 requests per second, round robin
let target = cache[rr++ % cache.length];
if (!alive.has(target) && retryOtherInstance) target = cache[rr++ % cache.length];
if (!alive.has(target)) { failed++; lastFail = t; }
}
}
// ...print the failure count
}Three variants: the instance crashes (kill -9, no chance to deregister); it crashes and the client retries a failed request on the next instance; and a graceful shutdown that deregisters first, waits one client refresh interval, then stops. Output from Node 22:
instance crashes at t=20s failed 66 of 600, last failure at t=39s
crash, client retries another instance failed 0 of 600
graceful: deregister, wait, then stop failed 0 of 600The crash case is worth tracing second by second, because it’s exactly what an interviewer will ask:
- The last heartbeat from
10.0.1.12was at t=15s. It died at t=20s. - The registry kept it until its entry was more than 15 seconds old, removing it at t=31s.
- The client’s cache was refreshed at t=30s, one second too early to notice. Its next refresh was at t=40s.
- From t=20s to t=39s, one request in three went to a dead address: 66 failures.
So the worst-case staleness is roughly the TTL plus the client refresh interval, plus up to one heartbeat interval. Shorter values reduce it but increase load on the registry and make false removals (an instance briefly paused by garbage collection) more likely. There’s no free setting; there’s a trade-off you should be able to describe.
Closing the gap
The second and third lines show the two practical fixes, and production systems use both.
Callers retry on a different instance. A connection refused means the request never reached the application, so retrying elsewhere is safe even for non-idempotent operations. Client-side load balancers (Spring Cloud LoadBalancer with retries, Envoy) can also eject an instance from their local list after consecutive failures, without waiting for the registry: this is called outlier detection. Timeouts after the request was sent are different, and only safe to retry for idempotent operations.
Instances leave gracefully. On shutdown, the instance deregisters (or fails its readiness probe), keeps serving for long enough that every client’s cache has refreshed, finishes in-flight requests, and only then exits. In Kubernetes, the pod is removed from the endpoints when termination starts, but kube-proxy rules and other nodes update asynchronously, so a common pattern is a short preStop sleep (a few seconds) before the application begins shutting down. Without it, a rolling deploy can produce a burst of connection errors on every release.
Crashes can’t be graceful, which is why the retry matters too.
DNS, caching and long-lived connections
DNS-based discovery has its own staleness, and it often surprises people:
- Client DNS caches. The JVM caches successful lookups for 30 seconds by default when no security manager is installed (controlled by
networkaddress.cache.ttl). Some older setups cached forever. Node doesn’t cachedns.lookupresults itself, but the OS resolver might. - Connection pools outlive lookups. HTTP keep-alive and gRPC channels hold connections open. After a scale-up, existing connections keep going to the old instances, and the new ones receive almost no traffic until connections are recycled. This is why gRPC behind a plain Kubernetes ClusterIP Service often balances badly: kube-proxy balances connections, not requests, and gRPC multiplexes every request over one connection. Headless Services with client-side balancing, or a mesh, fix it.
- Kubernetes Service IPs are stable. For ClusterIP Services, DNS returns the virtual IP, which doesn’t change when pods do. Staleness there lives in kube-proxy and the endpoints, not DNS. For headless Services, DNS returns pod IPs directly and caching matters again.
Readiness is part of discovery
An instance should only receive traffic when it can serve it. Registering at process start, before the database pool is warm or caches are loaded, sends traffic to an instance that will fail or be slow. In Kubernetes, the readiness probe gates entry into the endpoints; with self-registration, register only after startup work completes. The reverse applies too: an instance that becomes unable to serve (lost its database connection) should fail readiness and leave rotation, without being restarted, which is what the liveness probe is for.
Interview practice: errors on every deploy
“Each time the payments service deploys, its callers see a spike of connection refused and 503 errors lasting 10 to 20 seconds, then everything is fine. Payments runs on Kubernetes with a rolling update. What’s happening and how do you fix it?”
Answer
The old pods are stopping before every caller has stopped sending them traffic. When a pod is terminated, Kubernetes removes it from the Service’s endpoints and sends SIGTERM at roughly the same time, but endpoint changes propagate asynchronously to kube-proxy on every node, and to any client-side caches. If the application exits on SIGTERM immediately, there’s a window where traffic still arrives at a pod that’s gone.
I’d add a preStop hook that sleeps for a few seconds (enough for endpoint propagation), and make the application handle SIGTERM by stopping new work, finishing in-flight requests, then exiting, with terminationGracePeriodSeconds long enough for both. I’d check that new pods only become ready after they can actually serve, so the rollout doesn’t move traffic too early in the other direction. On the caller side, I’d make sure clients retry connection-refused errors on another instance, since those requests never reached payments, and only retry other failures when there’s an idempotency key. Then I’d watch the error rate during the next deploy to confirm the spike is gone.
Further reading
- Microservices interview questions
- Microservices MCQs
- Health checks
- Kubernetes termination and draining
- Pattern reference: Client-side discovery on microservices.io