Kubernetes failures rarely announce their cause. A user sees a timeout; a dashboard shows 5xx errors; a Pod says CrashLoopBackOff. The cause could be in the image, the config, the scheduler, the network, the node or the app itself, and it’s easy to spend an hour restarting things on a hunch. What interviewers look for in a troubleshooting question is an order: a sequence of cheap checks where each one narrows the search and tells you which check comes next.
Examples assume Kubernetes 1.30 or later. Command output is illustrative of what kubectl prints, not captured from a specific cluster.
Start from what the user sees, then pick a direction
Before running anything, pin down the symptom precisely. “The site is down” splits into very different investigations:
- Pods aren’t running (rollout stuck, crash, Pending): work down the Pod lifecycle.
- Pods are running but requests fail (refused, timeouts, 502s): work along the request path, client to Pod.
- Requests succeed but slowly or with wrong answers: that’s usually the application or a dependency, and metrics and traces beat kubectl.
Also ask what changed. A deploy ten minutes ago, a node pool upgrade, a new NetworkPolicy, an expired certificate. kubectl rollout history and the CI log answer the first in seconds, and “roll back first, investigate second” is often the right call when users are affected.
Down the Pod lifecycle
A Pod goes through scheduling, image pull, container creation, startup, and readiness. Each stage fails with its own signature, and the first command shows which stage you’re at:
$ kubectl get pods -n shop -l app=checkout -o wide
NAME READY STATUS RESTARTS AGE NODE
checkout-7b9c6d5f4-4hxkq 0/1 Pending 0 6m <none>
checkout-7b9c6d5f4-9vq2m 0/1 ImagePullBackOff 0 6m node-b
checkout-7b9c6d5f4-tq8wz 0/1 CrashLoopBackOff 5 (40s ago) 6m node-c
checkout-6d8f7c9b54-k2x7p 1/1 Running 0 2d node-aThat (contrived) listing shows the main branches at once:
- Pending, no node: the scheduler couldn’t place it. The reason is in events: insufficient CPU or memory, unmatched affinity, untolerated taints, an unbound PVC. See why a Pod stays Pending.
- ImagePullBackOff or ErrImagePull: wrong tag, missing registry credentials, or the node can’t reach the registry. Events quote the registry’s error.
- CreateContainerConfigError: a referenced Secret or ConfigMap (or a key in it) doesn’t exist, or
runAsNonRootcan’t be verified. - CrashLoopBackOff: the container starts and exits repeatedly. Logs and exit codes explain it. See the CrashLoopBackOff guide.
- Running but 0/1 Ready: readiness probe failing. The app is up but not accepting traffic.
The next command for nearly every branch is describe, because its Events section explains what the kubelet and scheduler tried:
$ kubectl describe pod checkout-7b9c6d5f4-9vq2m -n shop | sed -n '/Events:/,$p'
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m default-scheduler Successfully assigned shop/checkout-7b9c6d5f4-9vq2m to node-b
Normal Pulling 4m (x4 over 6m) kubelet Pulling image "registry.example.com/checkout:5.4.2"
Warning Failed 4m (x4 over 6m) kubelet Failed to pull image "registry.example.com/checkout:5.4.2": rpc error: code = NotFound desc = failed to resolve reference: not found
Warning Failed 4m (x4 over 6m) kubelet Error: ErrImagePull
Normal BackOff 60s (x20 over 6m) kubelet Back-off pulling image "registry.example.com/checkout:5.4.2"not found means the tag doesn’t exist (the CI pipeline probably failed to push it), as opposed to unauthorized or i/o timeout, which point at credentials or networking. One line of events often closes the investigation.
Events expire, by default an hour after the last occurrence. For an older incident, kubectl get events -n shop --sort-by=.lastTimestamp may show nothing, and you’ll need whatever your cluster ships events to.
Logs, including the ones from the container that died
For crashes, logs come next, and the first mistake is reading the wrong container’s. After a crash, the kubelet starts a fresh container, and kubectl logs shows the new one, which may have printed nothing yet. The crash is in the previous instance:
$ kubectl logs checkout-7b9c6d5f4-tq8wz -n shop --previous --tail=20
2026-10-07T09:41:02Z INFO starting checkout 5.4.2
2026-10-07T09:41:02Z FATAL config: required env var PAYMENTS_URL is not setPair that with the termination record, which survives even when the app logs nothing:
$ kubectl get pod checkout-7b9c6d5f4-tq8wz -n shop \
-o jsonpath='{.status.containerStatuses[0].lastState.terminated}'
{"exitCode":1,"reason":"Error","startedAt":"...","finishedAt":"..."}Exit codes narrow things quickly: 1 is usually an application error like the one above; 137 with reason OOMKilled is the memory limit (exit 137 guide); 137 without OOMKilled is often a liveness probe kill or a SIGKILL after an ignored SIGTERM; 127 means the command wasn’t found in the image. For multi-container Pods, add -c <name> and check init containers too, since a stuck init container holds the Pod in Init:0/1 with the real error only in its own logs.
Along the request path
If Pods are Running and Ready but requests fail, stop looking at the Pods and follow the traffic. The order goes from the client inward, and each step is a single command from a debug Pod in the client’s namespace:
- DNS.
nslookup checkout.shop.svc.cluster.local. A failure here points at CoreDNS or an egress NetworkPolicy blocking port 53. - Endpoints.
kubectl get endpointslices -n shop -l kubernetes.io/service-name=checkout. Empty means the Service’s selector matches no Ready Pods. A label typo is the classic cause (selectors guide). - Pod IP directly.
curl -m 3 http://<pod-ip>:8080/healthz. If that works but the Service doesn’t, the Service’stargetPortis likely wrong. - Service IP.
curl -m 3 http://checkout.shop/healthz. If the Pod IP works only from inside its own namespace, suspect a NetworkPolicy. - Ingress or load balancer. Check the ingress controller’s logs for the failing host and path, and its view of the backend.
The type of failure is evidence too. Connection refused means a packet arrived and nothing listened (no endpoints, wrong port, app bound to 127.0.0.1). A timeout means packets were dropped (NetworkPolicy, security groups, a dead node). A 502 or 503 from an ingress means the proxy reached nothing healthy upstream.
For debugging inside a distroless container that has no shell or curl, attach an ephemeral container instead of rebuilding the image:
kubectl debug -it checkout-6d8f7c9b54-k2x7p -n shop --image=busybox:1.36 --target=api -- sh--target shares the process namespace with the api container, so you can see its processes, and you already share its network.
The node, last but not never
If the problems cluster on one node (all failing Pods show the same NODE), look there: kubectl describe node node-c shows conditions like MemoryPressure, DiskPressure or Ready: False, and the allocated resources. Pods Evicted with “The node was low on resource: ephemeral-storage” point at full disks, often container logs or emptyDir usage. kubectl top node shows real usage if metrics-server is installed. Going straight to the node first is usually a mistake, though; most incidents are in the Pod spec or the app.
Interview practice
“Users report intermittent 502s from api.example.com. About one request in four fails. The api Deployment has four replicas, all Running. Walk through how you’d find the cause.”
Answer
One failure in four with four replicas suggests a single bad backend that’s still receiving traffic. First I’d check readiness: kubectl get pods -l app=api -o wide. If one Pod shows 0/1, it’s out of the Service and the 502s probably come from elsewhere; if all show 1/1, one Pod might be passing its probe while failing real requests, which happens when the readiness endpoint doesn’t exercise anything real.
Next, the endpoints: kubectl get endpointslices -l kubernetes.io/service-name=api to confirm four ready addresses. Then I’d hit each Pod IP directly from a debug Pod with a real request path, not just /healthz, to find the bad one, and read its logs with kubectl logs <pod> --since=15m. If all four answer correctly directly, I’d turn to the ingress controller’s logs, which name the upstream address for each 502. If they consistently point at an IP that’s no longer in the endpoint list, the ingress has stale upstreams, and if the 502s line up with deploys, it’s the shutdown race described in the termination guide. Once I know which Pod or which moment, I’d remove the bad Pod from service (delete it, or label it out of the selector to keep it for inspection) while the root cause gets fixed.