Q: What is a Service, and how do the ClusterIP, NodePort, LoadBalancer and ExternalName types differ?

Pods come and go with changing IPs, so a Service gives a set of pods, chosen by a label selector, a stable virtual IP and DNS name, and spreads traffic across the ready ones. - ClusterIP (the default): a virtual IP reachable only from inside the cluster. Used for service-to-service traffic. - NodePort: a ClusterIP plus a port opened on every node, by default from the 30000–32767 range. Traffic to any nodeIP:nodePort reaches the Service. - LoadBalancer: asks the cloud provider, or a load-balancer controller such as MetalLB on bare metal, to provision an external load balancer for the Service. By default it's built on top of a NodePort. It's the usual way to expose one service publicly. - ExternalName: no selector and no proxying. Cluster DNS just returns a CNAME to an external hostname such as db.example.com. A headless Service (clusterIP: None) is a variation with no virtual IP: DNS returns the pod IPs directly.

Question 1

What is Kubernetes, and what problems does it solve compared with running containers by hand?

Accepted Answer

Kubernetes is an open-source container orchestration platform, originally designed at Google and now hosted by the CNCF. You describe the desired state of your application declaratively, usually in YAML: which images, how many replicas, how they're exposed, how much CPU and memory they need. Kubernetes then works continuously to make the actual state match. That solves the problems you hit once you have more than a handful of containers on more than one machine: - Scheduling: placing containers on nodes that have capacity - Self-healing: restarting crashed containers and replacing pods from failed nodes - Scaling: changing replica counts, manually or automatically - Service discovery and load balancing: stable names and virtual IPs in front of changing pods - Rolling updates and rollbacks without downtime - Configuration and secrets kept out of the image The core idea is the reconciliation loop: controllers watch the desired state stored in the API server and keep correcting any drift.

Question 2

Explain the Kubernetes architecture. What runs on the control plane, and what runs on each worker node?

Accepted Answer

A cluster has a control plane that makes decisions and worker nodes that run the workloads. Control plane: - kube-apiserver: the front door. kubectl, controllers and kubelets all talk to it; it authenticates, authorizes, runs admission and persists objects. - etcd: a consistent key-value store holding all cluster state. Only the API server talks to it. - kube-scheduler: picks a node for each new pod, based on resources, affinity, taints and so on. - kube-controller-manager: runs the built-in controllers (Deployment, ReplicaSet, Node, Job…), each reconciling desired and actual state. - cloud-controller-manager (optional): integrates with a cloud provider for load balancers, routes and nodes. Each node runs: - kubelet: the agent that makes sure the pods assigned to its node are running, and reports their status. - Container runtime: containerd or CRI-O, driven by the kubelet through the CRI. - kube-proxy (optional): programs Service routing rules; some network plugins replace it.

Question 3

What is a Pod, and why does Kubernetes schedule Pods instead of individual containers?

Accepted Answer

A Pod is the smallest deployable unit in Kubernetes: one or more containers that are always scheduled together on the same node and share a context. Containers in a pod share a network namespace (one IP address and port space, so they reach each other on localhost) and can share volumes. Kubernetes schedules pods rather than containers because some processes are tightly coupled and must live and die together, like an app and its log shipper or proxy. The pod gives them a common lifecycle, IP and storage without merging them into one image. Pods are ephemeral. If a pod dies or its node fails, it isn't resurrected: a controller creates a new pod with a new name and a new IP. That's why you rarely create bare pods. You use a Deployment, StatefulSet, DaemonSet or Job to manage them, and put a Service in front for a stable address.

Question 4

What's the relationship between a Deployment, a ReplicaSet and a Pod? Why not just create a ReplicaSet directly?

Accepted Answer

They form a chain of controllers: - A ReplicaSet keeps a set number of identical pods running. It counts the pods matching its label selector and creates or deletes pods from its pod template to reach replicas. - A Deployment manages ReplicaSets. Each time you change the pod template (new image, env var, resources), it creates a new ReplicaSet and gradually scales it up while scaling the old one down. That's what gives you rolling updates, rollout history and kubectl rollout undo. You rarely create a ReplicaSet yourself because it has no update strategy: editing its template doesn't touch pods that already exist. Old ReplicaSets are kept at zero replicas as revision history, 10 by default (revisionHistoryLimit), and that's what a rollback scales back up. The selector must match the template's labels, and in apps/v1 a Deployment's selector is immutable after creation.

Question 5

What is a Service, and how do the ClusterIP, NodePort, LoadBalancer and ExternalName types differ?

Accepted Answer

Pods come and go with changing IPs, so a Service gives a set of pods, chosen by a label selector, a stable virtual IP and DNS name, and spreads traffic across the ready ones.

ClusterIP (the default): a virtual IP reachable only from inside the cluster. Used for service-to-service traffic.
NodePort: a ClusterIP plus a port opened on every node, by default from the 30000–32767 range. Traffic to any nodeIP:nodePort reaches the Service.
LoadBalancer: asks the cloud provider, or a load-balancer controller such as MetalLB on bare metal, to provision an external load balancer for the Service. By default it's built on top of a NodePort. It's the usual way to expose one service publicly.
ExternalName: no selector and no proxying. Cluster DNS just returns a CNAME to an external hostname such as db.example.com.

A headless Service (clusterIP: None) is a variation with no virtual IP: DNS returns the pod IPs directly.

Question 6

How does a Deployment rolling update work? What do maxSurge and maxUnavailable control, and how do you roll back?

Accepted Answer

When the pod template changes, the Deployment creates a new ReplicaSet and shifts pods over step by step: scale the new one up, wait for new pods to become available, scale the old one down, repeat.

maxSurge: how many pods may exist above the desired count during the rollout.
maxUnavailable: how many pods may be unavailable below the desired count.

Both take a number or a percentage, both default to 25%, and they can't both be 0. maxSurge: 1, maxUnavailable: 0 keeps full capacity but needs room for an extra pod; a higher maxUnavailable is faster and needs no spare capacity.

Readiness probes are what make this safe: a new pod that never becomes ready stalls the rollout instead of taking traffic. After progressDeadlineSeconds (600 by default) the Deployment reports ProgressDeadlineExceeded, but it does not roll back on its own.

To roll back, check kubectl rollout history deployment/web, then run kubectl rollout undo deployment/web, optionally with --to-revision=N.

Question 7

What's the difference between liveness, readiness and startup probes, and what happens when each one fails?

Accepted Answer

They answer different questions, and failures have different consequences:

Readiness: can this container serve traffic right now? On failure the pod is marked not Ready and removed from Service endpoints, but it is not restarted. It runs for the container's whole life, so it also covers temporary overload.
Liveness: is the container stuck beyond recovery, like a deadlock? After failureThreshold consecutive failures (3 by default) the kubelet restarts the container.
Startup: has the app finished starting? Until it succeeds, liveness and readiness probes don't run, so slow starters aren't killed early. If it keeps failing for failureThreshold × periodSeconds, the container is restarted.

Each probe can be httpGet (any status from 200 to 399 passes), tcpSocket, exec or grpc.

A classic mistake is a liveness check that tests the database. If the database blips, every pod restarts at once. Liveness should only check the process itself.

Question 8

How do ConfigMaps and Secrets differ, and what are the ways a Pod can consume them?

Accepted Answer

Both are namespaced key-value objects that keep configuration out of the image.

ConfigMap: non-sensitive settings such as feature flags, URLs or whole config files.
Secret: sensitive data such as passwords, tokens and TLS keys. Values are base64-encoded, which is encoding, not encryption. Secrets can be locked down separately with RBAC, and the kubelet keeps mounted secrets in tmpfs (memory) rather than on disk.

A pod can consume either one as:

Environment variables, one key with valueFrom or every key with envFrom
Files in a volume, one file per key
Command arguments built from those env vars

The practical difference: env vars are read once at container start, so changes need a restart. Mounted volumes are eventually updated by the kubelet (except with subPath), so an app that re-reads its files can pick up changes. Both objects are limited to 1 MiB and can be marked immutable: true.

Question 9

Are Kubernetes Secrets actually secure? Why aren't they encrypted by default, and how would you harden them?

Accepted Answer

Not by default. Secret values are only base64-encoded, and the API server stores them unencrypted in etcd. Anyone with access to etcd, an etcd backup, or get/list/watch on Secrets can read them. And anyone who can create a pod in a namespace can mount any Secret in it, so "create pods" effectively implies reading Secrets. Encryption at rest isn't on by default because it needs keys that someone has to generate, store and rotate, which is an operator decision. Many managed services configure it for you. To harden: - Enable encryption at rest with an EncryptionConfiguration passed to the API server via --encryption-provider-config, ideally the KMS v2 provider so the key lives in an external KMS. Then rewrite existing Secrets so they're stored encrypted. - Use least-privilege RBAC; remember list and watch reveal values too. - Protect etcd: TLS, restricted access, encrypted backups. - Consider an external secret store (Vault, a cloud secret manager) via the Secrets Store CSI Driver or External Secrets Operator.

Question 10

When would you use a StatefulSet instead of a Deployment, and what guarantees does it give you?

Accepted Answer

Use a StatefulSet when each replica has its own identity and data: databases, Kafka, ZooKeeper, Elasticsearch. It adds three guarantees a Deployment doesn't give:

Stable names: pods are db-0, db-1, db-2, and a replacement keeps the same name.
Stable network identity: through the headless Service named in serviceName, each pod gets its own DNS record, like db-0.db-headless.default.svc.cluster.local, so peers can find each other.
Stable storage: volumeClaimTemplates create one PVC per pod, and a rescheduled pod reattaches to the same volume.

With the default OrderedReady policy, pods are created in order from 0 up, each waiting for the previous one to be Running and Ready, and removed in reverse. Rolling updates go from the highest ordinal down, and partition allows staged updates.

Gotchas: by default, deleting or scaling down a StatefulSet does not delete its PVCs, and you have to create the headless Service yourself.

Question 11

What are resource requests and limits, and how does Kubernetes use each of them?

Accepted Answer

They're set per container, for CPU and memory:

Requests are what the container is guaranteed. The scheduler uses them: a pod only lands on a node whose unreserved allocatable capacity covers the sum of its requests. Requests also decide how CPU is shared under contention.
Limits are the ceiling, enforced by the kernel through cgroups on the node. A container that hits its CPU limit is throttled; one that exceeds its memory limit is OOM-killed.

CPU is measured in cores (250m is a quarter core) and memory in bytes (Mi, Gi). If you set only a limit, the request defaults to the limit. Together, requests and limits decide the pod's QoS class.

In practice, always set requests, because without them scheduling and autoscaling are guesswork. Set a memory limit to contain leaks. Many teams skip CPU limits for latency-sensitive services to avoid throttling. LimitRange can inject defaults and ResourceQuota can cap totals per namespace.

Question 12

A pod is in CrashLoopBackOff. What does that mean, and how do you troubleshoot it?

Accepted Answer

CrashLoopBackOff isn't a phase or a root cause. It means a container keeps exiting, and the kubelet is waiting before restarting it again. The delay grows exponentially (10s, 20s, 40s…) up to five minutes, and resets after the container runs cleanly for 10 minutes. My steps: - kubectl describe pod for the Last State: the reason and exit code, plus events. Exit code 1 is usually an app error; 137 means SIGKILL, often OOMKilled (the reason says so); 127 or 126 point at a bad command or entrypoint. - kubectl logs --previous to read the logs of the instance that crashed, since the current one may have just started. - Check config: missing env vars, a Secret or ConfigMap key that doesn't exist, wrong args, a database it can't reach. - Check probes: a too-aggressive liveness probe kills a healthy but slow app; the events show "Liveness probe failed". - If the image has a shell, override the command with sleep, or use kubectl debug to inspect it.

Question 13

What is an Ingress, and why does it need an ingress controller?

Accepted Answer

An Ingress is a set of layer-7 HTTP(S) routing rules: send shop.example.com/api to the api Service and everything else to web, and terminate TLS with a given certificate Secret. pathType can be Exact, Prefix or ImplementationSpecific. The Ingress object alone does nothing. An ingress controller (Traefik, HAProxy, an NGINX-based controller, or a cloud one like the AWS Load Balancer Controller) watches Ingress objects and configures a real proxy or load balancer to match. ingressClassName picks which controller handles it. Usually the controller itself sits behind one LoadBalancer Service, so dozens of services share a single external IP instead of paying for one load balancer each. Worth knowing: the Ingress API is frozen. It's GA and isn't going away, but it won't get new features, and the Kubernetes project now recommends Gateway API. The popular community ingress-nginx controller was retired in March 2026.

Question 14

What are namespaces used for, and are they a security boundary?

Accepted Answer

Namespaces divide one cluster into named scopes. Names must be unique within a namespace, not across the cluster, so dev and prod can each have a web Deployment. They're also the unit that most policies attach to:

RBAC Roles and RoleBindings grant access per namespace
ResourceQuota and LimitRange cap and default resource usage
NetworkPolicies and Pod Security labels are applied per namespace

A new cluster starts with default, kube-system, kube-public and kube-node-lease. Some resources aren't namespaced at all: Nodes, PersistentVolumes, StorageClasses, ClusterRoles and Namespaces themselves (kubectl api-resources --namespaced=false lists them). Services in other namespaces are reached by DNS as name.namespace.

On their own they're a soft boundary. Pods in different namespaces share nodes and kernels, and can talk to each other over the network unless NetworkPolicies say otherwise. For real multi-tenancy you add RBAC, NetworkPolicies, quotas and Pod Security Standards, or use separate clusters.

Question 15

What is the difference between labels and annotations, and how do selectors use labels?

Accepted Answer

Both are key-value metadata on objects, but they serve different purposes.

Labels identify and group objects, like app=web, tier=frontend, env=prod. They're short (values up to 63 characters) and they're what the system selects on. A Service finds its pods by label, a ReplicaSet counts pods by label, a NetworkPolicy targets pods by label, and kubectl get pods -l app=web filters by label.

Selectors come in two styles: equality-based (app=web, env!=prod) and set-based (env in (prod,staging), !canary). In manifests that's matchLabels and matchExpressions with In, NotIn, Exists and DoesNotExist.

Annotations hold non-identifying data that you can't select on and that can be much larger: build info, a git SHA, contact details, or tool configuration such as ingress controller settings or Prometheus scrape hints. kubectl apply stores its last-applied configuration in one.

A classic bug is a Service selector that doesn't match the pod labels: the Service exists but has no endpoints.

Question 16

What is a DaemonSet, and what would you use one for?

Accepted Answer

A DaemonSet runs one copy of a pod on every node, or on every node matching a selector. When a node joins the cluster it gets the pod automatically, and when a node is removed its pod is garbage-collected. You don't set a replica count. Typical uses are node-level agents: - Log collectors such as Fluent Bit or Fluentd - Monitoring agents such as Prometheus node-exporter - Networking: CNI agents and kube-proxy itself - Storage and security daemons, like CSI node plugins You limit it to some nodes with nodeSelector or node affinity, and add tolerations if it must run on tainted nodes such as the control plane. The controller also adds some tolerations automatically, which is why DaemonSet pods can run on cordoned nodes, and why kubectl drain needs --ignore-daemonsets. Updates use RollingUpdate by default, one node at a time unless you raise maxUnavailable, or OnDelete, where pods only update when you delete them.

Question 17

How does the Horizontal Pod Autoscaler decide how many replicas to run?

Accepted Answer

The HPA is a control loop in the controller manager, running every 15 seconds by default. It reads metrics, computes a replica count and updates the target's scale subresource:

desired = ceil(current × currentMetric / targetMetric)

So 4 pods averaging 90% CPU against a 70% target gives ceil(4 × 90/70) = 6. It skips changes within a 10% tolerance, clamps the result to minReplicas/maxReplicas, and uses a 5-minute scale-down stabilization window by default so it doesn't flap.

Key details:

CPU and memory come from the Metrics API, usually metrics-server. Custom and external metrics need an adapter, such as Prometheus Adapter or KEDA.
Utilization is relative to requests, so containers without requests can't be scaled on utilization.
The behavior field tunes scale-up and scale-down rates.
Don't hard-code replicas in a Deployment the HPA manages, or every kubectl apply fights it.

Question 18

Explain PersistentVolumes, PersistentVolumeClaims and StorageClasses, and how dynamic provisioning works.

Accepted Answer

They separate what storage an app needs from how it's provided: - A PersistentVolume (PV) is a cluster-scoped piece of real storage, like a cloud disk or an NFS export, whose lifecycle is independent of any pod. - A PersistentVolumeClaim (PVC) is a namespaced request: "20Gi, ReadWriteOnce, class fast-ssd". It binds one-to-one to a matching PV, and pods mount the claim, not the PV. - A StorageClass describes a kind of storage: its CSI provisioner, parameters, reclaimPolicy and volumeBindingMode. With dynamic provisioning, a PVC that names a class (or falls back to the default class) makes the provisioner create a PV on demand, so no admin pre-creates volumes. WaitForFirstConsumer delays that until a pod is scheduled, so the disk is created in the pod's zone; the default is Immediate. Access modes are ReadWriteOnce (one node), ReadOnlyMany, ReadWriteMany and ReadWriteOncePod (one pod). Dynamically provisioned PVs default to the Delete reclaim policy; use Retain for data you can't lose.

Question 19

How does traffic sent to a Service ClusterIP actually reach a pod? What does kube-proxy do?

Accepted Answer

A ClusterIP is virtual: in the default mode no network interface owns it. Three pieces make it work: - The EndpointSlice controller keeps a list of the IPs and ports of the ready pods matching the Service's selector. - kube-proxy, running on every node, watches Services and EndpointSlices and programs packet-forwarding rules in the node's kernel. - When a pod connects to ClusterIP:port, those rules on the client's node DNAT the packet to one backend pod IP, picked at random (or by sessionAffinity: ClientIP). Connection tracking keeps the rest of that connection on the same pod, and the pod network carries it to whichever node the pod is on. So kube-proxy isn't in the data path; the kernel does the forwarding. On Linux the default mode is iptables, nftables is the newer replacement and is expected to become the default, and IPVS mode is deprecated. Some CNIs, such as Cilium, replace kube-proxy with eBPF. A consequence: balancing is per connection, not per request, so long-lived HTTP/2 or gRPC connections can pile onto a few pods.

Question 20

Walk me through how an HTTPS request from a user on the internet reaches a container in your cluster.

Accepted Answer

A typical path, with an ingress controller or gateway in front:

DNS resolves shop.example.com to the external load balancer the cloud created for the controller's LoadBalancer Service.
The cloud load balancer forwards to a node's NodePort, or straight to pod IPs if it supports IP-target mode.
On the node, kube-proxy rules route it to an ingress controller pod, possibly on another node. With externalTrafficPolicy: Local only nodes running a controller pod receive traffic, which also preserves the client IP.
The ingress controller terminates TLS (unless the load balancer already did), matches host and path, and picks a backend. Many controllers send traffic directly to pod IPs from the Service's EndpointSlices rather than through the ClusterIP.
The pod network (the CNI plugin) delivers the packet to the target pod's IP. Pod-to-pod traffic needs no NAT.
The container receives it on its containerPort. Only ready pods are listed as endpoints, which is why readiness probes matter for zero-downtime deploys.

The response returns along the same connection path.

Question 21

What are the three Pod QoS classes, how is a pod’s class decided, and why does it matter?

Accepted Answer

You don't set the QoS class; Kubernetes derives it from requests and limits and records it in status.qosClass:

Guaranteed: every container has CPU and memory requests and limits, and each request equals its limit.
Burstable: not Guaranteed, but at least one container has a CPU or memory request or limit.
BestEffort: no container has any requests or limits.

It matters when a node runs short of memory. The kubelet sets each container's oom_score_adj from its class (-997 for Guaranteed, 1000 for BestEffort, in between for Burstable), so the kernel OOM killer picks BestEffort processes first.

For node-pressure eviction, the kubelet ranks pods by whether their usage exceeds their requests, then by priority, then by how far over requests they are. It doesn't use the QoS class directly, but the result is roughly BestEffort first and Guaranteed last.

So critical workloads should have honest memory requests, ideally Guaranteed. Guaranteed pods with whole-number CPU requests can also get exclusive cores under the static CPU manager policy.

Question 22

One container keeps getting OOMKilled, and another is slow even though its average CPU usage looks low. What is going on in each case?

Accepted Answer

They're the two ways limits bite, because memory is incompressible and CPU is compressible.

OOMKilled: when a container goes over its memory limit, the kernel OOM killer kills a process in its cgroup. The status shows Reason: OOMKilled and exit code 137, and the container restarts, often into CrashLoopBackOff. A pod can also be killed or evicted under its limit if the whole node runs out of memory. Fix it by sizing the limit from real usage (kubectl top, working-set metrics, VPA recommendations), fixing leaks, and making runtimes container-aware, such as the JVM's -XX:MaxRAMPercentage.

CPU throttling: a CPU limit is enforced as a CFS quota: CPU time per scheduling period, usually 100 ms. A multi-threaded app can burn its whole quota early in a period and then stall until the next one. Average usage looks low while tail latency spikes. The throttling counters in cAdvisor metrics show it. Fix it by raising or removing the CPU limit while keeping an accurate request, or by using fewer threads.

Question 23

A pod has been stuck in Pending for ten minutes. How do you find out why?

Accepted Answer

Pending means the pod was accepted but isn't running yet. Usually it hasn't been scheduled; sometimes it's scheduled and still pulling images. kubectl describe pod tells you which: a FailedScheduling event from the scheduler explains why each group of nodes was rejected. Common causes: - Insufficient CPU or memory: no node has enough unreserved allocatable capacity for the pod's requests. Actual usage doesn't matter. Lower the requests or add nodes; with a cluster autoscaler, look for scale-up events. - Taints the pod doesn't tolerate, or a nodeSelector or affinity that matches no node. - Anti-affinity or topology spread rules that can't be satisfied, for example more replicas than nodes with required per-node anti-affinity. - An unbound PVC: no default StorageClass, no matching PV, or a volume in a different zone. If the pod already has a node assigned, look at container states instead: it's probably pulling a large image or waiting on a volume mount. And if a ResourceQuota is the problem, the pod is never created at all, so check the ReplicaSet's events.

Question 24

What does ImagePullBackOff mean, and what are the usual causes?

Accepted Answer

The kubelet couldn't pull the container image. You first see ErrImagePull, then ImagePullBackOff while it keeps retrying with an increasing delay, capped at five minutes. The container never starts, so there are no logs; the answer is in kubectl describe pod events, which show the registry's actual error.

Usual causes:

A typo in the image name, or a tag that doesn't exist ("manifest unknown", "not found").
A private registry without credentials ("unauthorized"). Create a docker-registry Secret and reference it in imagePullSecrets on the pod or its ServiceAccount, or grant the nodes registry access through cloud IAM.
Registry rate limits, like Docker Hub's anonymous pull limits.
Network problems from the node: DNS, egress firewall, proxy ("i/o timeout").
Wrong architecture: the image has no variant for the node's platform, such as arm64.

To avoid surprises, pin images by tag or digest instead of :latest, and remember that :latest with no explicit imagePullPolicy means Always.

Question 25

How does RBAC work in Kubernetes? Explain Role, ClusterRole, RoleBinding and ClusterRoleBinding.

Accepted Answer

RBAC decides whether a subject may perform a verb on a resource. It's purely additive: there are no deny rules, so anything not granted by some binding is refused.

Role: a list of rules (API groups, resources, verbs) inside one namespace.
ClusterRole: the same, but not namespaced. Use it for cluster-scoped resources like nodes, for non-resource URLs, or as a reusable template.
RoleBinding: grants a Role or a ClusterRole to subjects (users, groups, ServiceAccounts) within one namespace. Binding a ClusterRole this way limits it to that namespace; that's how the built-in view, edit and admin roles are usually handed out.
ClusterRoleBinding: grants a ClusterRole across the whole cluster.

Kubernetes has no User objects; users and groups come from authentication, such as certificates or OIDC. A binding's roleRef can't be changed; you recreate the binding. Test with kubectl auth can-i list pods -n dev --as=system:serviceaccount:dev:ci-bot, and avoid wildcards, cluster-admin, and broad access to Secrets or pods/exec.

Question 26

What is a ServiceAccount, and how does a pod authenticate to the Kubernetes API?

Accepted Answer

A ServiceAccount is an identity for workloads, as opposed to human users. It's namespaced, every namespace gets one called default, and a pod uses default unless you set serviceAccountName.

By default the kubelet mounts a short-lived, automatically rotated token into the pod at /var/run/secrets/kubernetes.io/serviceaccount/, together with the cluster CA and the namespace name. The token comes from the TokenRequest API and is bound to the pod, so it stops working once the pod is deleted. Since v1.24, Kubernetes no longer auto-creates long-lived token Secrets for each ServiceAccount.

Client libraries use this "in-cluster config": they find the API server through the KUBERNETES_SERVICE_HOST variable and send the token as a bearer token. The API server authenticates it as system:serviceaccount:<namespace>:<name> and RBAC authorizes it.

Good practice: one ServiceAccount per app with minimal RBAC, no permissions on default, and automountServiceAccountToken: false for pods that never call the API. Cloud workload identity features map ServiceAccounts to cloud IAM roles.

Question 27

What are Jobs and CronJobs, and which settings control retries, parallelism and overlapping runs?

Accepted Answer

A Job runs pods to completion, unlike a Deployment, which keeps pods running forever. It's done when enough pods succeed: - completions and parallelism control how many successes you need and how many pods run at once. completionMode: Indexed gives each pod an index for sharded work. - backoffLimit sets how many failures are retried before the Job is marked failed (default 6). activeDeadlineSeconds caps total runtime. - ttlSecondsAfterFinished cleans up finished Jobs automatically. - The pod's restartPolicy must be OnFailure or Never. A CronJob creates Jobs on a cron schedule. concurrencyPolicy decides what happens if the previous run is still going: Allow (the default), Forbid (skip the new run) or Replace (cancel the old one). You can also set startingDeadlineSeconds for late starts, history limits (3 successful and 1 failed Job kept by default), suspend and timeZone. A CronJob creates a Job roughly once per scheduled time, but occasionally two or none, so jobs should be idempotent.

Question 28

How do taints and tolerations work, and how would you dedicate a pool of GPU nodes to specific workloads?

Accepted Answer

A taint on a node repels pods; a toleration on a pod lets it ignore a matching taint. That's the opposite of affinity: affinity attracts, taints repel. Taint effects: - NoSchedule: pods without a matching toleration won't be scheduled there; running pods stay. - PreferNoSchedule: a soft version; the scheduler tries to avoid the node. - NoExecute: also evicts running pods that don't tolerate it. tolerationSeconds lets a pod stay for a limited time. Tolerations match with operator: Equal (key, value and effect) or Exists (any value). The key point for dedicated nodes: a toleration only allows a pod onto a node, it doesn't send it there. So taint the GPU nodes to keep everything else off, and give the GPU workloads both the toleration and a nodeSelector or node affinity for those nodes. Kubernetes uses taints itself: the node controller adds not-ready and unreachable NoExecute taints, and pods get default 300-second tolerations for them, which is why pods on a dead node are evicted after about five minutes.

Question 29

Compare nodeSelector, node affinity and pod anti-affinity. How would you keep replicas of a service off the same node?

Accepted Answer

All three constrain where the scheduler places a pod:

nodeSelector: the simplest. The node must have all the listed labels.
Node affinity: the same idea with operators (In, NotIn, Exists, DoesNotExist, Gt, Lt) and two strengths: required… (hard) and preferred… (weighted, soft). "IgnoredDuringExecution" means pods already running aren't evicted if node labels change later.
Pod affinity and anti-affinity: rules relative to other pods' labels within a topology domain set by topologyKey, such as kubernetes.io/hostname for a node or topology.kubernetes.io/zone for a zone. Affinity co-locates, say a cache next to its app; anti-affinity spreads.

To keep replicas apart, add pod anti-affinity on the app's own label with topologyKey: kubernetes.io/hostname. Prefer the preferred form: with the required form, replicas beyond the number of nodes stay Pending. Inter-pod affinity is also expensive in large clusters, and topologySpreadConstraints is often the better tool for even spreading.

Question 30

What is Helm, and what are charts, values and releases?

Accepted Answer

Helm is the package manager for Kubernetes. It saves you from hand-editing near-identical YAML for every environment.

A chart is a package: templated manifests written in Go templates, a Chart.yaml with metadata, and a values.yaml with defaults. Charts can depend on other charts and are shared through repositories or OCI registries.
Values are the configuration you supply at install time, with -f values-prod.yaml or --set image.tag=1.4, so one chart serves dev, staging and prod.
A release is an installed instance of a chart, with a name and a numbered revision history. Every upgrade creates a new revision, and helm rollback returns to an earlier one.

Everyday commands are helm upgrade --install (idempotent, good for CI), helm rollback, helm list, and helm template to render manifests locally for review.

Since Helm 3 there's no server-side Tiller: the CLI talks to the API server with your own credentials and stores release state in Secrets in the release's namespace. The main alternative is Kustomize, which uses template-free overlays and is built into kubectl as kubectl apply -k.

Question 31

What are init containers, and when would you use them?

Accepted Answer

Init containers, listed under spec.initContainers, run before the app containers start. They run one at a time, in order, and each must exit successfully before the next one starts. If one fails, the kubelet retries it according to the pod's restartPolicy; with Never, the pod fails. While they run, kubectl get pods shows a status like Init:0/2. Typical uses: - Wait for a dependency, for example loop until the database's DNS name resolves. - Prepare files: fetch config, clone a repo or render a template into a shared emptyDir volume. - Fix permissions on a mounted volume. - One-off setup per pod. Schema migrations are usually better as a separate Job, because an init container runs once for every replica. The advantage is separation: setup tools like git or curl stay out of the app image, and the init container can have different permissions. Regular init containers don't support liveness, readiness or startup probes. An init container with restartPolicy: Always is a different thing: a native sidecar.

Question 32

When would you run more than one container in a Pod, and how do those containers communicate?

Accepted Answer

Only when containers are tightly coupled: they must run on the same node, start and stop together, and scale together. The common patterns: - Sidecar: extends the main app, like a log shipper, a service-mesh proxy, or a config or certificate reloader. - Ambassador: a local proxy for outbound connections, such as a database or cloud SQL proxy on localhost. - Adapter: translates the app's output, for example an exporter that turns app stats into Prometheus metrics. Containers in a pod communicate cheaply because they share context: - The same network namespace: they talk over localhost and can't both bind the same port. - Shared volumes, usually an emptyDir, for exchanging files. - Optionally a shared process namespace (shareProcessNamespace: true), so they can see and signal each other's processes. The classic anti-pattern is putting an app and its database in one pod. They scale differently, a restart takes both down, and the database can't survive the pod. Those belong in separate workloads connected by a Service.

Question 33

What problems do native sidecar containers solve, and how do you declare one?

Accepted Answer

Sidecars used to be just extra entries in containers, which caused three problems:

No startup order: the app could start before its proxy was ready, so early requests failed.
No shutdown order: the sidecar could exit before the app finished draining.
Jobs never finished: the Job waited for every container, and a log shipper or proxy never exits.

Native sidecars, stable since v1.33, are init containers with restartPolicy: Always:

They start in order with the other init containers, and the kubelet moves on as soon as the sidecar has started, gated by its startup probe if it has one. So they're running before the app containers.
They keep running for the pod's lifetime and are restarted if they exit, whatever the pod's restart policy.
On termination they're stopped after the main containers, in reverse order.
They don't block Job completion.
They support probes, and a sidecar's readiness counts toward the pod's readiness.

Service-mesh proxies and log or metrics agents are the main users.

Question 34

How do pods discover other services in the cluster? What DNS names does Kubernetes create?

Accepted Answer

Through cluster DNS, usually CoreDNS running in kube-system behind a Service called kube-dns. It watches Services and serves records for them: - my-svc.my-ns.svc.cluster.local resolves to the Service's ClusterIP. cluster.local is the default cluster domain, but it's configurable. - Named ports get SRV records such as _http._tcp.my-svc.my-ns.svc.cluster.local. - A headless Service resolves to the IPs of its ready pods, and StatefulSet pods get names like db-0.db.my-ns.svc.cluster.local. The kubelet writes each pod's /etc/resolv.conf to point at the DNS Service and adds search domains, starting with my-ns.svc.cluster.local. That's why a pod can just call http://my-svc in its own namespace, and my-svc.other-ns for another namespace. One gotcha: the default ndots:5 makes short external names like api.example.com try every search domain first, which multiplies DNS queries. A trailing dot or a tuned dnsConfig avoids that. There's also an older mechanism: environment variables such as MY_SVC_SERVICE_HOST, but only for Services that existed when the pod started.

Question 35

What is a headless Service, and when do you need one?

Accepted Answer

A headless Service sets clusterIP: None. It gets no virtual IP, kube-proxy ignores it, and nothing load-balances it. Instead, a DNS lookup of the Service name returns the IPs of all its ready pods directly. You need one when clients must see or address individual pods: - StatefulSets: the governing Service in serviceName gives each pod a stable name like db-0.db.shop.svc.cluster.local, so clients can reach the primary specifically, and cluster members like Kafka brokers or etcd peers can find each other. - Client-side load balancing: a gRPC client can resolve every pod IP and balance per request, which works around the per-connection balancing of a normal ClusterIP. - Discovery for clustered software that wants the full member list. publishNotReadyAddresses: true publishes pods before they're ready, which helps members find each other while a cluster is bootstrapping. The trade-off is that clients take on the work: they must re-resolve DNS as pods come and go, and not cache old IPs for too long.

Question 36

How do NetworkPolicies work? How would you lock down traffic to an API so only the frontend can reach it?

Accepted Answer

By default the pod network is flat: every pod can reach every other pod, across namespaces. A NetworkPolicy is a layer 3/4 firewall for pods, selected by labels.

It's enforced by the network plugin, such as Calico or Cilium. If the plugin doesn't support NetworkPolicy, the objects are accepted but do nothing.
A pod becomes isolated for ingress or egress as soon as any policy of that type selects it. From then on, only traffic some policy allows gets through. Policies are additive: there's no deny rule and no ordering. Replies to allowed connections are allowed automatically.
Peers are chosen with podSelector, namespaceSelector (both in one entry means AND) or ipBlock, plus ports.

The usual pattern is a default-deny policy per namespace (podSelector: {} with no allow rules), then narrow allow policies like the one shown. If you also deny egress, remember to allow DNS to kube-dns on port 53, or everything breaks.

For layer-7 rules, explicit denies or cluster-wide policies you need CNI-specific resources, such as Cilium's or Calico's own policy types.

Question 37

Compare rolling, blue-green and canary deployments. How would you implement each on Kubernetes?

Accepted Answer

- Rolling update: the Deployment default. Pods are replaced gradually under maxSurge and maxUnavailable. It needs no extra infrastructure, but both versions serve traffic during the rollout, so changes must be backward compatible, and a rollback is just another rollout. (Recreate is the blunt alternative: stop everything, then start the new version, with downtime.) - Blue-green: run the complete new version (green) next to the old one (blue), test it, then switch all traffic at once, for example by changing a Service selector from version: blue to version: green, or repointing a route. Rollback is an instant switch back. The cost is double capacity during the switch. - Canary: send a small share of real traffic to the new version, watch error rates and latency, then increase step by step. The crude way is two Deployments behind one Service, where traffic roughly follows the replica ratio. The precise way is weighted routing: Gateway API backendRefs weights, a service mesh, or ingress controller features. Tools like Argo Rollouts and Flagger automate blue-green and canary, including metric analysis and automatic rollback.

Question 38

Why is etcd so critical, and how do you back it up and restore it?

Accepted Answer

etcd stores all cluster state: every object, including Secrets. Running containers keep going without it for a while, but nothing can be created, changed or rescheduled. Losing it without a backup means rebuilding the cluster. Backup: take a snapshot from a live member with etcdctl snapshot save, using the etcd client TLS certificates, or snapshot the etcd data volume. Check it with etcdutl snapshot status. Run it on a schedule, store copies off the cluster and encrypted (a snapshot holds your Secrets, in plaintext unless encryption at rest is on), and rehearse restores. Restore: stop all API servers, restore the snapshot into a new data directory on each etcd member with etcdutl snapshot restore, point etcd at that directory (on kubeadm, in the etcd static pod manifest), start etcd, then the API servers. Restarting the scheduler, controller manager and kubelets is recommended so nothing acts on stale data. On managed services like EKS, GKE or AKS you can't reach etcd. You back up at the resource level instead, with Velero or GitOps, plus volume snapshots.

Question 39

How does the kube-scheduler decide which node a pod runs on?

Accepted Answer

The scheduler watches for pods with no spec.nodeName and handles each in two phases: - Filtering removes nodes that can't run the pod: not enough unreserved allocatable CPU or memory for its requests, a nodeSelector or required affinity that doesn't match, taints it doesn't tolerate, host port conflicts, volume zone constraints, required pod anti-affinity, unschedulable nodes. - Scoring ranks the remaining nodes with scoring plugins: spreading replicas, balancing resource usage, preferred affinity weights, topology spread, whether the image is already on the node. It picks the highest-scoring node, breaking ties randomly, and binds the pod by setting its node through the API server. The kubelet on that node sees the assignment and starts the containers. If no node passes filtering, the pod stays Pending with a FailedScheduling event. A higher-priority pod, set through a PriorityClass, can trigger preemption, evicting lower-priority pods to make room. The scheduler is built from plugins, supports profiles and can run alongside other schedulers (schedulerName). It only places pods once; it never moves running pods. The separate descheduler project does that.

Question 40

What is a PodDisruptionBudget, and what does it protect against and not protect against?

Accepted Answer

A PDB limits how many pods of an application can be taken down at once by voluntary disruptions: node drains, cluster upgrades, autoscaler scale-downs, anything that goes through the Eviction API. You set minAvailable or maxUnavailable, as a number or a percentage, and a selector.

If an eviction would break the budget, the API server refuses it with 429 Too Many Requests, and kubectl drain keeps retrying until replacement pods are healthy somewhere else. That's how a three-replica service survives a rolling node upgrade.

What it doesn't do:

It can't prevent involuntary disruptions like node crashes or OOM kills, though those count against the budget.
Deleting pods or a Deployment directly bypasses it.
Deployment and StatefulSet rolling updates aren't limited by PDBs; maxUnavailable in the workload controls those.

Pitfalls: minAvailable equal to the replica count, or maxUnavailable: 0, blocks drains forever, and so does minAvailable: 1 on a single-replica app. Crash-looping pods can also block drains; unhealthyPodEvictionPolicy: AlwaysAllow lets them be evicted.

Question 41

Compare the Horizontal Pod Autoscaler, the Vertical Pod Autoscaler and the cluster autoscaler. How do they work together?

Accepted Answer

They scale different things: - HPA changes the number of pods, based on CPU, memory or custom metrics. It's built in. - VPA changes each pod's resource requests (and limits proportionally) based on observed usage. It's an add-on with its own CRD. Modes include Off (recommendations only), Initial (set at pod creation), Recreate (evict pods to apply) and InPlaceOrRecreate, which uses in-place pod resize where it can. - Cluster autoscaler changes the number of nodes. It adds nodes to a node group when pods are Pending for lack of capacity, and removes underused nodes whose pods fit elsewhere, respecting PDBs. Karpenter is an alternative that provisions right-sized nodes directly. They chain naturally: load rises, the HPA adds pods, some can't be scheduled, the cluster autoscaler adds nodes. The VPA keeps requests honest so both decide well, since scheduling works on requests, not usage. The main rule: don't let the HPA and VPA act on the same CPU or memory metric for one workload, or they fight. Use VPA in Off mode, or scale the HPA on custom metrics.

Question 42

What are Custom Resource Definitions and Operators, and how does an operator work?

Accepted Answer

A CustomResourceDefinition extends the Kubernetes API with a new resource type, like PostgresCluster or Certificate. It declares an OpenAPI v3 schema for validation, versions and optionally a status subresource. After that, the API server serves the new type like a built-in one: kubectl get, RBAC and watches all work. But a CRD on its own just stores data. An operator is CRDs plus a custom controller that encodes the operational knowledge for one piece of software. Its reconcile loop: - Watches its custom resources and the objects it creates. - Compares the desired spec with what actually exists. - Acts: creates or updates StatefulSets, Services and Secrets, runs backups, handles failover and version upgrades. - Reports progress in status. Good operators are level-triggered and idempotent. They set owner references so garbage collection removes child objects, and use finalizers to clean up external resources before deletion. Examples include cert-manager, the Prometheus Operator, CloudNativePG and Strimzi. They're usually built with Kubebuilder or the Operator SDK. The trade-offs: another privileged component to run and trust, and CRD version upgrades to manage.

Question 43

Which kubectl commands do you use day to day to inspect, change and debug workloads?

Accepted Answer

Grouped by what I'm doing: - Inspect: kubectl get pods -n shop -o wide, filtered with -l app=web, across namespaces with -A, or live with -w. kubectl describe pod is the first stop because of its events. kubectl get deploy web -o yaml shows the full object. - Logs and access: kubectl logs -c -f, --previous after a crash, kubectl exec -it -- sh, kubectl port-forward svc/web 8080:80, and kubectl debug for images without a shell. - Change: kubectl apply -f, with kubectl diff -f first; kubectl rollout status, history, undo and restart; kubectl scale deploy/web --replicas=5. - Cluster health: kubectl top pods and kubectl top nodes (needs metrics-server), kubectl events, kubectl cordon and kubectl drain, kubectl auth can-i. - Discovery: kubectl explain deployment.spec.strategy, kubectl api-resources. - Scaffolding: kubectl create deployment web --image=nginx --dry-run=client -o yaml to generate a starting manifest. And always check kubectl config current-context before changing anything, so you don't apply to production by accident.

Question 44

How does Kubernetes compare with Docker Compose and Amazon ECS, and when would you pick each?

Accepted Answer

- Docker Compose describes a multi-container app in one YAML file and runs it on a single Docker host. It's ideal for local development and simple single-server setups, but there's no scheduling across machines, no rescheduling when a host dies, and no autoscaling. - Amazon ECS is AWS's managed orchestrator. You write task definitions and services instead of pods and Deployments, and run them on EC2 or serverless on Fargate. There's no control plane to operate, and it integrates tightly with AWS: IAM roles per task, load balancers, CloudWatch. The trade-offs are AWS lock-in and a smaller, less extensible ecosystem. - Kubernetes is the portable open standard across clouds and on-premises, with an extensible API and a huge ecosystem: Helm, operators, service meshes, GitOps. The cost is complexity: upgrades, networking, security and platform skills, even on EKS, GKE or AKS. My rule of thumb: Compose for local development; ECS with Fargate for smaller teams all-in on AWS who want minimal operations; Kubernetes when you need portability, hybrid or multi-cloud, many teams and services, or its ecosystem.

Question 45

What are the most important security best practices for workloads running on Kubernetes?

Accepted Answer

I think in layers: - Pod hardening: run as a non-root numeric UID, block privilege escalation, drop all capabilities, use a read-only root filesystem and the RuntimeDefault seccomp profile. Avoid privileged, hostNetwork, hostPID and hostPath. Enforce it with Pod Security Admission at restricted. - Identity and access: least-privilege RBAC, one ServiceAccount per app, token automount off where unused, and no cluster-admin for CI pipelines. - Network: default-deny NetworkPolicies with explicit allows, and mTLS for sensitive traffic. - Supply chain: minimal or distroless images, CVE scanning, pinning by digest, signing images with cosign and verifying signatures at admission, and only trusted registries. - Secrets: encryption at rest, ideally with KMS, or an external secret store, and never secrets baked into images. - Cluster: stay on a supported, patched version, keep the API endpoint private, enable audit logging, lock down etcd and kubelet access, and set resource limits so one workload can't starve a node.

Question 46

What are the Pod Security Standards, and how does Pod Security Admission enforce them?

Accepted Answer

The Pod Security Standards define three profiles: - Privileged: unrestricted, for trusted system workloads such as CNI or storage agents. - Baseline: blocks known privilege escalations, such as privileged containers, host namespaces, hostPath volumes, host ports and dangerous capabilities. - Restricted: current hardening practice. On top of Baseline, pods must run as non-root, set allowPrivilegeEscalation: false, drop ALL capabilities (only NET_BIND_SERVICE may be added back), use a RuntimeDefault or Localhost seccomp profile, and use only a limited set of volume types. Pod Security Admission is the built-in admission controller that replaced PodSecurityPolicy, which was removed in v1.25. You apply it per namespace with labels such as pod-security.kubernetes.io/enforce: restricted. There are three modes: enforce rejects violating pods, audit records them in the audit log, and warn returns a warning to the user. A safe rollout is to enforce baseline with warn and audit set to restricted, fix the violations, then enforce restricted. Enforcement applies to pods, so a Deployment can be accepted while its pods are rejected; look at the ReplicaSet's events. For custom rules, use ValidatingAdmissionPolicy, Kyverno or Gatekeeper.

Question 47

Pods restart every few minutes with Liveness probe failed, or never become Ready. How do you debug probe failures?

Accepted Answer

Start with kubectl describe pod: the events say exactly how the probe failed, and the message points to the cause. - "connection refused": nothing is listening on that port. The probe targets the wrong port, the app is still starting, or it's bound to 127.0.0.1 while the kubelet probes the pod IP. - Timeout ("context deadline exceeded"): timeoutSeconds defaults to just 1 second. A slow health endpoint, a GC pause or CPU throttling makes it fail. Make the endpoint cheap, or raise the timeout. - Bad status code: anything outside 200–399 fails, for example a 401 because the health route requires authentication. - Killed while starting: liveness runs before the app is up, causing a restart loop. Add a startupProbe whose failureThreshold × periodSeconds covers the worst-case startup. - Never Ready: check what the readiness endpoint depends on; a missing downstream service keeps it false. Then reproduce the probe yourself: kubectl exec into the pod and curl the endpoint, or port-forward to it. kubectl get endpointslices -l kubernetes.io/service-name=web shows which endpoints are ready.

Question 48

What is Gateway API, and how does it improve on Ingress?

Accepted Answer

Gateway API is the successor to Ingress, developed by SIG Network. It's an add-on: you install its CRDs, and implementations such as Envoy Gateway, Istio, Cilium or cloud load balancers provide the controller.

What it improves:

Role-oriented resources instead of one overloaded object. A GatewayClass (the infrastructure provider) says which controller to use. A Gateway (the cluster operator) defines listeners, ports, TLS and which namespaces may attach routes. Routes such as HTTPRoute and GRPCRoute (the app teams) attach to a Gateway through parentRefs.
Expressive, portable features in the spec: header and query matching, weighted traffic splitting, redirects, rewrites, header changes and mirroring. With Ingress those needed controller-specific annotations.
Safe cross-namespace sharing: a Gateway's allowedRoutes controls who can attach, and ReferenceGrant is required to reference backends in another namespace.
More protocols: HTTP and gRPC routes are stable; others exist at different maturity levels.

GatewayClass, Gateway, HTTPRoute and GRPCRoute are stable. With the Ingress API frozen, it's the recommended path, and the ingress2gateway tool helps migrate.

Question 49

What is a service mesh, what does it give you, and when is it worth the cost?

Accepted Answer

A service mesh is an infrastructure layer that handles service-to-service traffic without changing application code. Its data plane is a set of proxies that intercept traffic: traditionally a sidecar in every pod (Envoy in Istio, Linkerd's own proxy), or per-node proxies as in Istio's ambient mode. Its control plane configures the proxies and issues workload certificates.

What you get:

Security: automatic mTLS between services, with workload identities and authorization policies such as "only checkout may call payments".
Traffic control: retries, timeouts, circuit breaking, per-request load balancing (which fixes gRPC imbalance), weighted canaries, mirroring and fault injection.
Observability: consistent latency, error and throughput metrics for every hop, plus tracing spans, though apps must still propagate trace headers.

The costs are real: another complex system to run and upgrade, extra latency and CPU and memory per hop, and one more layer to debug.

It's worth it with many services, a zero-trust or mTLS requirement, or advanced traffic management needs. For a handful of services, Gateway API, NetworkPolicies and good client libraries are usually enough.

Question 50

What happens when a pod is terminated, and how do you make rolling deployments truly zero-downtime?

Accepted Answer

When a pod is deleted, by a rollout, a scale-down or a drain:

It's marked terminating and its grace period starts (30 seconds by default).
In parallel, the control plane marks its endpoints as not ready, so kube-proxy, ingress controllers and load balancers stop sending new traffic, while the kubelet runs any preStop hook and then sends SIGTERM to each container's PID 1.
Whatever is still running when the grace period ends gets SIGKILL.

The catch is that race. SIGTERM can arrive before every node and proxy has stopped routing to the pod, so an app that exits instantly drops requests.

Zero-downtime checklist:

Handle SIGTERM: stop accepting connections, finish in-flight requests, then exit. Make sure your process is PID 1 or runs under an init that forwards signals; a shell-form entrypoint often swallows SIGTERM.
Add a short preStop sleep so routing updates propagate first.
Set terminationGracePeriodSeconds longer than the preStop plus drain time, since preStop counts against it.
Use readiness probes, maxUnavailable: 0, and PDBs for node drains.

Kubernetes interview questions & answers

Top 58 Kubernetes interview questions most asked first