Kubernetes

The Kubernetes objects, defaults and failure modes interviewers ask about, from the control plane down to a pod stuck in CrashLoopBackOff.

Architecture

• Declarative model: you store desired state in the API; controllers run loops that observe the actual state and act to close the gap (reconciliation).

• Add-ons: CoreDNS (service discovery), a CNI plugin (Calico, Cilium, Flannel), metrics-server (kubectl top, HPA), an Ingress or Gateway controller.
• Networking model: every pod gets its own IP and can reach every other pod without NAT; containers in a pod share localhost.

Core objects

• Service DNS: web.prod.svc.cluster.local (just web from the same namespace). StatefulSet pods get db-0.db.prod.svc.cluster.local.
• externalTrafficPolicy: Local keeps the client source IP and skips the extra hop, but only nodes with a ready pod receive traffic.

Deployment + Service

apiVersion: apps/v1
kind: Deployment
metadata: { name: web }
spec:
  replicas: 3
  selector: { matchLabels: { app: web } }
  template:
    metadata: { labels: { app: web } }
    spec:
      containers:
        - { name: web, image: nginx:1.27, ports: [{ containerPort: 80 }] }

apiVersion: v1
kind: Service
metadata: { name: web }
spec:
  selector: { app: web }
  ports: [{ port: 80, targetPort: 80 }]

• selector.matchLabels must match the template labels and is immutable after creation. The Service finds pods by label, not by Deployment.
• port is the Service’s port, targetPort the container’s, nodePort the port opened on every node (NodePort/LoadBalancer only).
• Defaults: strategy: RollingUpdate with maxSurge: 25% and maxUnavailable: 25%, revisionHistoryLimit: 10, progressDeadlineSeconds: 600.

Probes

• Handlers: httpGet (status 200–399 passes), tcpSocket, exec (exit 0 passes), grpc.
• Defaults: initialDelaySeconds 0, periodSeconds 10, timeoutSeconds 1, successThreshold 1 (must be 1 for liveness and startup), failureThreshold 3.

# per container
resources:
  requests: { cpu: 250m, memory: 256Mi }
  limits: { memory: 256Mi }
startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  failureThreshold: 30
  periodSeconds: 10
readinessProbe:
  httpGet: { path: /ready, port: 8080 }
livenessProbe:
  httpGet: { path: /healthz, port: 8080 }

Requests, limits & QoS

• Requests are what the scheduler reserves (a pod fits if the node’s allocatable minus other requests covers it). Limits are enforced at run time.
• Over the CPU limit the container is throttled; over the memory limit it is OOMKilled (exit 137).
• Units: CPU 1 = one vCPU/core, 250m = a quarter. Memory Mi/Gi are binary; 128m means 0.128 bytes.
• Only a limit set? The request defaults to the limit.

• LimitRange sets per-container defaults and min/max in a namespace; ResourceQuota caps a namespace’s totals (and then pods must declare those resources).
• In-place resize (stable since v1.35) changes a running pod’s CPU/memory through the resize subresource without recreating it.

Scaling

• HPA formula: desired = ceil(current × currentMetric / targetMetric). It re-evaluates every 15 s and by default waits a 5-minute stabilization window before scaling down.
• kubectl autoscale deployment web --min=2 --max=10 --cpu=70% (older kubectl: --cpu-percent=70). Manual: kubectl scale deployment web --replicas=5.
• Don’t let HPA and VPA both act on the same CPU/memory metric.

Scheduling

• nodeSelector: { disktype: ssd } is the simplest node constraint.
• Node affinity: requiredDuringSchedulingIgnoredDuringExecution (hard) or preferredDuringScheduling… (soft, weighted). “IgnoredDuringExecution”: later label changes don’t evict running pods.
• Pod (anti-)affinity with a topologyKey (kubernetes.io/hostname, topology.kubernetes.io/zone) co-locates or separates pods; topologySpreadConstraints (maxSkew, whenUnsatisfiable) spread replicas evenly across zones or nodes.
• Taints repel, tolerations allow: kubectl taint nodes n1 gpu=true:NoSchedule; remove with kubectl taint nodes n1 gpu:NoSchedule-.

• A toleration doesn’t attract pods: dedicate nodes with a taint plus node affinity.

RBAC & NetworkPolicy

• Role (namespaced) and ClusterRole (cluster-wide or reusable) list rules: apiGroups, resources, verbs (get, list, watch, create, update, patch, delete).
• RoleBinding / ClusterRoleBinding grant a role to subjects: User, Group or ServiceAccount. A RoleBinding may reference a ClusterRole to grant it in one namespace only.
• Permissions are purely additive (no deny rules). Users aren’t API objects; they come from certificates or OIDC.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: pod-reader, namespace: dev }
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "list", "watch"]

• Bind it and test it: kubectl create rolebinding ci-read --role=pod-reader --serviceaccount=dev:ci -n dev, then kubectl auth can-i list pods --as=system:serviceaccount:dev:ci -n dev.
• NetworkPolicy (L3/L4 only) needs a CNI that enforces it; otherwise it silently does nothing. Pods are open until a policy selects them for a direction; policies then allow the union of their rules.
• Default deny: podSelector: {} with policyTypes: [Ingress] and no rules. Remember to allow DNS (port 53) when you deny egress.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: api-from-frontend }
spec:
  podSelector: { matchLabels: { app: api } }
  policyTypes: [Ingress]
  ingress:
    - from: [{ podSelector: { matchLabels: { app: frontend } } }]
      ports: [{ protocol: TCP, port: 8080 }]

Rollouts & deployment strategies

kubectl set image deployment/web web=nginx:1.28   # template change = rollout
kubectl rollout status deployment/web
kubectl rollout history deployment/web
kubectl rollout undo deployment/web --to-revision=2
kubectl rollout restart deployment/web            # new pods, same spec
kubectl rollout pause deployment/web              # batch edits, then resume

• Zero-downtime needs readiness probes, enough replicas, a PodDisruptionBudget for drains, and graceful shutdown: the pod gets SIGTERM, then SIGKILL after terminationGracePeriodSeconds (default 30).

kubectl cheat table

Troubleshooting

Start with kubectl get pods → describe pod (read the Events) → logs --previous → exec or debug.

Helm basics

• Chart = templated package (Chart.yaml, values.yaml, templates/); release = an installed instance with revision history; charts come from repos or OCI registries.
• helm install web ./chart -f values-prod.yaml --set image.tag=1.4.2; helm upgrade --install web ./chart --wait is the idempotent CI form.
• helm list, helm history web, helm rollback web 3, helm uninstall web, helm template (render locally), helm lint, helm show values repo/chart.
• Value precedence: chart defaults < -f files (the last one wins) < --set.
• Helm 3 dropped Tiller and stores release state as Secrets in the release’s namespace. Helm 4 (Nov 2025) uses server-side apply for new releases and renames --atomic to --rollback-on-failure.
• Helm templates and versions packages; Kustomize patches plain YAML with overlays (kubectl apply -k).

Security best practices

• Least-privilege RBAC with one ServiceAccount per app; automountServiceAccountToken: false when the pod never calls the API.
• securityContext: runAsNonRoot: true, allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: ["ALL"] }, seccompProfile: { type: RuntimeDefault }.
• Enforce Pod Security Standards (privileged, baseline, restricted) with the namespace label pod-security.kubernetes.io/enforce: restricted.
• Default-deny NetworkPolicies, then allow specific flows.
• Encrypt Secrets at rest (a KMS provider), or sync them from an external manager; restrict get on Secrets, which exposes their values.
• Pin images by digest, scan them, and enforce policy at admission (ValidatingAdmissionPolicy, Kyverno, OPA Gatekeeper).
• Keep the API server private, enable audit logs, upgrade regularly, and never mount the host’s container runtime socket.

Gotcha

Base64 is encoding, not encryption: anyone who can read a Secret object, or etcd, can read its value. The same goes for anyone who can create pods in the namespace, since a pod can mount any Secret there.

Quick answers

• Pod vs container? A pod is the scheduling unit: one or more containers that share an IP, ports and volumes and always run on the same node.
• Deployment vs StatefulSet? Deployments run interchangeable pods; StatefulSets give each pod a stable name, stable storage and ordered rollout.
• Why a Service? Pod IPs change; a Service gives a stable virtual IP and DNS name and balances across ready pods.
• ClusterIP vs NodePort vs LoadBalancer? Internal only; plus a port on every node; plus a cloud load balancer in front.
• Liveness vs readiness? Liveness failure restarts the container; readiness failure only takes it out of load balancing.
• Requests vs limits? Requests drive scheduling; limits cap usage (CPU throttled, memory OOM-killed).
• ConfigMap vs Secret? Same shape; Secrets are for sensitive data, get tighter RBAC and can be encrypted at rest.
• What happens on kubectl apply? API server authenticates, authorizes, runs admission and stores the object in etcd; the Deployment controller creates a ReplicaSet, which creates pods; the scheduler binds them; the kubelet starts them.
• How do you roll back? kubectl rollout undo deployment/web (optionally --to-revision); old ReplicaSets are kept for that.
• What is etcd? The Raft-based key-value store holding all cluster state; lose it without a backup and you lose the cluster.
• Ingress vs Gateway API? Both route HTTP into the cluster; Ingress is frozen, and Gateway API is its role-oriented, more expressive successor.
• Does a namespace isolate workloads? Only by name; add RBAC, NetworkPolicies and quotas for real isolation.
• Why is my pod Pending? The scheduler can’t place it: check the events for insufficient resources, taints, affinity or an unbound PVC.

Gotchas & traps

• Re-pushing the same tag doesn’t roll anything out: the pod template didn’t change. Use immutable tags or digests (or rollout restart).
• An omitted tag or :latest defaults imagePullPolicy to Always; any other tag defaults to IfNotPresent.
• A liveness probe that checks the database restarts every pod when the database blips.
• CPU limits throttle even on an idle node; many teams set CPU requests without CPU limits but always set memory limits.
• ConfigMap and Secret changes reach mounted files eventually, but never env vars or subPath mounts: restart the pods.
• An RWO volume plus RollingUpdate can deadlock: the new pod lands on another node and hits a Multi-Attach error. Use Recreate or a StatefulSet.
• After SIGTERM, endpoint removal and shutdown race, so a pod may still get requests: handle them, or add a short preStop sleep.
• Deleting a namespace deletes everything in it; it hangs in Terminating if a finalizer can’t complete.
• A NetworkPolicy on a cluster whose CNI doesn’t enforce it is silently ignored.