Deployments treat Pods as interchangeable: any replica can serve any request, and a replacement has a fresh random name and an empty disk. Databases, message brokers and consensus systems don’t work like that. A Kafka broker has a broker ID, a ZooKeeper member has a server number in a fixed ensemble, a Postgres replica has its own copy of the data. They need each instance to come back as the same instance. That’s what a StatefulSet adds, and an interview usually tests whether you know exactly how far its guarantees go.
Version baseline: Kubernetes 1.30 or newer. The manifests parse as valid YAML; DNS records and PVC listings are representative examples.
The three identities a StatefulSet gives each Pod
Take a three-replica Postgres cluster named pg:
apiVersion: v1
kind: Service
metadata:
name: pg
namespace: data
spec:
clusterIP: None
selector:
app: pg
ports:
- name: postgres
port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: pg
namespace: data
spec:
serviceName: pg
replicas: 3
selector:
matchLabels:
app: pg
template:
metadata:
labels:
app: pg
spec:
terminationGracePeriodSeconds: 60
containers:
- name: postgres
image: postgres:16
ports:
- name: postgres
containerPort: 5432
env:
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: gp3
resources:
requests:
storage: 50Gi(A real Postgres setup also needs a password Secret and replication configuration; they’re left out to keep attention on the StatefulSet.)
A stable name. Pods are named <statefulset>-<ordinal>: pg-0, pg-1, pg-2. If pg-1 is deleted or its node dies, the controller creates a new Pod called pg-1. Never pg-3, never a random suffix. The ordinal is also exposed as the label apps.kubernetes.io/pod-index, which is handy for configuration scripts.
A stable network identity. serviceName: pg points at the headless Service (clusterIP: None). A headless Service has no virtual IP; DNS returns Pod IPs directly. Combined with a StatefulSet, each Pod also gets its own DNS record:
pg-0.pg.data.svc.cluster.local -> 10.244.2.31
pg-1.pg.data.svc.cluster.local -> 10.244.1.12
pg-2.pg.data.svc.cluster.local -> 10.244.3.40The IP behind pg-1 changes when the Pod is recreated, but the name doesn’t, so replicas can be configured to follow pg-0.pg.data.svc.cluster.local as their primary. One subtlety interviewers like: DNS records for a Pod generally appear only once it’s Ready (unless the Service sets publishNotReadyAddresses: true), so a member that must find its peers before it’s Ready needs that setting. Also, the headless Service must exist; the StatefulSet doesn’t create it, and a typo in serviceName means no per-Pod DNS.
Stable storage. volumeClaimTemplates creates one PVC per Pod, named <template>-<statefulset>-<ordinal>:
$ kubectl get pvc -n data
NAME STATUS VOLUME CAPACITY STORAGECLASS
data-pg-0 Bound pvc-0b1e7c3a-... 50Gi gp3
data-pg-1 Bound pvc-5d2f9a81-... 50Gi gp3
data-pg-2 Bound pvc-a7c4e2d9-... 50Gi gp3When pg-1 is recreated, on whichever node, it mounts data-pg-1 again. Its data comes back with its name. That’s the property a Deployment can’t give you: a Deployment with a PVC in its Pod template shares one claim among all replicas.
Ordering: startup, scaling and updates
With the default podManagementPolicy: OrderedReady, the controller creates pg-0, waits for it to be Running and Ready, then creates pg-1, and so on. Scaling down removes the highest ordinal first, waiting for each to terminate completely. This suits systems where the first member bootstraps the cluster and later members join it.
The cost is that one stuck Pod stops everything after it. If pg-1 never becomes Ready (bad config, a probe that fails), pg-2 is never created. podManagementPolicy: Parallel launches and removes Pods all at once, for systems that coordinate membership themselves. It only affects scaling, not updates.
Updates with the default RollingUpdate strategy go in reverse ordinal order: pg-2, then pg-1, then pg-0, each deleted, recreated with the new template, and waited on until Ready. One Pod at a time: a maxUnavailable field for StatefulSets has existed behind a feature gate since 1.24 (alpha) and has only recently moved toward general availability, so don’t count on it in every cluster. If the new version is broken and pg-2 never becomes Ready, the rollout stops there, leaving two Pods on the old version.
Two update knobs come up:
partition:rollingUpdate.partition: 2means only Pods with ordinal ≥ 2 get the new template. It’s a built-in canary: updatepg-2, watch it, then lower the partition.OnDelete: the controller doesn’t replace Pods at all; each gets the new template only when you delete it. Operators of databases often use this to control failover manually.
There’s one known trap with rollbacks: if a bad template leaves a Pod stuck (for example, never Ready because the image doesn’t exist), reverting the template doesn’t always fix it automatically. With OrderedReady, the docs say you may need to delete the stuck Pod after reverting so it’s recreated from the good template.
What happens to storage when you scale or delete
Scale pg from 3 to 2 and pg-2 is deleted, but data-pg-2 stays. Scale back to 3 and the new pg-2 reattaches the old data. Delete the whole StatefulSet and, by default, all three claims stay. That’s deliberately conservative: losing a database’s disks because of a scaling command is far worse than paying for unused volumes.
You can choose otherwise with persistentVolumeClaimRetentionPolicy (stable since 1.32, beta and on by default in 1.27 to 1.31):
spec:
persistentVolumeClaimRetentionPolicy:
whenDeleted: Retain
whenScaled: DeletewhenScaled: Delete cleans up claims of removed ordinals, sensible for caches. whenDeleted: Delete removes all claims with the StatefulSet. Whether the underlying disk then disappears depends on the StorageClass reclaim policy, covered in the persistent storage guide.
What a StatefulSet doesn’t do
This is where interview answers often overreach. A StatefulSet doesn’t:
- replicate data.
data-pg-1starts empty. Postgres streaming replication, Kafka partition replication, Elasticsearch shard allocation: the application does that. - know which Pod is primary.
pg-0isn’t special to Kubernetes. If your scripts assumepg-0is always primary and it fails, someone (Patroni, an operator, a human) has to promote another. - give you fast failover. If the node running
pg-1becomes unreachable, the controller won’t create a newpg-1until the old one is confirmed gone, because two Pods with the same identity writing the same volume is the outcome it exists to prevent. The Pod sits inTerminatingorUnknown. You either wait for the node to recover, delete the Node object once you know it’s really down, or use the out-of-service taint (node.kubernetes.io/out-of-service) that lets the control plane force-detach volumes and recreate the Pod. - back anything up.
That’s why most production databases on Kubernetes run under an operator (CloudNativePG, Strimzi, the Elastic operator). The operator uses StatefulSets or Pods with PVCs underneath, and adds the logic for primaries, replication and failover.
Interview practice
“You run a three-node etcd cluster as a StatefulSet with OrderedReady. You push a config change with a typo. etcd-2 restarts and goes into CrashLoopBackOff; etcd-0 and etcd-1 are untouched. What state is the cluster in, and how do you recover?”
Answer
Updates go in reverse ordinal order, one at a time, waiting for Ready. etcd-2 got the new template first and isn’t becoming Ready, so the controller stops: etcd-1 and etcd-0 keep the old, working revision. etcd still has two of three members, which is quorum, so the cluster serves reads and writes, but it can’t tolerate another failure right now. That makes the fix urgent rather than an emergency.
To recover, revert the template (kubectl rollout undo statefulset/etcd, or reapply the previous manifest). Because the controller may wait for the broken Pod to become Ready before acting further, I’d then delete etcd-2 so it’s recreated from the reverted template; its PVC data-etcd-2 is untouched, so it rejoins with its existing data and member ID. I’d check etcdctl endpoint health across all three members before doing anything else. Going forward, I’d use partition: 2 to canary changes on one member, plus a readiness probe that checks membership health, so a bad change stalls on one member instead of risking quorum.