Containers lose their writable layer when they restart, and Pods lose everything when they’re rescheduled. Anything that must outlive a Pod (a database’s data directory, uploaded files, a message broker’s log) needs storage whose lifecycle is separate from the Pod’s. Kubernetes splits that into two objects so that the people who write applications don’t need to know whether the disk underneath is an AWS EBS volume, a Ceph RBD image or an NFS export.
The examples target Kubernetes 1.30 or later with a CSI driver such as the AWS EBS driver installed. Manifests were parsed for valid YAML; kubectl output is a representative sample.
Claims, volumes and the binding between them
A PersistentVolume (PV) is a piece of storage the cluster knows about: its size, its access modes, and how to attach it (a CSI driver name and volume handle). PVs are cluster-scoped.
A PersistentVolumeClaim (PVC) is a namespaced request: “I need 20Gi, mountable read-write by one node, from storage class gp3”. The control plane finds a PV that satisfies it, or creates one, and binds them. Binding is one-to-one. A bound PV belongs to that claim and no other claim can take it, even if the PV is far larger than requested.
The Pod never references the PV. It references the claim:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: orders-db-data
namespace: shop
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: gp3
resources:
requests:
storage: 20Gi
---
apiVersion: v1
kind: Pod
metadata:
name: orders-db
namespace: shop
spec:
containers:
- name: postgres
image: postgres:16
env:
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
volumes:
- name: data
persistentVolumeClaim:
claimName: orders-db-dataThe PGDATA subdirectory is a small but real detail: many block volumes are formatted with ext4 and contain a lost+found directory at their root, and Postgres refuses to initialise a non-empty data directory. Pointing it one level down avoids that.
In practice you’d run Postgres as a StatefulSet with volumeClaimTemplates so each replica gets its own claim. A bare Pod keeps the example small.
Dynamic provisioning and StorageClasses
Nobody wants to pre-create PVs by hand. A StorageClass tells Kubernetes which provisioner to call and with what parameters when a claim asks for that class:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp3
provisioner: ebs.csi.aws.com
parameters:
type: gp3
encrypted: "true"
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: trueWhen the PVC above is created, the external provisioner sidecar of the EBS CSI driver sees it, calls the cloud API to create a 20Gi gp3 volume, creates a PV describing it, and the PV controller binds them. A claim with no storageClassName gets the cluster’s default class (the one annotated storageclass.kubernetes.io/is-default-class: "true"). A claim with storageClassName: "" explicitly asks for no class, meaning “bind me to a pre-created PV only”.
volumeBindingMode deserves attention, because it’s behind a classic production bug. With Immediate, the volume is created the moment the claim exists. An EBS volume lives in one availability zone. If it gets created in eu-west-1a and the scheduler later finds room for the Pod only in eu-west-1b, the Pod can’t attach it and stays Pending with a volume node affinity conflict event. WaitForFirstConsumer delays provisioning until a Pod using the claim is scheduled, then creates the disk in that Pod’s zone. That’s why the claim shows Pending until a Pod exists:
$ kubectl get pvc -n shop
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
orders-db-data Pending gp3 12s
$ kubectl describe pvc orders-db-data -n shop | tail -2
Normal WaitForFirstConsumer 10s persistentvolume-controller
waiting for first consumer to be created before bindingThat Pending is expected, not a failure. Interviewers sometimes present it as a broken claim to see whether you read the event.
Access modes say less than people think
Access modes describe how a volume may be mounted:
ReadWriteOnce(RWO): read-write by a single node. Several Pods on that same node can all mount it.ReadOnlyMany(ROX): read-only by many nodes.ReadWriteMany(RWX): read-write by many nodes. Needs a filesystem that supports it (NFS, CephFS, EFS, Azure Files); block devices like EBS can’t.ReadWriteOncePod(RWOP): read-write by a single Pod across the whole cluster. Stable since Kubernetes 1.29 and supported only by CSI volumes.
The RWO “single node” wording trips people up. Two replicas of a Deployment on the same node can both mount an RWO volume and both write to it, which for a database is a corruption risk, not a safety net. If you need the guarantee that only one Pod ever writes, use RWOP.
RWO also explains why Deployments with a single shared RWO claim deadlock during rolling updates. The new Pod lands on a different node, the volume is still attached to the old Pod’s node, and the new Pod waits with Multi-Attach error for volume ... Volume is already used by pod(s) orders-db-6f9.... Meanwhile the old Pod won’t be removed until the new one is ready. Fix it with strategy: Recreate or by moving to a StatefulSet.
Access modes are also not enforced as permissions inside the filesystem. A claim with RWX doesn’t make files writable by your container’s user; that’s fsGroup and file ownership, covered in the security contexts guide.
Reclaim policy and how data gets lost
When a bound claim is deleted, the PV’s persistentVolumeReclaimPolicy decides what happens:
- Delete (the default for dynamically provisioned volumes in most classes): the PV and the underlying cloud disk are deleted.
- Retain: the PV moves to
Releasedand the disk stays. Nobody can bind to it until an administrator cleans it up or removes the oldclaimRef.
So kubectl delete namespace shop with a Delete class removes every database disk in the namespace, permanently. For anything you can’t regenerate, either use a class with reclaimPolicy: Retain or patch the PV after provisioning:
kubectl patch pv pvc-3f1c9a2e-... -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'Kubernetes adds one safety net: the kubernetes.io/pvc-protection finalizer stops a claim from being removed while a Pod still uses it, so it sits in Terminating instead. That protects against accidents during normal operation, not against deleting the workload and the claim together. Backups (CSI VolumeSnapshot objects or a tool like Velero) are still your real protection.
Growing a volume
With allowVolumeExpansion: true you edit the claim’s spec.resources.requests.storage from 20Gi to 50Gi. The CSI driver resizes the disk, and the filesystem is grown by the kubelet, online for most drivers. Shrinking isn’t supported. The claim’s status.capacity updates once the filesystem resize finishes; if it’s stuck, the claim’s conditions will show FileSystemResizePending.
Interview practice
“A team’s Postgres Pod has been Pending for ten minutes after a node failure moved it. kubectl describe pod shows 1 node(s) had volume node affinity conflict, 5 node(s) didn't match Pod's node affinity/selector. The cluster spans three zones. What happened, and how do you prevent it?”
Answer
The Pod’s PVC is bound to a zonal disk, and the PV carries node affinity for that zone. The node that failed was the only node in that zone the Pod could use (the other five are excluded by the Pod’s own node selector, or are in other zones), so the scheduler can’t find a node that can attach the disk. The data is fine; the Pod just can’t reach it from another zone.
Immediate fixes: bring up capacity in the volume’s zone (scale that zone’s node group, or remove the overly strict node selector if it’s excluding usable nodes there). For prevention, check that the StorageClass uses WaitForFirstConsumer so initial placement and disk zone agree, and that every zone holding volumes has spare nodes, for example with one node group per zone so the autoscaler can add nodes where the disk is. If the database must survive a whole-zone outage, that’s a replication question: run replicas in each zone with their own volumes, because a block volume will never follow a Pod across zones.