Kubernetes Pod Stuck in Pending: '0/3 nodes are available' Fixed

You apply a Deployment and one or more Pods sit in Pending for minutes. kubectl describe pod ends with an event like this on Kubernetes 1.30 and later:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  42s   default-scheduler  0/3 nodes are available: 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.

Pending means the Pod object exists but at least one condition for running it is unmet. With a FailedScheduling event, the scheduler checked every node, found none that passes all its filters, and tells you how many nodes failed each filter. It retries automatically, so the Pod starts as soon as the cause goes away.

Quick fix checklist

• Run kubectl describe pod <name> and read the FailedScheduling message word by word.
• Insufficient cpu or Insufficient memory: compare the Pod’s requests with kubectl describe node allocatable and “Allocated resources”.
• didn't match Pod's node affinity/selector: compare nodeSelector/affinity with kubectl get nodes --show-labels.
• had untolerated taint: add the right toleration or schedule elsewhere; don’t strip taints from control-plane nodes.
• unbound immediate PersistentVolumeClaims: check kubectl get pvc and the StorageClass.
• No Pod at all? Check kubectl describe rs or the Job for exceeded quota.
• With a cluster autoscaler, look for TriggeredScaleUp or NotTriggerScaleUp events on the Pod.

Before you start

You need kubectl access with permission to read Pods, Nodes, PVCs, events and ResourceQuotas in the namespace. Reading node details (kubectl describe node) needs cluster-scoped read access, which some managed setups restrict. Know the difference between Pod status columns: Pending with no node assigned is a scheduling problem; ContainerCreating is also phase Pending but the Pod already has a node and is pulling images or mounting volumes, which is a different investigation.

Why it happens

The scheduler places a Pod in two phases. Filtering removes nodes that cannot run it; scoring ranks the rest. If filtering removes every node, the Pod stays unscheduled. The main filters:

• Resources. The sum of container requests must fit in the node’s allocatable capacity minus requests of Pods already there. Actual usage is irrelevant: a node at 5% CPU is “full” if its Pods request all of it. A Pod with no requests needs nothing and is never rejected for resources (it simply lands wherever).
• Node selection. nodeSelector and required nodeAffinity must match node labels exactly. A typo in a label value excludes every node.
• Taints and tolerations. A NoSchedule taint repels Pods that lack a matching toleration. Control-plane nodes carry node-role.kubernetes.io/control-plane:NoSchedule, which is why a three-node cluster often reports one node rejected for a taint.
• Volumes. A PVC that isn’t bound blocks a Pod using an Immediate StorageClass. With volumeBindingMode: WaitForFirstConsumer the PVC stays Pending until the Pod is scheduled, which is normal, but zonal volumes then restrict which nodes qualify.
• Topology and anti-affinity. Required Pod anti-affinity or topologySpreadConstraints with whenUnsatisfiable: DoNotSchedule can rule out nodes that have spare capacity.

After filtering fails, the scheduler tries preemption: evicting lower-priority Pods to make room. The second half of the message reports that attempt. A cluster autoscaler, if installed, watches for unschedulable Pods and adds a node when a node group’s template would fit the Pod.

Quotas behave differently. A ResourceQuota is enforced by admission, before the Pod is stored. A Pod that would exceed it is rejected outright, so you see fewer Pods than replicas, not a Pending Pod.

Step-by-step walkthrough

Step 1: Read the scheduler’s reasons

kubectl get pods -o wide            # NODE column is <none> for unscheduled Pods
kubectl describe pod web-7d9f6c-abcde | sed -n '/Events:/,$p'
kubectl get events --field-selector reason=FailedScheduling --sort-by=.lastTimestamp

The message is a tally. 0/5 nodes are available: 2 Insufficient memory, 3 node(s) didn't match Pod's node affinity/selector. means three nodes were excluded by labels and the two that matched were too full. Fixing only memory or only labels won’t help unless you address what applies to the nodes you actually want.

Step 2: Compare requests with allocatable capacity

kubectl get pod web-7d9f6c-abcde -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.resources.requests}{"\n"}{end}'
kubectl describe node worker-1 | sed -n '/Allocatable:/,/Events:/p'

“Allocated resources” shows requests already committed on the node as a percentage of allocatable. If the Pod asks for cpu: 2 and every node has 1.5 cores of headroom, nothing fits, even though the cluster in total has plenty. Init containers count too: the effective request is the larger of the biggest init container and the sum of app containers (plus sidecars).

Step 3: Check labels, taints and volumes

kubectl get nodes --show-labels
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
kubectl get pvc
kubectl describe pvc data-web-0

For a PVC stuck Pending, its own events say why: no default StorageClass, a misspelled storageClassName, or a provisioner error. A PV in another zone than every eligible node gives node(s) had volume node affinity conflict.

Step 4: Fix the cause at its source

Pick the fix that matches the reason:

resources:
  requests:
    cpu: 250m        # measured need, not a guess of 2 cores
    memory: 256Mi
  limits:
    memory: 512Mi
nodeSelector:
  kubernetes.io/arch: amd64       # a label that exists on the nodes
tolerations:
  - key: dedicated
    operator: Equal
    value: batch
    effect: NoSchedule

Only add a toleration when the Pod belongs on those nodes; a toleration allows scheduling there, it does not attract the Pod. Pair it with a selector if the Pod must land there.

Step 5: Check quotas and autoscaling

kubectl describe rs -l app=web | sed -n '/Events:/,$p'
kubectl describe resourcequota
kubectl describe pod web-7d9f6c-abcde | grep -i scale

A quota failure appears as FailedCreate with exceeded quota: compute, requested: requests.cpu=500m, used: requests.cpu=3800m, limited: requests.cpu=4. An autoscaler that can’t help emits NotTriggerScaleUp with a reason such as the node group being at its maximum size or no group whose nodes would satisfy the selector.

Worked scenario

A team deploys a reporting service to a three-node cluster (one control-plane node, two workers with 4 CPUs each). Two of three replicas stay Pending:

0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 Insufficient cpu. preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod.

The manifest:

containers:
  - name: report
    image: registry.example.com/report:1.8.0
    resources:
      requests:
        cpu: "3"
        memory: 1Gi

Diagnosis: the control-plane node is correctly excluded. Each worker has about 3.8 CPUs allocatable after system reservations, and other Pods already request 1.5 on each, so neither can fit a 3-CPU request, though the first replica fit before those Pods arrived. Metrics showed the service uses about 300m at peak; the 3-CPU request was copied from a JVM batch job. The fix sets a measured request:

    resources:
      requests:
        cpu: 500m
        memory: 1Gi
      limits:
        memory: 1536Mi

All three replicas schedule within seconds. Nobody removed the control-plane taint or added nodes.

Common mistake

Removing the request to make the Pod schedule. It works, but the Pod becomes BestEffort (or Burstable with lopsided values), is the first to be evicted under node pressure, and the scheduler can pack too much onto one node. Set requests from measured usage instead.

Removing the control-plane taint. Running app workloads beside the API server and etcd risks the whole cluster when an app misbehaves.

Deleting the Pending Pod and hoping. The controller recreates an identical Pod with the same requests and constraints, and it lands in the same state. The scheduler already retries on its own.

Assuming a missing Pod is Pending. If kubectl get pods shows fewer Pods than replicas and none are Pending, the problem is admission (quota, LimitRange, Pod Security), visible on the ReplicaSet.

Verify the behavior

After the fix, every replica has a node and moves past Pending:

kubectl rollout status deploy/report
kubectl get pods -l app=report -o wide

Expect deployment "report" successfully rolled out and a real node name in the NODE column. kubectl get events --field-selector reason=FailedScheduling should show no new entries for the Pods, and kubectl describe node should show allocated CPU requests below 100%. To prove headroom, scale up by one (kubectl scale deploy/report --replicas=4) and confirm the extra Pod schedules too.

Interview exercise

A Pod has been Pending for ten minutes with 0/6 nodes are available: 6 Insufficient memory, yet monitoring says every node uses under 40% of its memory. Explain the contradiction and how you would resolve it.

Answer and reasoning

The scheduler doesn’t look at usage; it sums the memory requests of Pods already on each node and checks whether the new Pod’s request fits into allocatable memory. Nodes can be fully “booked” by requests while actual consumption is low, which happens when teams set generous requests. I would compare the Pending Pod’s request with “Allocated resources” on a node, then check whether the request is realistic. The fixes, in order: right-size this Pod’s request from observed usage, right-size the over-requesting workloads that are reserving memory they never use, and only then add capacity (or let the cluster autoscaler add a node). I would not delete the request, since memory requests also determine eviction order and protect the Pod under pressure.

Continue learning

• Kubernetes interview questions and Kubernetes MCQs
• Resource requests and limits
• Node placement: selectors, affinity and taints
• Persistent storage
• Kubernetes CrashLoopBackOff
• Official: Taints and tolerations, Assigning Pods to nodes, Debug Pods and Resource quotas