The scheduler’s job sounds simple: pick a node with enough room. In practice, teams need more than room. GPU jobs must land on GPU nodes, and nothing else should waste those nodes. Replicas of a service should spread across zones. A database shouldn’t share a node with a CPU-hungry batch job. Kubernetes gives you several tools for this, and the interview question is usually less “what’s the syntax” than “which tool expresses this intent, and what goes wrong if you pick the other one”.
Target version: Kubernetes 1.30 or later. Every manifest was parsed to confirm valid YAML; the scheduler event text is a representative example.
Two directions: attraction and repulsion
The first distinction to get straight is who states the rule.
- nodeSelector and node affinity live on the Pod. They say “I want to run on nodes like this”. They attract.
- Taints live on the node. They say “Pods can’t run here unless they tolerate me”. They repel. Tolerations live on the Pod and say “I accept that taint”.
A toleration is permission, not a request. A Pod that tolerates gpu=true:NoSchedule can still be placed on any untainted node. That’s the most common misunderstanding in this area, and it leads to a familiar incident: someone taints the GPU pool, adds a toleration to the training job, and is surprised when the job lands on a regular node and fails with no CUDA-capable device.
A dedicated GPU pool, done properly
Say you have three nodes with an NVIDIA GPU, labelled accelerator=nvidia-l4. You want only training Pods there, and you want training Pods only there.
Repel everything else with a taint on each GPU node:
kubectl taint nodes gpu-node-1 gpu-node-2 gpu-node-3 dedicated=gpu:NoScheduleThen give the training Pod both halves: a toleration so it’s allowed on, and affinity so it’s required to go there.
apiVersion: v1
kind: Pod
metadata:
name: train-resnet
namespace: ml
spec:
tolerations:
- key: dedicated
operator: Equal
value: gpu
effect: NoSchedule
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: accelerator
operator: In
values: ["nvidia-l4", "nvidia-a100"]
containers:
- name: trainer
image: registry.example.com/ml/trainer:2.3.1
resources:
limits:
nvidia.com/gpu: 1The nvidia.com/gpu limit also matters: it’s an extended resource the device plugin advertises, so the scheduler won’t put two single-GPU Pods on a one-GPU node. Many managed platforms add the taint and even the toleration for you when a Pod requests a GPU, but the pattern underneath is the same.
For a simple equality match, nodeSelector: {accelerator: nvidia-l4} would do the same job as the affinity block. Affinity earns its verbosity when you need In with several values, NotIn, Exists, Gt/Lt, or several alternative terms. Multiple entries under nodeSelectorTerms are ORed; multiple matchExpressions inside one term are ANDed.
Required versus preferred
Affinity comes in two strengths.
requiredDuringSchedulingIgnoredDuringExecution is a filter. Nodes that don’t match are removed from consideration. If none match, the Pod stays Pending with an event like:
0/12 nodes are available: 3 node(s) had untolerated taint {dedicated: gpu},
9 node(s) didn't match Pod's node affinity/selector.
preemption: 0/12 nodes are available: 12 Preemption is not helpful for scheduling.Read that message carefully, because it tells you which rule eliminated which nodes. Here a Pod has the affinity but forgot the toleration, so the three GPU nodes reject it on the taint and the other nine fail affinity. The Pending Pod guide covers the other reasons this event appears.
preferredDuringSchedulingIgnoredDuringExecution is a score. Each preference has a weight from 1 to 100, and nodes that match get that weight added to their score. The Pod schedules somewhere either way. Use it for “nice to have”: prefer spot instances for a batch job, prefer the zone where the cache lives.
The IgnoredDuringExecution suffix is the other half of the name and is often asked about. If you relabel a node after a Pod is running so the Pod no longer matches, nothing happens to the Pod. Affinity is checked at scheduling time only. A RequiredDuringExecution variant has been discussed for years but doesn’t exist in the API.
Taint effects and eviction
Taints have three effects:
NoSchedule: new Pods without a matching toleration won’t be scheduled. Running Pods stay.PreferNoSchedule: the scheduler tries to avoid the node but will use it if nothing else fits.NoExecute: new Pods are blocked and running Pods without a toleration are evicted.
NoExecute is how Kubernetes itself reacts to node failures. When a node stops reporting, the node controller adds node.kubernetes.io/unreachable:NoExecute (or not-ready). Every Pod gets default tolerations for those with tolerationSeconds: 300, which is why a Pod on a dead node takes about five minutes to be replaced. A latency-sensitive Deployment can shorten that:
tolerations:
- key: node.kubernetes.io/unreachable
operator: Exists
effect: NoExecute
tolerationSeconds: 30Shortening it trades faster failover for more churn during brief network blips, so 30 seconds isn’t automatically better.
kubectl drain uses a different mechanism: it cordons the node (sets spec.unschedulable, which the control plane reflects as a node.kubernetes.io/unschedulable:NoSchedule taint) and then evicts Pods through the Eviction API, which respects PodDisruptionBudgets. A NoExecute taint bypasses PDBs entirely.
Placing Pods relative to other Pods
Node affinity is about node labels. Pod affinity and anti-affinity are about which other Pods are already on a node or in a zone. The topologyKey sets the scope: kubernetes.io/hostname means “same node”, topology.kubernetes.io/zone means “same zone”.
Keeping replicas of checkout off the same node:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
topologyKey: kubernetes.io/hostname
labelSelector:
matchLabels:
app: checkoutWhy preferred and not required? With required anti-affinity on hostname, you can never run more replicas than you have nodes, and a rollout with surge Pods can stall because the new Pod has nowhere to go. Required anti-affinity makes sense for small, fixed sets like a three-member etcd or ZooKeeper ensemble. For general spreading, topologySpreadConstraints is usually the better tool: it expresses “keep the counts per zone within 1 of each other” directly, rather than “never two together”.
Inter-pod affinity is also expensive for the scheduler in large clusters because it has to examine existing Pods across topology domains. That’s one reason the docs recommend it sparingly in clusters with thousands of nodes.
Interview practice
“You have a node pool of high-memory nodes for an in-memory analytics service. Other teams’ Pods keep landing there and the analytics Pods go Pending. Separately, an analytics Pod once ran on a regular node and was OOMKilled. Design the placement rules.”
Answer
Two problems, two tools. Other teams landing on the pool means nothing repels them, so I’d taint the high-memory nodes, say workload=analytics:NoSchedule. Existing foreign Pods would stay (NoSchedule doesn’t evict), so I’d either drain the nodes one at a time to move them politely, respecting their PDBs, or use NoExecute if they can be evicted immediately.
The analytics Pod running on a regular node means it wasn’t required to go to the pool, so I’d label the nodes (node-pool=highmem) and add required node affinity on that label, plus the toleration for the taint. Toleration without affinity would only let it in; affinity without the toleration would leave it Pending forever. I’d also check the memory request: if the scheduler thought a regular node had room, the request was probably far below real usage, which is the root cause of the OOMKill (exit code 137 guide). Setting an honest request makes the placement and the memory guarantee agree.
Further reading
- Kubernetes interview questions
- Kubernetes MCQs
- Requests, limits and scheduling
- Pod topology spread
- Official docs: Assigning Pods to Nodes