OpenShift Pod Stuck in Pending: How to Debug Taints, Tolerations, Affinity and Topology Spread

A Pod stays Pending when no node satisfies every required scheduling rule. Often no single rule is wrong. Several correct rules together can leave no eligible node.

This guide is for engineers who deploy workloads on OpenShift. It explains what each rule means, why it exists, and how to find which one is blocking your Pod. You should know Pods, Deployments and the oc command line. Every oc get and oc describe here also works with kubectl.

Key terms

• Scheduler. The component that picks a node for each Pod.
• Required rule. A rule the scheduler must satisfy, or the Pod stays Pending.
• Preferred rule. A rule the scheduler tries to satisfy but may ignore.
• Eligible node. A node that passes every required rule.
• Label. A key-value tag on a node or Pod that rules read.
• Taint. A mark on a node that keeps Pods away unless they carry a matching toleration.
• Toleration. A Pod setting that ignores a matching taint.
• Affinity. A Pod setting that names nodes or Pods to run near. Anti-affinity names Pods to run away from.
• Topology spread. A Pod setting that spreads replicas evenly across groups, such as zones.

A cloud analogy

The scheduler works like a cloud provisioning system placing virtual machines on hosts.

A firewall allow rule does not send traffic to a server. It only removes the block. A toleration works the same way. It does not send a Pod to a node. It only stops the taint from rejecting the Pod.

How the scheduler decides

For each new Pod, the scheduler:

1. Filters out every node that fails a required rule.
2. Scores the remaining nodes using preferred rules.
3. Binds the Pod to the highest-scoring node.

If a node fails even one required rule it is excluded, even if it passes all the others. If no nodes are left, the Pod stays Pending. Check with oc get pod <pod> -o wide. If the NODE column shows <none>, scheduling has failed.

Why does the scheduler not just pick the closest node? Required rules are guarantees. Ignoring them hides problems, such as putting every replica on one node. A Pending Pod is a visible failure, which is better than a hidden one.

Why each rule exists

Each rule addresses a production problem that more capacity alone cannot fix.

Every required rule protects a guarantee but also reduces the number of eligible nodes. Ask: "If this breaks, is it an outage or just a less ideal placement?" Use a required rule for outages and a preferred rule for the rest.

The rules in practice

Rules depend on labels, so check and add them first:

1oc get nodes -L topology.kubernetes.io/zone
2oc label node <node> workload=payments

Taints and tolerations. Reserve a node with a taint, and admit a workload with a toleration:

1oc adm taint nodes <node> dedicated=database:NoSchedule

1tolerations:
2  - key: dedicated
3    operator: Equal
4    value: database
5    effect: NoSchedule

Add a trailing dash to the taint to remove it. NoSchedule blocks new Pods, PreferNoSchedule only discourages them, and NoExecute also evicts running Pods. Use a taint to keep other Pods away, and node affinity to attract your own.

Affinity. Required affinity must match. Preferred affinity only affects scoring. This rule keeps checkout replicas on separate nodes:

1podAntiAffinity:
2  requiredDuringSchedulingIgnoredDuringExecution:
3    - labelSelector:
4        matchLabels:
5          app: checkout
6      topologyKey: kubernetes.io/hostname

With three replicas and only two eligible nodes, the third replica cannot be scheduled.

Topology spread.

1topologySpreadConstraints:
2  - maxSkew: 1
3    topologyKey: topology.kubernetes.io/zone
4    whenUnsatisfiable: DoNotSchedule
5    labelSelector:
6      matchLabels:
7        app: checkout

maxSkew is the largest allowed difference in Pod counts between the fullest and emptiest zone. DoNotSchedule makes the rule required. ScheduleAnyway makes it preferred. If a Pod cannot use a zone, for example because of an untolerated taint, that zone still counts as empty, so the rule still expects a Pod there. The worked example below shows how that goes wrong.

Why correct rules combine badly

Each rule looks sensible alone. The failure happens at the intersection:

If every node fails at least one rule, none is eligible. Before removing any rule, work out which nodes each rule excludes.

Read the event

The event groups rejected nodes by reason, so it can name several causes at once.

Preemption removes lower-priority Pods to free resources. It cannot solve problems with labels, taints or spread constraints.

OpenShift details and other causes

• Tainted roles. Control plane nodes carry a node-role.kubernetes.io/master taint, and many clusters taint infra nodes. These appear as untolerated taints by design.
• Hidden node selectors. The namespace annotation openshift.io/node-selector and the cluster defaultNodeSelector add selectors that never appear in your Deployment. Check with oc get namespace <ns> -o yaml and oc get scheduler cluster -o yaml.
• MachineSets. A MachineSet creates a group of identical nodes. New nodes inherit its labels and taints. If the template lacks your required label, new nodes cannot run the Pod.
• Other causes. Unbound volume claims (oc get pvc), nodes that are cordoned or NotReady (oc get nodes), and resource requests larger than any node. The scheduler uses requested resources, not actual usage, so oc adm top nodes can show idle nodes while the event says Insufficient cpu.

Worked example: the third replica that would not schedule

Six workers sit in three zones. All carry the taint dedicated=payments:NoSchedule and the label workload=payments. The two zone-c nodes also carry a forgotten maintenance=true:NoSchedule taint.

The checkout Deployment has three replicas, a workload: payments node selector, a toleration for dedicated, and the zone spread rule from the previous section. Nothing tolerates maintenance.

Two replicas run in zone-a and zone-b. The third is Pending:

10/6 nodes are available:
22 node(s) had untolerated taint {maintenance: true},
34 node(s) didn't match pod topology spread constraints.

Zone-a and zone-b each have one Pod. Zone-c has none. With maxSkew set to 1, the third Pod must go to zone-c, and the taint prevents it. No node meets both rules. Every rule in the Deployment is correct. The leftover taint creates the conflict.

Remove the taint:

1oc adm taint nodes worker-c1 worker-c2 maintenance=true:NoSchedule-

The scheduler retries on its own and the third replica lands in zone-c. If the taint was intentional, use whenUnsatisfiable: ScheduleAnyway instead.

Debug workflow

1. Confirm the Pod has no node.
2. Read the full event with oc describe pod, not only the first line.
3. Map each message to a cause with the table above.
4. Compare node labels, taints and zones with oc get nodes --show-labels and oc describe node.
5. Check requests, affinity, spread rules and hidden node selectors.
6. Work out which nodes are still eligible, then make the smallest change that adds one more.

Update the Deployment template rather than the Pod, since the controller recreates Pods from the template. In production, use required rules only for true guarantees, put labels and taints on the MachineSet template, and remove temporary taints after maintenance.

Conclusion

A scheduling failure is rarely just a full cluster. Resource requests, taints, affinity and spread rules combine to leave no eligible node. Each rule buys isolation or availability and costs flexibility. Keep only the guarantees you need.

Do not guess from the first line of the error. Check which nodes were available, which rule rejected each one, and what the eligible set was. Then change the label, taint, toleration, rule or capacity that actually unblocks it.

Sources

• OpenShift: Controlling pod placement onto nodes
• Kubernetes: Taints and tolerations
• Kubernetes: Pod topology spread constraints