PodDisruptionBudgets: the reason your node drain hangs

kubectl drain node-7 sits there. No error, no timeout, just a periodic line about evicting a pod that never goes away. Somewhere a PodDisruptionBudget is doing exactly what it was told to do.

A PDB constrains one specific thing

It bounds voluntary disruption: the eviction API, which is what kubectl drain, cluster autoscalers and node upgrades use.

It has no effect on involuntary disruption. A node losing power, a kernel panic, an OOM kill, a kubelet that stops reporting — none of these consult a PDB, because none of them go through the eviction API. Your minAvailable: 3 will not keep three pods running when the rack loses power.

That distinction disappoints people who read PDBs as an availability guarantee. They are a politeness contract with the things that drain nodes on purpose.

The trap that blocks upgrades indefinitely

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: api

Paired with a Deployment running replicas: 1, this says: at least one pod must always be available, and there is exactly one pod. Evicting it would leave zero. So the eviction API refuses, forever, and the drain retries until a human intervenes.

This is not a rare misconfiguration. It is the default shape of a first PDB written by someone protecting a service that has not been scaled yet, and it is the most common cause of an OpenShift or Kubernetes upgrade that stops halfway through a worker pool.

Find them before they find you:

kubectl get pdb -A -o custom-columns=\
NS:.metadata.namespace,NAME:.metadata.name,\
MIN:.spec.minAvailable,MAX:.spec.maxUnavailable,\
ALLOWED:.status.disruptionsAllowed,CURRENT:.status.currentHealthy

Any row with ALLOWED at 0 will block a drain right now. Some of those are correct and temporary — a pod mid-rollout, a deployment that is genuinely degraded. A row that has been at zero for days is a configuration problem.

minAvailable or maxUnavailable

Both exist and you may set only one.

maxUnavailable is usually the better choice, because it expresses the tolerance rather than the target and it keeps meaning the right thing when the replica count changes.

spec:
  maxUnavailable: 1     # one pod may go at a time, whatever the replica count

With minAvailable: 3 and a scale-down to 3 replicas, you have silently created the blocking configuration above. With maxUnavailable: 1, scaling changes nothing about whether maintenance can proceed.

Percentages work for both and round in the direction that favours availability, which is worth remembering when the numbers are small: minAvailable: 50% of 3 replicas is 2, not 1.5.

Sensible defaults by workload shape

Workload	Setting	Why
Stateless web, 3+ replicas	maxUnavailable: 1	Drains proceed one node at a time
Stateless web, large fleet	maxUnavailable: 10%	Drains parallelise without collapsing capacity
Quorum store (etcd, ZooKeeper)	maxUnavailable: 1	Never risk two members at once
Single-replica stateful	no PDB	A PDB cannot help, and will block maintenance
Batch / Job	no PDB	Restart is the normal path

The single-replica row is the one worth arguing about internally. A PDB there does not add availability — there is no second replica to keep — it only converts a brief restart into a stalled upgrade. If the workload genuinely cannot tolerate a restart, the fix is a second replica or a scheduled maintenance window, not a budget that refuses eviction.

Clearing a stuck drain safely

When a drain is blocked and you need it to proceed, in order of preference:

Scale up first. If the app can run two replicas, kubectl scale --replicas=2 makes the PDB satisfiable and the drain completes with no downtime. This is almost always available and almost always skipped.

Wait, if the block is transient. disruptionsAllowed: 0 during a rollout resolves itself once the new pods are Ready. Check currentHealthy against desiredHealthy before assuming it is stuck.

Delete the pod, accepting the disruption. kubectl delete pod bypasses the eviction API and the PDB entirely. This is a decision to take the outage, made deliberately, in a window — not a trick to make the warning go away.

Never delete the PDB to unblock a drain. It works, and it removes the protection everywhere it applied, and nobody puts it back.

Where PDBs and drains meet in practice

A drain evicts in no guaranteed order, so a PDB is the only thing stopping two members of a quorum leaving together. Combine it with topologySpreadConstraints so replicas are not all on the node being drained in the first place:

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: api

Spread constraints reduce how often the PDB has to say no. The PDB is the backstop for when it does.

If you run OpenShift, this is the mechanism behind upgrades that stall on a worker pool, and the PDB query above is the first thing to run when one does.