Why OpenShift upgrades stall, and how to clear them

An OpenShift upgrade that has not moved in forty minutes is usually fine. One that has not moved in four hours is usually blocked on something specific and findable. Telling the two apart is most of the skill.

Start with what the cluster says about itself

Three commands, in this order, before touching anything.

# Where the upgrade is, and whether it is progressing
oc get clusterversion

# Which operator is unhappy — this is the answer most of the time
oc get clusteroperators | grep -v 'True.*False.*False'

# Whether nodes are the problem
oc get machineconfigpool

The second command filters to operators that are not cleanly Available, not Progressing and not Degraded. Whatever it prints is where to look next. An empty result means the operators are fine and the problem is on the nodes.

clusterversion also carries a human-readable reason:

oc get clusterversion -o jsonpath='{.items[0].status.conditions[?(@.type=="Progressing")].message}{"\n"}'

Blocker one: a pod that cannot be evicted

This is the most common cause of a worker upgrade that sits at “1 of 6 nodes updated” indefinitely.

Applying a MachineConfig requires draining the node. Draining evicts pods. Eviction respects PodDisruptionBudgets, and a PDB that can never be satisfied blocks the drain forever rather than failing.

The classic shape: a Deployment with replicas: 1 and a PDB with minAvailable: 1. Evicting the only pod would take availability to zero, so the API refuses, and the drain retries until someone notices.

# Find PDBs with no headroom
oc get pdb -A -o custom-columns=\
NS:.metadata.namespace,NAME:.metadata.name,MIN:.spec.minAvailable,\
ALLOWED:.status.disruptionsAllowed | awk '$4=="0"'

Anything showing 0 allowed disruptions is a candidate. The fix is to scale the workload to two replicas, not to delete the PDB — deleting it makes the upgrade proceed and the outage happen. PodDisruptionBudgets and drain semantics covers the details.

Single-replica stateful workloads are the real exception. For those, plan the disruption into the maintenance window rather than pretending it will not happen.

Blocker two: a degraded MachineConfigPool

A MachineConfigPool goes Degraded when a node fails to apply its configuration, and it stops the whole pool.

oc get mcp
oc describe mcp worker | sed -n '/Conditions/,/Events/p'

Then go to the node and read the machine-config-daemon’s account:

oc logs -n openshift-machine-config-operator \
  -l k8s-app=machine-config-daemon --tail=50 \
  --field-selector spec.nodeName=<node>

The usual causes are a hand-edited file on the node that the daemon refuses to overwrite, a MachineConfig that produces invalid Ignition, or a node with no disk space. All three are node-local and none of them clear on their own.

Blocker three: an operator waiting on something you own

Cluster operators that go Degraded during an upgrade frequently do so because a resource you created is invalid — a bad IngressController replica count, a Console customisation referencing a deleted secret, an image registry with no storage backend.

The registry one catches new clusters often: if the image registry is set to Removed or has no PVC on a bare-metal install, its operator never reaches Available and the upgrade waits on it.

oc describe clusteroperator <name> | sed -n '/Conditions/,$p'

Read the message rather than restarting the operator. Operators are level-triggered; restarting one that is correctly reporting a broken input just delays the same message.

Blocker four: the channel will not offer the version

Not a stall so much as an upgrade that never starts.

oc get clusterversion -o jsonpath='{.items[0].spec.channel}{"\n"}'
oc adm upgrade

stable-4.x lags fast-4.x deliberately. You cannot skip a minor version, so 4.14 to 4.16 is two upgrades and the intermediate one is not optional. If oc adm upgrade shows no available updates, the channel is either wrong for the target or the version has not been promoted into it yet.

There is also a conditional-update mechanism: OpenShift may withhold a version it believes is risky for your cluster’s specific configuration and tell you why. That message is worth reading rather than forcing past with --allow-not-recommended.

Judging slow versus stuck

Rough numbers from clusters we run, for a minor-version upgrade:

Phase	Reasonable
Operators rolling	30–60 minutes
Control plane, 3 nodes	45–90 minutes
Workers	10–20 minutes per node, serialised

A 30-node cluster with maxUnavailable: 1 is a multi-hour job. Raise maxUnavailable on the worker pool if your workloads tolerate it, and expect no improvement on the control plane, which upgrades one node at a time regardless.

Before declaring it stuck, check that something is still changing:

oc get nodes -w
oc get co -w

If node ages and operator versions are still moving, it is working. If nothing has changed in twenty minutes and no pod is pending, start at the top of this post.

What to do before the next one

Most stalled upgrades are avoidable, and the same three checks catch nearly all of them:

• Run the PDB query above and fix the zero-disruption entries while nothing is upgrading
• Confirm every cluster operator is Available on an ordinary Tuesday, not just on upgrade day
• Upgrade a non-production cluster on the same version path first, with the same operators installed

Backup and restore sits alongside this work, and it is the EX380 objective people most often underestimate. An upgrade you cannot roll back is a different risk from one you can.