The liveness probe that makes your uptime worse

Most Kubernetes misconfigurations are neutral until something else goes wrong. A bad liveness probe is different. It takes a service that was struggling and kills it.

Three probes, three different jobs

Readiness answers: should this pod receive traffic right now? Failing it removes the pod from Service endpoints. Nothing is killed. When it passes again, traffic returns.

Liveness answers: is this container wedged beyond recovery? Failing it restarts the container. That’s the whole mechanism: a reboot, applied automatically, based on your definition of wedged.

Startup answers: has this container finished booting? While it runs, liveness and readiness are suspended. It exists because slow-starting applications were being killed by liveness probes before they ever came up.

The names suggest a spectrum. They’re three different tools.

	Readiness	Liveness	Startup
Failing it removes the pod from Service endpoints	●Yes	–No	–No
Failing it restarts the container	–No	●Yes	●Yes
Suspends the other probes while it runs	–No	–No	●Yes
Safe to check a database or downstream	●Yes	–No	–No
Recovers on its own when the check passes again	●Yes	–No	–No
Can turn a slow dependency into an outage	–No	●Yes	–No

The row that causes incidents is the fourth. A dependency check belongs in readiness, where failure sheds traffic — putting it in liveness converts a degraded system into a restart storm.

The failure mode

This is the configuration we find most often, and it’s close to a worst case:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 3

…where /health checks the database connection.

Now the database gets slow. Every pod’s /health starts timing out. Every pod fails liveness three times. Kubernetes restarts every pod simultaneously.

You’ve taken a degraded-but-serving system and converted it into a total outage, plus a thundering herd of reconnecting clients hitting the database that was already struggling. And the restart can’t help, because the container was never the problem.

The rule

A liveness probe must only test conditions that a restart can fix.

Deadlock, an exhausted internal thread pool, a corrupted in-memory state machine: a restart fixes those. A slow dependency, a saturated downstream, an expired credential: a restart fixes none of them, and attempting one makes things worse.

The practical form of the rule is that liveness probes must not have dependencies. No database check, no downstream HTTP call, no cache ping. /livez returns 200 if the process can serve a request, and that’s all.

Dependency health belongs in the readiness probe, where failure removes the pod from load balancing without killing it. That’s graceful degradation. The liveness version is an outage.

Timeouts nobody sets

timeoutSeconds defaults to 1 second. That’s aggressive for anything doing real work under load, and it’s the value most manifests inherit silently.

Under load, a healthy process may take longer than a second to answer an HTTP probe simply because it’s busy, which is exactly the moment you least want it restarted. If you keep a liveness probe, give it a timeout that reflects a genuinely wedged process (several seconds) and a failureThreshold that tolerates a transient blip.

Startup probes exist to stop you tuning initialDelaySeconds

The old pattern was pushing initialDelaySeconds high enough that slow boots survived. The cost was that a container which wedged after starting went undetected for that whole window.

Startup probes solve it properly:

startupProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 5
  failureThreshold: 30      # up to 150s to start
livenessProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3

The application now gets 150 seconds to boot, and once it has booted, a wedge is caught within 30. Neither number compromises the other.

A shape that works

• /livez — no dependencies. Returns 200 if the process can serve. Used by startup and liveness.
• /readyz — checks dependencies. Used by readiness only.
• Startup probe with a generous failureThreshold for boot time.
• Liveness probe with a realistic timeout, pointed at /livez.
• Readiness probe pointed at /readyz.

Ask what a restart fixes

Before adding a liveness probe at all, ask: what does this application do that a restart fixes?

If you can’t answer concretely, you may not need one. A pod with only a readiness probe degrades gracefully under dependency failure and never restart-storms. For a stateless HTTP service that crashes cleanly on unrecoverable errors, that’s often the better configuration, because the process manager already handles the case liveness was meant to catch.

Kubernetes will restart a container that exits. Liveness is for the case where it should have exited and didn’t.

Probe behaviour also shows up in CKAD’s observability domain, though it matters far more in production than in any exam.