release engineering · SEV-2

a fresh deploy crashloops on a startup probe that the new code definitely implements

scroll to render

How to confirm it

Compare the running digest against the registry
```
kubectl -n NS get pod POD -o jsonpath='{.status.containerStatuses[0].imageID}'
```
The image ID is the digest actually running. Against the digest the build just pushed, this is a one-line verdict.
Confirm the policy, not the intention
```
kubectl -n NS get deploy SVC -o jsonpath='{.spec.template.spec.containers[0].imagePullPolicy}'
```
An unset policy is inferred from the tag — a floating dev tag defaults to always, any other tag defaults to if-not-present. That inference is the whole bug.
Check the probe the way the kubelet does
```
kubectl -n NS exec POD -- curl -sS -o /dev/null -w '%{http_code}' localhost:PORT/health/startup
```
A 404 here means the code is old, not that the probe is misconfigured. Do not start tuning probe thresholds.
Expect the sync to fight you
```
kubectl -n argocd get app APP -o jsonpath='{.status.sync.status}'
```
Auto-sync fires on any push to the shared GitOps branch even with self-heal off, so a patch applied right after a pipeline run is reverted by that one sync.

Read the source

Cause

The release published a mutable tag — same tag, new digest, every build — and the chart pulled with if-not-present. The node already had that tag cached, so it served the old image and the new probe endpoint returned 404. Lower environments never saw it because they used a tag the platform special-cases into always-pull.

Fix

Set the pull policy to always for any mutable tag, and commit it to the values file the release reads. A live patch works for one rollout and is then reverted by the next GitOps sync — the fix has to land in the branch the build renders from. The better long-term answer is immutable, digest-pinned tags.

Next incident

intermittent unknown-host errors on an external name that resolves perfectly when you test it by hand ↗