progressive delivery · SEV-1

canary promotion times out waiting for a pause that the rollout had already reached once

scroll to render

How to confirm it

Diff the two revisions' pod templates
```
kubectl -n NS get rs -l app=SVC -o json | jq -r '.items[] | .metadata.name + " " + (.spec.template.metadata.annotations|tostring)'
```
When the only difference between two ReplicaSets is the config checksum, the re-render is the cause and nothing about the application changed.
Trust the rollout, not the app health
```
kubectl -n NS get rollout SVC -o jsonpath='{.status.phase}{"\t"}{.status.currentStepIndex}'
```
A paused canary reports Degraded to the GitOps controller by design, so the app can read Degraded while the rollout is genuinely Healthy. The deploy often succeeds underneath a red pipeline.
Watch the step index reset in real time
```
kubectl -n NS get rollout SVC -w -o custom-columns=STEP:.status.currentStepIndex,HASH:.status.currentPodHash
```
The hash changing while the step index drops back to 0 is the race, captured. Manually jumping the step index past an unprocessed pause gets reset by the controller anyway.
Stop re-running the pipeline
```
kubectl -n NS scale rs SVC-ORPHANHASH --replicas=0
```
Each re-run re-triggers the race and churns a production rollout. Scale the orphaned ReplicaSet to zero, let the abort converge, then do one clean full run.

Read the source

Cause

Two renders, half a minute apart. The GitOps commit carried both inline values and a new target revision; the inline values applied immediately while the values file was read from a repo checkout that lagged behind. The second render produced a different config checksum, that checksum lives in the pod template, so the controller saw a new revision and restarted the canary at step zero.

Fix

Make the wait tolerate a re-render by resetting its timer on a pod-template-hash change, and fix the root cause by ordering the release so the values revision is resolved before the image tag is written. Only releases that also change config are affected — an image-only release renders identically twice, which is exactly why it looks intermittent.

Next incident

app is permanently OutOfSync on one resource while every sync reports success ↗