Architecture

Progressive Delivery

A release moving through weighted canary steps, with the analysis that promotes it and the two ways it gets stuck instead.

scroll to render

What you're looking at

A merge writes desired state to a GitOps repository. The sync engine renders that into a rollout object, which does not replace pods all at once — it stands up a canary alongside the stable set and shifts a percentage of traffic to it. At each weight it pauses, an analysis job queries the metrics backend, and a healthy verdict advances to the next step. A bad one aborts and traffic returns to stable.

Why the pause is the whole design

A deploy that cannot pause cannot be judged. The pause is where a human or an automated analysis gets to look at error rate and latency on real traffic at low blast radius, and it is the only point at which rolling back is cheap. Everything else in progressive delivery exists to make that pause possible and to make aborting from it safe.

What goes wrong here

The subtle failure is a second render. The config checksum lives in the pod template, so if the sync engine renders twice — inline values applied immediately, the values file read from a repo checkout that lags behind — the second render produces a different template hash, the controller treats it as a new revision, and the canary restarts at step zero. The promotion job is still waiting for a pause it already saw, and times out. It only happens on releases that also change config, which is why it looks intermittent.

The trap underneath

A paused canary legitimately reports as degraded to the sync engine, so any wait that treats degraded as failure will fail a perfectly healthy release. Worse, the deploy frequently succeeds underneath a red pipeline — the rollout settles at full weight while the build is still marked failed. Trust the rollout's own phase, not the app health, and do not keep re-running: each re-run re-triggers the race and churns production again.

Inspect it yourself

Watch the step index and the hash together
```
kubectl -n NS get rollout SVC -w \
  -o custom-columns=PHASE:.status.phase,STEP:.status.currentStepIndex,HASH:.status.currentPodHash
```
The hash changing while the step index drops back to zero is the re-render race, captured live. Nothing else produces that pattern.
Diff the two revisions' pod templates
```
kubectl -n NS get rs -l app=SVC -o json \
  | jq -r '.items[] | .metadata.name + " " + (.spec.template.metadata.annotations|tostring)'
```
If the only difference between two ReplicaSets is the config checksum, nothing about the application changed and the render is the cause.
Is it actually stuck, or just paused?
```
kubectl -n NS get rollout SVC -o jsonpath='{.status.phase}{"\t"}{.status.message}'
```
Paused is the design working. Progressing at step zero long after the release started is the race. The sync engine's own health field will say degraded in both cases, so do not read that one.
Unwedge without another full run
```
kubectl -n NS scale rs SVC-ORPHANHASH --replicas=0
```
Scale the orphaned canary ReplicaSet to zero and let the abort converge, then do a single clean run. Repeated re-runs re-trigger the same race.

Read the source

Components

MERGE— to main
GITOPS STATE— tag + values
SYNC ENGINE— renders chart
ROLLOUT— weighted steps
CANARY RS— 10% → 50% → 100%
STABLE RS— serving the rest
ANALYSIS— error rate · latency
METRICS— queried per step
USERS

Flows

merge→git
git→syncrevision
sync→rolloutrender
rollout→canaryshift weight
rollout→stable
metrics→analysis
analysis⇢rolloutpromote / abort
canary→users
stable→users