ALL NOTES

gitops · SEV-2

app is permanently OutOfSync on one resource while every sync reports success

scroll to render

How to confirm it

  • Find which resource is stuck

    kubectl -n argocd get app APP -o json | jq '.status.resources[] | select(.status!="Synced")'

    One resource OutOfSync forever, with a successful operation state, is the signature. A whole app failing is a different problem.

  • Catch the lie in the controller log

    kubectl -n argocd logs argocd-application-controller-0 | grep RESOURCE

    The log says the resource was configured, and it was — with a truncated payload. Success in the log is not success in the object.

  • Prove the write lands but is short

    kubectl get RESOURCE NAME -o jsonpath='{.metadata.annotations.kubectl\.kubernetes\.io/last-applied-configuration}' | jq '.spec.data | length'

    The last-applied record already shows the old list. That distinguishes a truncated apply from a correct apply being reverted afterwards.

  • Render the chart the way the controller does

    helm template REL ./helm -n NS -f helm/values.yaml -f helm/values/ENV.yaml

    Diff desired against live for that one field. This is the step that turns "GitOps is broken" into "the patch is truncated".

  • Remember env vars do not hot reload

    kubectl -n NS rollout restart deploy/SVC

    A secret consumed by reference is injected at pod start. Fixing the sync leaves running pods on the old values, and a missing key is silent — no crash, the variable simply does not exist.

Read the source

Cause

The app ignored three fields using a path expression that walked into the elements of a list. With respect-ignore-differences enabled, the sync payload takes ignored paths from the live object — so the whole list got rebuilt from live and the patch shipped the old entries every time. Non-converging by construction: the apply succeeds, nothing moves.

Fix

Delete the element-traversing path expressions. Field-level pointers that do not reach into a list are safe; walking into list elements is not. The comment justifying the rule was stale — the chart had started emitting those fields explicitly, so there was nothing left to ignore.