Architecture

Secret Delivery Path

How a secret gets from a managed vault into a running container without ever being committed, and why a missing one fails completely silently.

scroll to render

What you're looking at

The secret lives in a managed store. An operator inside the cluster holds a workload identity that is authorised to read specific versions, reconciles on a schedule, and projects the result into a native Kubernetes Secret. The workload consumes that Secret by reference. Nothing sensitive is ever in Git — what is in Git is the declaration of which keys should exist and where they come from.

Why workload identity is the point

The operator does not hold a static credential. It presents a cluster-issued identity that the cloud IAM layer maps to a service account with a narrow grant — read this secret, nothing else. There is no key file to rotate, leak or find in a repository years later, and revoking access is a policy change rather than a redeployment.

What goes wrong here

Missing secrets fail silently, which is the defining property of this path. A workload consuming a Secret by reference does not crash when a key is absent — the environment variable simply does not exist, and the failure surfaces much later as an unauthenticated call to some downstream service. There is no crash loop to alert on and no error in the operator, because from its point of view it synced exactly what it was told to sync.

The GitOps trap on top of it

If the sync engine is configured to ignore certain fields, and the ignore path walks into the elements of a list rather than naming a scalar, the engine rebuilds that whole list from the live object when it constructs its patch. New keys then never ship. Every sync reports success, the controller log literally says the resource was configured, and the app stays out of sync forever. Non-converging by construction — and because the consuming workload injects at pod start, even a correct fix needs a restart before anything changes.

Inspect it yourself

Did the key actually arrive?
```
kubectl -n NS get secret NAME -o jsonpath='{.data}' | jq 'keys | length'
```
Compare against what the chart renders. A count that is short by one or two, with no errors anywhere, is the signature of this whole class of failure.
Is the operator healthy but wrong?
```
kubectl -n NS get externalsecret NAME -o jsonpath='{.status.conditions[*].reason}'
```
Ready and synced against the wrong key list looks identical to Ready and synced against the right one. The operator cannot tell you what you forgot to ask for.
Catch a truncated patch
```
kubectl -n NS get externalsecret NAME \
  -o jsonpath='{.metadata.annotations.kubectl\.kubernetes\.io/last-applied-configuration}' | jq '.spec.data | length'
```
If the last-applied record already shows the short list, the patch that was sent was truncated. That distinguishes it from a correct apply being reverted afterwards.
Remember the restart
```
kubectl -n NS rollout restart deploy/SVC
```
Environment injected by reference is read once at pod start. Fixing the secret leaves running pods on the old values indefinitely.
Prove the identity mapping
```
kubectl -n NS get sa SA -o jsonpath='{.metadata.annotations}' | jq
```
Most access failures here are a missing binding between the cluster service account and the cloud identity, not a missing grant on the secret itself.

Read the source

Components

SECRET STORE— versioned
IAM GRANT— read this, nothing else
WORKLOAD IDENTITY— no static key
SECRET OPERATOR— reconciles
GIT DECLARATION— keys, not values
SYNC ENGINE— ignore-paths trap
K8S SECRET— projected
POD— envFrom, at start
DOWNSTREAM— 401 if key absent

Flows

iam→wi
wi⇢opassumes
vault→opread version
git→sync
sync→opdesired keys
op→secretproject
secret→podenvFrom
pod⇢downsilent if missing