build infrastructure · SEV-3

build agents die mid-job with lost-contact and process-spawn errors that name the wrong component entirely

scroll to render

How to confirm it

Separate eviction from cancellation first
```
kubectl -n NS get events --sort-by=.lastTimestamp | grep -iE 'evict|oom'
```
Spawn failures and lost-agent messages are tear-down artifacts of a pod dying mid-job, not causes. The event tells you whether it was evicted or simply cancelled by a rapid re-push.
Check the limit, not the disk
```
kubectl -n NS exec POD -- df -h /tmp/work
```
Plenty of free space on the node alongside an eviction message means it was limit enforcement, not physical exhaustion. That reframes the whole investigation.
Catch the volume-type drift
```
kubectl -n NS get scaledjob NAME -o jsonpath='{.spec.jobTargetRef.template.spec.volumes}' | jq
```
Compare against committed source. An emptyDir counts toward the ephemeral limit; a host path does not. A sibling pool on host path being unaffected is how the drift was localised.
Spot a poisoned partial cache in the log
```
grep -E 'cache hit|Downloading' build.log | head -30
```
A reported cache hit followed by downloads of everything means the archive is a sliver. Changing one line in the requirements file changes the key — and if that first build dies, the partial is what every later build inherits.
Patch surgically on a drifted resource
```
kubectl -n NS patch scaledjob NAME --type=json -p '[{"op":"replace","path":"/spec/.../ephemeral-storage","value":"50Gi"}]'
```
A full apply of committed source would wipe the teammate hand-edits that are keeping the pool working. Single-field patches, and confirm with the owner.

Read the source

Cause

Two compounding faults. The workspace volume was an in-memory-class ephemeral volume, which counts against the container's ephemeral-storage limit, and the committed manifest said otherwise — that was live drift from a hand edit. Separately the dependency cache was keyed on a content hash, and the first build on a new key had been killed mid-download, so it saved a partial archive. The cache layer only writes on a miss and never overwrites a hit, so every later build hit that partial cache, re-downloaded the rest, bloated the workspace past the limit, got evicted before finishing, and never saved a good cache. A closed loop.

Fix

Raise the ephemeral limit enough for a full cold download to complete — one green run then saves a complete cache and the loop self-heals. Size it from the node's actual disk, not optimism: a limit larger than the disk guarantees node-level pressure that evicts every pod on the node, which is strictly worse than a per-pod limit.

Next incident

users cannot join voice sessions, while pod restart counts sit at zero and load metrics look calm ↗