build infrastructure · SEV-3
builds start failing with 429 Too Many Requests from a public package registry, with no change to the code
How to confirm it
Rule out port exhaustion before blaming the network
gcloud logging read 'jsonPayload.allocation_status="DROPPED"' --limit=10Zero NAT drops means this is an application-layer 429, not a connection-tracking limit. Two very different fixes.
Count your egress addresses
gcloud compute routers nats describe NAT --router ROUTER --region REGION --format='value(natIps)'One manually assigned address for an entire CI fleet is the finding. Compare against the workload gateway, which usually has several.
Prove the cache never persists
kubectl -n NS get scaledjob NAME -o jsonpath='{.spec.jobTargetRef.template.spec.volumes}'An ephemeral volume for the workspace plus an unset cache-home variable means the cache dies with the pod. The log tell is a download of the full dependency set on every single run.
Check whether a proxy ever existed
gcloud artifacts repositories list --format='table(name,format,mode)'Remote-mode repositories are the mirrors. If the only entries are standard mode, builds have been resolving public dependencies directly all along — the regression is volume, not a deleted proxy.
Read the source
Cause
Every CI agent egressed through a single NAT address, and no dependency cache survived a job — the workspace was ephemeral and the tool's cache directory was never pointed anywhere persistent. So every build cold-downloaded the full dependency graph from one IP. A single address plus a cold cache is a permanent ceiling; a busier day simply crossed it.
Fix
Put a pull-through mirror in the private registry so agents only ever talk to it, and let it fetch upstream server-side — then per-IP throttling is structurally impossible. Add addresses to the NAT gateway for immediate relief and persist the dependency cache so builds stop being cold. Retrying inside the window just re-throttles the same address.