load balancing · SEV-2

load balancer returns a 502 error page while every pod reads Running and Ready

scroll to render

How to confirm it

The health check is the tell, not the pods
```
gcloud compute backend-services get-health BACKEND --global
```
"Listed 0 items" means the backend service exists but has no endpoints attached. Pods being Ready is irrelevant — the endpoint group is what the load balancer reads.
Find the real error in the CR, not the events
```
kubectl -n NS get svcneg NAME -o yaml | grep -A3 'reason:'
```
The literal QUOTA_EXCEEDED string lives here as NegSyncFailed. Nothing upstream surfaces it.
Prove the controller is wedged, not still failing
```
kubectl -n NS get svcneg NAME -o jsonpath='{.status.lastSyncTime}'
```
Older than the quota grant means the work item was dropped permanently. The condition message is frozen at its transition time — it is a cached string, not a live reading, so it will still quote the old limit.
Do not count zero-endpoint groups as reclaimable
```
gcloud compute network-endpoint-groups list --format='table(name,size,zone)'
```
Most groups legitimately report size 0 — pods rarely spread across every zone. Only a group with zero endpoints in all zones is dead. Capacity is the lever here, not cleanup.
Count backends honestly
```
gcloud compute backend-services describe BACKEND --global --format=json | jq '.backends | length'
```
A value-formatted query collapses the repeated field and makes three attached zones look like one.

Read the source

Cause

Network endpoint groups are a per-project, per-region quota shared by every cluster in the project, and the cluster creates one group per service per port per zone — so cost grows as services x 3. Onboarding one more service crossed the ceiling. The backend service existed with zero backends attached, so the request never left the provider edge.

Fix

Raise the regional quota, then unwedge the controller. Restoring quota alone does not recover it: the per-service sync is in a terminal no-retry state and never self-heals. Removing and re-adding the endpoint-group annotation on the Service forces a fresh reconcile, and groups reappear in all zones within about thirty seconds.

Next incident

pod is Ready and answering locally, but the load balancer calls it unhealthy and serves 503 ↗