networking · SEV-2

pod is Ready and answering locally, but the load balancer calls it unhealthy and serves 503

scroll to render

How to confirm it

Read the pod's own access log
```
kubectl -n NS logs POD --tail=200 | grep -oE '^[0-9.]+' | sort -u
```
If only the node-local kubelet address appears and none of the documented prober ranges do, the health checker is firewalled out. This single check settles it.
Ask the load balancer which layer failed
```
gcloud logging read 'resource.type="http_load_balancer"' --limit=20 --format='value(jsonPayload.statusDetails)'
```
failed_to_pick_backend = route matched, zero healthy endpoints, request never reached the pod. response_sent_by_backend = the pod answered. Compare a broken route against a working one on the same host to isolate instantly.
Query the host project, not yours
```
gcloud compute firewall-rules list --project HOST_PROJECT --filter='allowed.ports~PORT'
```
In a shared VPC a service project cannot list the firewall rules that govern it — a networks list returns zero and reads like nothing exists.
Diff the port list before and after
```
gcloud compute firewall-rules describe RULE --project HOST_PROJECT --format=json | jq '.allowed'
```
Prove nothing was dropped. A rule silently narrowed is a worse outage than the one being fixed.

Read the source

Cause

Kubelet probes and load-balancer probes are different network paths. The shared VPC allowed the health-check ranges only on a hand-curated list of ports, and the service listened on one that was not on the list. So the kubelet probe succeeded from inside the node and the external prober was dropped at the firewall.

Fix

Add the port to the health-check firewall rule in the host project. The decisive fix is at the network layer, not the workload — but note the allow flag replaces the port list rather than appending to it, so every existing port has to be restated and the before/after diffed.

Next incident

canary promotion times out waiting for a pause that the rollout had already reached once ↗