← ALL ARCHITECTURE

Architecture

Edge to Pod

Every hop a request makes between the public internet and a container, and the four independent layers that can each make a healthy pod look down.

scroll to render
01

What you're looking at

A request enters at the CDN, is origin-locked to it by an edge policy, and reaches a global load balancer. The balancer does not talk to pods — it talks to an endpoint group, which the cluster controller keeps in sync with the pods behind a Service. That indirection is the whole point and also the whole problem: the endpoint group is a cloud resource with its own quota, its own controller and its own failure modes, and none of them show up in pod status.

02

Why a healthy pod can still be down

The prober on the left is a separate path from the kubelet. The kubelet probes over the node-local interface; the load balancer probes from reserved public ranges that must be allowed through the VPC firewall on that specific port. A pod reads Ready because the kubelet is happy, while the balancer reports zero healthy backends because its own probes are being dropped. Both statements are true at once, and only one of them is the one users experience.

03

What goes wrong here

In rough order of how long each took to find. Endpoint-group quota is regional and shared across every cluster in the project, so onboarding one more service can exhaust it and the controller then wedges in a state that never self-heals. The health-check firewall allows a hand-curated port list that drifts. A strict mesh policy rejects the plaintext prober. And an origin-lock policy returns a deny to anyone testing the origin host directly, which looks like an outage and is actually the control working.

04

Diagnose it top-down, because each layer masks the next

Fix the quota and you find the firewall. Fix the firewall and you find the mesh policy. Every one of these presents as the same symptom — a generic 502 or 503 with healthy pods — so the only way through is to walk the path in order and confirm each hop rather than guessing which layer it is.

Inspect it yourself

  • Ask the balancer which layer failed

    gcloud logging read 'resource.type="http_load_balancer"' --limit=20 \
  --format='value(jsonPayload.statusDetails)'

    failed_to_pick_backend means the route matched and there were zero healthy endpoints — it never reached your pod. response_sent_by_backend means the pod answered. Comparing a broken route with a working one on the same host isolates it in a single query.

  • Is anything actually attached?

    gcloud compute backend-services get-health BACKEND --global --format=json | jq '.[].status.healthStatus | length'

    Zero here with Running pods means the endpoint group is the problem, not the workload. Do not read the value-formatted output — it collapses repeated fields and makes three zones look like one.

  • Read the pod's own access log

    kubectl -n NS logs POD --tail=200 | grep -oE '^[0-9.]+' | sort -u

    Only the node-local kubelet address and none of the documented prober ranges means the health checker is firewalled out. This is the fastest decisive test in the whole chain.

  • Find the real error on the endpoint group

    kubectl -n NS get svcneg NAME -o yaml | grep -A3 'reason:'

    Quota errors surface here as a sync failure and nowhere else. If the last sync time predates the quota grant, the controller is wedged and needs the annotation toggle, not more waiting.

Read the source

Components

  • CLIENT
  • CDN— origin-locked
  • GLOBAL LB— one anycast VIP
  • EDGE POLICY— deny by default
  • ENDPOINT GRP— regional quota
  • HEALTH PROBER— reserved ranges
  • VPC FIREWALL— per-port allow
  • SERVICE— endpoint slice
  • PODS— Ready 1/1
  • KUBELET— node-local probe

Flows

  • user→cdn
  • cdn→lborigin
  • armor⇢lballow/deny
  • lb→negbackend
  • neg→svcsync
  • svc→pods
  • prober→fw
  • fw⇢podsdropped
  • kubelet→podsprobe OK

Other systems