observability · SEV-3

APM init container stuck in ImagePullBackOff across several services

scroll to render

How to confirm it

Why won't the image pull?
```
kubectl describe pod POD | sed -n '/Events:/,$p'
```
ImagePullBackOff with 'manifest unknown' means the tag does not exist — a different problem from a credentials failure, which says 'unauthorized'.
Does the tag exist at all?
```
crane ls REGISTRY/IMAGE | head -20
```
Checks the registry directly, without Kubernetes in the way. If the tag is absent, no amount of cluster debugging helps.
Which languages support an init container?
```
kubectl get instrumentation -A -o yaml | grep -A2 'language\|image:'
```
Compiled languages generally instrument at build time. An init container for one is a configuration that can never succeed.

Read the source

Cause

The rollout assumed every language had an agent init container. Compiled languages instrument at build time — the image being pulled never existed.

Fix

Deleted the impossible resources and instrumented at compile time instead. Worth checking that a documented approach exists before rolling it out widely.

Part of this work

Observability Across 50+ Microservices ↗

Distributed tracing and real-time metrics across 50+ services — and 30% less alert noise.

Next incident

Dashboards green, services listed as onboarded, no data arriving ↗