observability · SEV-3
Dashboards green, services listed as onboarded, no data arriving
scroll to render
How to confirm it
Injected, or just annotated?
kubectl get pods -A -o json | jq -r '.items[] | "\(.metadata.namespace)/\(.metadata.name) containers=\(.spec.containers|length)"' | grep 'containers=1'An annotation without an extra container means the operator never acted. The manifest looks right and nothing was injected.
Is data actually arriving?
# entity exists != entity reporting — check last-seen, not presence curl -s -H "Api-Key: $NR_KEY" 'https://api.newrelic.com/graphql' -d '{"query":"{actor{entitySearch(query:\"reporting=false\"){results{entities{name}}}}}"}'Stale entities keep appearing in dashboards long after they stop reporting. Filter on reporting=false to find them.
What did the CRD migration drop?
kubectl get deploy -A -o json | jq -r '.items[] | select(.spec.template.metadata.annotations != null and .spec.template.metadata.labels["app-language"] == null) | .metadata.name'Annotation present, label missing — the exact signature of the regression.
Read the source
Cause
A CRD migration dropped the labels while leaving the annotation in place. Entities stayed in the backend as stale objects, so nothing looked broken.
Fix
Audited every deployment end to end — labels, injection, and actual data arrival — rather than trusting the onboarding doc. True coverage was 40%.