dns · SEV-3
intermittent unknown-host errors on an external name that resolves perfectly when you test it by hand
How to confirm it
Count the round trips, do not guess
kubectl -n NS exec POD -- cat /etc/resolv.confRead ndots and the search list together. Dots-in-name below ndots multiplied by search entries is exactly how many failed lookups precede the real one.
Reproduce the search-domain expansion
kubectl -n NS exec POD -- sh -c 'for i in 1 2 3 4 5; do getent hosts HOST || echo MISS; done'A clean five-for-five by hand is the trap — the failure needs concurrency and a DNS hiccup. Absence of reproduction is not absence of cause.
Look for the negative cache, not the network
kubectl -n NS exec POD -- jcmd 1 VM.system_properties | grep negative.ttlA runtime that caches negative lookups for seconds turns one dropped packet into a cluster of user-visible errors. This explains the shape of the incident.
Rule out a deploy as the cause
kubectl get deploy -A -o json | jq -r '.items[] | select(.spec.template.spec.dnsConfig==null) | .metadata.name' | wc -lWhen no workload sets any DNS config, every pod inherits the same default and a recent migration cannot be the trigger. That killed the leading theory here.
Read the source
Cause
The cluster default sets a high ndots value with six search domains. A name with fewer dots than that threshold is tried against every search domain first, so one lookup becomes six round trips to cluster DNS before the absolute name is ever attempted. One transient failure on any of those variants fails the lookup, and the runtime then caches the negative result — which is what turns a blip into a run of failures.
Fix
Lower ndots for the workload so the name is tried absolutely first, or append a trailing dot in config to bypass search domains entirely. Setting the negative-cache TTL to zero removes the stickiness but not the storm. The fully general answer is resolving inside the mesh.