realtime media · SEV-1

users cannot join voice sessions, while pod restart counts sit at zero and load metrics look calm

scroll to render

How to confirm it

Do not trust restart counts with forked workers
```
kubectl -n NS get pods -o custom-columns=NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount
```
Zero restarts through a real crash storm. The signal is the non-zero exit code in the application log, not the pod status.
Correlate candidate error lines against crash minutes
```
kubectl -n NS logs -l app=SVC --since=8h | grep -oE 'no worker is available|no response from servers' | sort | uniq -c
```
Bucket each line per minute and compare crash minutes against quiet ones. One line fires several times more often during crashes; the other is flat. That ratio picks the real signal out of two loud candidates.
Check whether the error names a target
```
kubectl -n NS logs -l app=SVC --since=8h | grep 'no response from servers' | grep -c 'agentName":""'
```
An error that is always unnamed, occurring alongside successful named dispatches, is structural noise from an implicit attempt. Chasing it costs hours.
Rule out capacity explicitly
```
curl -s localhost:9090/metrics | grep -E 'worker_load|active_job_count'
```
Load peaking well under threshold while registration holds steady means scaling up cannot help. Say so early, before someone doubles the fleet.
Establish whether it is a regression
```
kubectl -n NS logs -l app=SVC --since=336h | grep -c 'exit code -11'
```
Counting per day over weeks showed it was chronic, not new. That changed the response from an emergency rollback to a scheduled fix — and explained why a lower environment on identical config looked fine: it simply carried a third of the concurrency.

Read the source

Cause

Each session ran in a forked subprocess, so a segfault in a native audio library killed the job and not the pod — restart counts stayed clean through a crash storm of hundreds per hour. Every crash triggered a respawn whose native warm-up took several seconds, and during that window the worker's free-process count dropped and it advertised itself unavailable. A named dispatch then found no worker, the session never got an agent, and it closed on the idle timeout.

Fix

Serialise the native initialisation so respawns do not starve the pool, and patch or upgrade the pinned library. Scaling out does not help: workers go unavailable on process-pool exhaustion, not load. Two candidate error lines looked equally plausible — one carried a named agent and correlated with crashes, the other was unnamed and appeared alongside successful sessions. Only the first was the failure.

Next incident

connection reset by peer, immediately after enabling STRICT mTLS ↗