← ALL POSTMORTEMS

Roblox

73 hours down, and the monitoring depended on the thing that broke

A Consul feature flag enabled a week earlier met a latent BoltDB free-page bug. Two unrelated faults, one of which only appeared under the other's load.

Duration
73 hours
Blast radius
50M users · 18,000 servers · 170,000 containers
Organisation
Roblox

The part that stings

Every diagnostic they reached for ran on top of Consul. The observability stack went down with the thing it was meant to observe, so for two days they were debugging a distributed system essentially blind.

Timeline

  1. Oct 28 13:37

    Vault degrades; one Consul server shows high CPU. Looks like a bad node.

  2. Oct 28 16:35

    Online players fall to 50%. Full outage follows.

  3. Oct 29 02:00

    Attempt 1 — reset the cluster from a snapshot.

  4. Oct 29 04:00

    Snapshot reset fails. Consul degrades again even at reduced load.

  5. Oct 29 16:00

    Attempt 2 — disable non-essential Consul use, scale services to single digits.

  6. Oct 30 02:00

    Attempt 2 fails. Hardware theories are exhausted; focus moves to Consul internals.

  7. Oct 30 12:00

    Root causes identified from perf reports and flame graphs.

  8. Oct 30 15:51

    Streaming disabled. Median KV write latency drops 2s → 300ms within minutes.

  9. Oct 31 16:45

    Full service restored. 73 hours.

Fault 1 — Consul streaming contention

Consul 1.10 shipped a streaming feature to replace long-polling and cut CPU and bandwidth. It used fewer Go channels for concurrency control. Under simultaneous high read and write load that concentrated contention on a single channel, blocking writes. Dual-socket NUMA hardware made it worse: cross-socket contention on shared resources amplified the stall.

Fault 2 — BoltDB freelist amplification

Consul stores Raft logs in BoltDB. Snapshots delete old entries, but BoltDB doesn't return pages to the OS — it tracks freed 4KB pages in a freelist. Under this workload the freelist grew to 7.8MB holding nearly one million page IDs, inside a 4.2GB file containing just 489MB of real data. BoltDB rewrites the entire freelist on every write, so a 16KB Raft append wrote 7.8MB — roughly 500× amplification. That backpressure filled TCP receive buffers and produced 2–3 second write latencies on the leader.

Why it took 73 hours

The two faults were unrelated and neither was visible in normal metrics. Streaming had been on for a day before symptoms appeared, so it didn't look like a recent change. They replaced a node, upgraded the whole cluster to 128-core servers with faster NVMe, then went back to 64-core — none of it helped, which is what finally ruled out hardware and forced them into the code.

Commands that mattered

  • Where they actually found it

    perf record -F 99 -a -g -- sleep 30 && perf script | stackcollapse-perf.pl | flamegraph.pl > consul.svg

    The flame graph showed contention in kernel spin locks reached through the streaming code path. Metrics showed a slow system; the profile showed which line.

  • The freelist, visible in one command

    bbolt stats /var/lib/consul/raft/raft.db

    Reports page count, freelist size and data size. A 4.2GB file holding 489MB of data is the whole story in three numbers.

  • Checking Raft health without the UI

    consul operator raft list-peers && consul info | grep -E 'commit_index|last_log_index|applied_index'

    A widening gap between last_log_index and applied_index means the leader is accepting writes faster than followers apply them.

  • The fix

    consul reload   # after setting rpc.enable_streaming = false

    Disabling one feature flag returned write latency to normal within minutes, after two days of hardware swaps.

What it teaches

  • 01Observability that runs on the system it observes is not observability. Roblox's telemetry depended on Consul, so it died with it — the first fix they shipped afterwards was removing that circular dependency.
  • 02A feature enabled days ago is still a recent change. Streaming went on a day before symptoms; correlation by wall-clock is what delayed the diagnosis.
  • 03Metrics tell you a system is slow. Profiles tell you why. Nothing in dashboards would have surfaced a freelist rewrite.
  • 04Storage engines lie about disk usage. 4.2GB on disk, 489MB of data, and no alert anywhere covered that ratio.
  • 05Bigger hardware is a seductive hypothesis because it's easy to test. Two upgrades cost them a day and disproved nothing until they went back down again.

This rhymes with something I hit

Trace ingestion stopped for every tenant at once ↗

Same shape as the shared-queue jam: one component everyone depends on, degrading under a load pattern nobody modelled, taking every tenant down at once.

Next postmortem

Cloudflare — 27 minutes ↗

Sources

This is an analysis of a published incident at Roblox. It is not my own work, and every figure above is taken from the source linked here.