Gateway identify-storm circuit breaker — protect puppet tokens from Discord's 1000-connections/day auto-reset #129

New issue

Closed

opened 2026-08-08 02:48:46 +00:00 by robocub · 2 comments

robocub commented

2026-08-08 02:48:46 +00:00

Collaborator

Motivation — 2026-08-08 incident (mautrix-discord, same class of failure)

On 2026-08-07 21:38 UTC, Discord abruptly closed the mautrix-discord text bridge's established gateway connection with 4002: Error while decoding payload. The client library (discordgo) then looped for ~5 hours: reconnect → resume rejected (Op9) → fresh IDENTIFY → READY → dead again ~15 s later. ~1,333 IDENTIFYs crossed Discord's 1000-connections/day protection and Discord automatically reset the bot's token (email to owner). That bridge's Discord side was dead until manual re-login.

Nothing in nether-voicebridge caused this — but the same failure class applies here:

serenity's ShardQueuer retries identifies forever with only a ~5 s floor. A server-side rejection loop (Discord change, glitch, or client connection-state corruption) would burn ~1000 identifies within a day → puppet bot token auto-reset → that puppet is dead until an operator mints a new token and edits the config.
Voice connections do not count toward the limit (only main-gateway IDENTIFYs), and the e2e harness uses separate test bots, so normal operation is a handful of identifies/day. The risk is purely the pathological-loop case.
Until today this would have been invisible: prod's log filter dropped all serenity targets (0 serenity lines in 14 days of journal).

Mitigations already deployed (ops-side)

Prod RUST_LOG now appends serenity=info so gateway lifecycle lines ("Attempting to reconnect", "Received session invalidation", "Failed to resume", …) reach the journal (pending service restart).
nvb-watch tripwire: >20 serenity reconnect/invalid-session lines per 15-min window → 🌪 alert to the Ops room (sustained storm ≈ 60+/15 min; normal ≈ a handful/day).

Asks (code-side)

Default log filter: add serenity=info to the built-in EnvFilter default in crates/bridge/src/main.rs so every deployment gets gateway visibility without env config.
Supervisor circuit breaker: track gateway session deaths per puppet bot; if a bot's gateway dies N times within M minutes (suggest N=10, M=10 min — far above legitimate resume churn), park the bot (stop reconnecting, release its (bot,guild) allocations so other pool bots cover) and log a loud ERROR (nvb-watch already forwards ERRORs to the Ops room). A parked bot re-arms after a cool-down (suggest 1 h) or on process restart.
Consider counting IDENTIFYs specifically (not just session deaths) if serenity exposes the distinction, since the 1000/day budget is on IDENTIFY.

Design note: parking must go through the puppet-pool allocator (per-(bot,guild) unit, #46) so anchors fail over rather than leaving a channel uncaptured.

## Motivation — 2026-08-08 incident (mautrix-discord, same class of failure) On 2026-08-07 21:38 UTC, Discord abruptly closed the mautrix-discord text bridge's established gateway connection with `4002: Error while decoding payload`. The client library (discordgo) then looped for ~5 hours: reconnect → resume rejected (Op9) → fresh IDENTIFY → READY → dead again ~15 s later. ~1,333 IDENTIFYs crossed Discord's **1000-connections/day** protection and Discord **automatically reset the bot's token** (email to owner). That bridge's Discord side was dead until manual re-login. Nothing in nether-voicebridge caused this — but the same failure class applies here: - serenity's `ShardQueuer` retries identifies forever with only a ~5 s floor. A server-side rejection loop (Discord change, glitch, or client connection-state corruption) would burn ~1000 identifies within a day → **puppet bot token auto-reset** → that puppet is dead until an operator mints a new token and edits the config. - Voice connections do *not* count toward the limit (only main-gateway IDENTIFYs), and the e2e harness uses separate test bots, so normal operation is a handful of identifies/day. The risk is purely the pathological-loop case. - Until today this would have been **invisible**: prod's log filter dropped all `serenity` targets (0 serenity lines in 14 days of journal). ## Mitigations already deployed (ops-side) 1. Prod `RUST_LOG` now appends `serenity=info` so gateway lifecycle lines ("Attempting to reconnect", "Received session invalidation", "Failed to resume", …) reach the journal *(pending service restart)*. 2. nvb-watch tripwire: >20 serenity reconnect/invalid-session lines per 15-min window → 🌪 alert to the Ops room (sustained storm ≈ 60+/15 min; normal ≈ a handful/day). ## Asks (code-side) 1. **Default log filter**: add `serenity=info` to the built-in `EnvFilter` default in `crates/bridge/src/main.rs` so every deployment gets gateway visibility without env config. 2. **Supervisor circuit breaker**: track gateway session deaths per puppet bot; if a bot's gateway dies N times within M minutes (suggest N=10, M=10 min — far above legitimate resume churn), **park the bot** (stop reconnecting, release its (bot,guild) allocations so other pool bots cover) and log a loud ERROR (nvb-watch already forwards ERRORs to the Ops room). A parked bot re-arms after a cool-down (suggest 1 h) or on process restart. 3. Consider counting IDENTIFYs specifically (not just session deaths) if serenity exposes the distinction, since the 1000/day budget is on IDENTIFY. Design note: parking must go through the puppet-pool allocator (per-(bot,guild) unit, #46) so anchors fail over rather than leaving a channel uncaptured.

robocub referenced this issue from a commit

2026-08-19 23:46:19 +00:00

feat(discord): gateway identify-storm circuit breaker — park a looping bot before Discord resets its token (#129)

robocub commented

2026-08-19 23:47:35 +00:00

Author

Collaborator

Implementation on branch feat/129-identify-circuit-breaker (ff36121), awaiting review/merge. All three asks:

Default log filter: serenity=info added to the built-in EnvFilter default — every deployment gets gateway lifecycle visibility without env config.
Circuit breaker: every gateway ready = one completed IDENTIFY (a successful RESUME fires no ready), so per-bot ready-counting counts exactly what Discord's 1000/day budget counts — which also answers ask 3 (we count IDENTIFYs, not just session deaths). 10 readies in 10 min → the bot is parked: allocations fail over through the per-(bot,guild) allocator (anchors included — reuses the join-timeout failover path), guild footprint cleared, command loop retired, gateway shut down via ShardManager. Loud ERROR (nvb-watch forwards it).
Re-arm: after a 1 h cooldown run() rebuilds the client; the fresh ready re-seeds guilds and rebalances. Worst case a still-broken bot spends ~11 identifies per ~70 min (≈200/day) — safely inside budget while still self-healing.

Hermetic tests cover the window math and the park→skip→re-arm allocator lifecycle (367 workspace tests green). Live storm injection isn't reasonably testable — the ops-side tripwire (🌪 in nvb-watch) plus the new serenity logs are the observability net.

Implementation on branch `feat/129-identify-circuit-breaker` (`ff36121`), awaiting review/merge. All three asks: 1. **Default log filter**: `serenity=info` added to the built-in EnvFilter default — every deployment gets gateway lifecycle visibility without env config. 2. **Circuit breaker**: every gateway `ready` = one completed IDENTIFY (a successful RESUME fires no `ready`), so per-bot ready-counting counts exactly what Discord's 1000/day budget counts — which also answers ask 3 (we count IDENTIFYs, not just session deaths). 10 readies in 10 min → the bot is **parked**: allocations fail over through the per-(bot,guild) allocator (anchors included — reuses the join-timeout failover path), guild footprint cleared, command loop retired, gateway shut down via ShardManager. Loud ERROR (nvb-watch forwards it). 3. **Re-arm**: after a 1 h cooldown `run()` rebuilds the client; the fresh `ready` re-seeds guilds and rebalances. Worst case a still-broken bot spends ~11 identifies per ~70 min (≈200/day) — safely inside budget while still self-healing. Hermetic tests cover the window math and the park→skip→re-arm allocator lifecycle (367 workspace tests green). Live storm injection isn't reasonably testable — the ops-side tripwire (🌪 in nvb-watch) plus the new serenity logs are the observability net.

robocub referenced this issue from a commit

2026-08-19 23:54:17 +00:00

feat(discord): gateway identify-storm circuit breaker — park a looping bot before Discord resets its token (#129)

robocub referenced this issue from a commit

2026-08-20 00:05:27 +00:00

Merge feat/129-identify-circuit-breaker: park a looping puppet gateway before Discord resets its token (#129)

robocub referenced this issue from a commit

2026-08-20 00:05:27 +00:00

release: 0.3.3 — mgmt-bot boot resilience + identify-storm circuit breaker (#126, #129)

robocub commented

2026-08-20 00:06:33 +00:00

Author

Collaborator

Merged to master (40de972) and released in v0.3.3. serenity=info default logging, per-bot IDENTIFY counting (10 in 10 min parks the bot via the allocator with failover + gateway shutdown), 1h cooldown re-arm. Closing.

Merged to master (40de972) and released in **v0.3.3**. serenity=info default logging, per-bot IDENTIFY counting (10 in 10 min parks the bot via the allocator with failover + gateway shutdown), 1h cooldown re-arm. Closing.

robocub closed this issue

2026-08-20 00:06:33 +00:00