graceful SIGTERM leaves an active anchor puppet in Discord voice ~90s (voice-leave not flushed before exit) #74

New issue

Closed

opened 2026-07-10 21:48:23 +00:00 by robocub · 1 comment

robocub commented

2026-07-10 21:48:23 +00:00

Collaborator

Summary

The #38 e2e scenario shutdown_drain (D3) surfaced this: on graceful SIGTERM, an active capture-anchor puppet lingers in Discord voice for ~90-100s after the bridge process exits, rather than dropping promptly.

What happens

The drain itself is correct - the bridge logs, in order:

Discord shutdown - draining puppets (leave voice, reset nicknames)
puppet left voice and released (idx 0, the anchor)
puppets drained
all bridges drained cleanly

...and then the process exits, all within ~250ms of the SIGTERM. But because the process tears down milliseconds after songbird's leave(), the voice-state-update does not reliably flush to Discord's gateway before exit. Discord then clears the bot's voice state via its RTC keepalive timeout (~90-100s) instead of immediately.

Measured on staging: a ConnCountProbe sees the anchor puppet still in voice at 60s post-exit; it clears by ~100s.

Why it matters

Constraint #6 ("teardown returns the connection count to baseline") is only satisfied after a ~90s Discord-side delay for an active anchor. On a bridge restart, puppet bots appear stuck in the voice channel for ~1.5 min before Discord drops them. Self-heals, but ugly.

Only affects the active anchor

The normal release path (human leaves -> grace -> puppet released while the process keeps running) flushes the leave fine and Discord clears it in seconds. The gap is specific to leave-then-immediately-exit.

Suggested fix

On shutdown, await/flush the puppet voice-leave gateway update (or a brief bounded delay) before exiting, so Discord observes the leave promptly.

Repro

nvb-e2e run --scenario shutdown_drain - the scenario currently uses a 150s baseline window to accommodate the RTC-timeout clear; a prompt flush would let that tighten back down.

Found by the #38 D3 shutdown_drain scenario (batch 1, merged to master e16082d).

## Summary The #38 e2e scenario `shutdown_drain` (D3) surfaced this: on graceful `SIGTERM`, an **active capture-anchor** puppet lingers in Discord voice for ~90-100s after the bridge process exits, rather than dropping promptly. ## What happens The drain itself is correct - the bridge logs, in order: - `Discord shutdown - draining puppets (leave voice, reset nicknames)` - `puppet left voice and released` (idx 0, the anchor) - `puppets drained` - `all bridges drained cleanly` ...and then the process exits, all within ~250ms of the SIGTERM. But because the process tears down **milliseconds** after songbird's `leave()`, the voice-state-update does not reliably flush to Discord's gateway before exit. Discord then clears the bot's voice state via its RTC keepalive **timeout (~90-100s)** instead of immediately. Measured on staging: a `ConnCountProbe` sees the anchor puppet still in voice at 60s post-exit; it clears by ~100s. ## Why it matters Constraint #6 ("teardown returns the connection count to baseline") is only satisfied after a ~90s Discord-side delay for an active anchor. On a bridge restart, puppet bots appear stuck in the voice channel for ~1.5 min before Discord drops them. Self-heals, but ugly. ## Only affects the active anchor The normal release path (human leaves -> grace -> puppet released while the process keeps running) flushes the leave fine and Discord clears it in seconds. The gap is specific to leave-then-immediately-exit. ## Suggested fix On shutdown, await/flush the puppet voice-leave gateway update (or a brief bounded delay) before exiting, so Discord observes the leave promptly. ## Repro `nvb-e2e run --scenario shutdown_drain` - the scenario currently uses a 150s baseline window to accommodate the RTC-timeout clear; a prompt flush would let that tighten back down. _Found by the #38 D3 `shutdown_drain` scenario (batch 1, merged to master `e16082d`)._

robocub referenced this issue

2026-07-11 16:11:51 +00:00

e2e nightly: shutdown_drain + external_disconnect_recovery + force_move_recovery fail in CI (first scheduled run) #81

robocub referenced this issue from a commit

2026-07-11 17:06:23 +00:00

fix(discord): flush puppet voice-leave on shutdown before dropping clients (#74)

robocub commented

2026-07-11 17:07:02 +00:00

Author

Collaborator

Fixed in cf82850 (master). On graceful shutdown the pool now keeps the puppet clients POLLED a bounded VOICE_LEAVE_FLUSH_GRACE (1s, well under the 10s teardown timeout) after queuing each voice-leave, so the Op4 voice-state-update flushes over the gateway before the sockets close.

Live-verified on staging: shutdown_drain PASS 26.6s with the baseline window tightened 150s→30s, even under host load 13. Bridge log shows the leave queued (puppet left voice and released) → ~1.0s flush grace → all bridges drained cleanly, and Discord reflected baseline well within 30s instead of the prior ~90s RTC-keepalive lag. Closing.

Fixed in `cf82850` (master). On graceful shutdown the pool now keeps the puppet clients POLLED a bounded `VOICE_LEAVE_FLUSH_GRACE` (1s, well under the 10s teardown timeout) after queuing each voice-leave, so the Op4 voice-state-update flushes over the gateway before the sockets close. Live-verified on staging: `shutdown_drain` PASS 26.6s with the baseline window tightened 150s→30s, even under host load 13. Bridge log shows the leave queued (`puppet left voice and released`) → ~1.0s flush grace → `all bridges drained cleanly`, and Discord reflected baseline well within 30s instead of the prior ~90s RTC-keepalive lag. Closing.

robocub closed this issue

2026-07-11 17:07:03 +00:00

robocub referenced this issue

2026-07-11 17:07:14 +00:00

e2e nightly: shutdown_drain + external_disconnect_recovery + force_move_recovery fail in CI (first scheduled run) #81

robocub referenced this issue

2026-07-11 22:33:36 +00:00

Automatic Testing #38