Shared pool: ~2-min DEGRADED after a cross-channel steal (hop join timeout → 120s quarantine) #23

New issue

Closed

opened 2026-06-17 23:17:23 +00:00 by robocub · 1 comment

robocub commented

2026-06-17 23:17:23 +00:00

Collaborator

Symptom (observed live 2026-06-17, 3 bridges sharing a 4-bot pool). When a channel activates and the pool is full, it steals a bot from a donor channel. The stolen bot must leave its old Discord voice channel and join the new one (a leave→join hop). Under concurrent load this hop sometimes times out, and the channel then logs puppet join failed and no healthy pool bot free — channel DEGRADED repeatedly for ~2 minutes before the bot finally re-joins.

Root cause. A join timeout calls on_join_failed (crates/discord/src/puppet.rs), which quarantines the stuck bot via unhealthy=true. The quarantine is only cleared after QUARANTINE_COOLDOWN = 120s (puppet.rs:54) by recover_bot, which then re-activates the channel. So the ~2-min DEGRADED window is exactly the 120s quarantine cooldown: the steal hop fails, the only candidate bot is quarantined, no spare exists, and the channel waits the full cooldown to recover.

Why it needs its own fix. This is a voice-join-reliability / timing problem (the leave→join hop is racy under load), distinct from the allocation-bookkeeping bug just fixed (cross-channel overflow re-promotion). Fixing it safely needs concurrent-load reproduction to tell a transient hop timeout from a genuinely stuck bot.

Candidate directions (not yet validated):

Distinguish a deliberate hop timeout (steal: leave→join of a healthy bot) from a genuinely stuck bot, and retry the hop with a short first backoff instead of the full 120s.
Ensure the manager.remove (leave) fully settles before manager.join (re-join) — the hop may be joining too soon after leaving.
Consider a shorter first-retry cooldown with backoff, rather than a flat 120s.

Related: the cross-channel re-promotion fix (same session) and the earlier puppet-channel-hop-join-timeout quarantine+failover work.

Filed from a live investigation; see the steal path steal_for_activation and on_join_failed/recover_bot in crates/discord/src/puppet.rs.

**Symptom (observed live 2026-06-17, 3 bridges sharing a 4-bot pool).** When a channel activates and the pool is full, it *steals* a bot from a donor channel. The stolen bot must leave its old Discord voice channel and join the new one (a leave→join hop). Under concurrent load this hop sometimes times out, and the channel then logs `puppet join failed and no healthy pool bot free — channel DEGRADED` repeatedly for **~2 minutes** before the bot finally re-joins. **Root cause.** A join timeout calls `on_join_failed` (`crates/discord/src/puppet.rs`), which quarantines the stuck bot via `unhealthy=true`. The quarantine is only cleared after `QUARANTINE_COOLDOWN = 120s` (`puppet.rs:54`) by `recover_bot`, which then re-activates the channel. So the ~2-min DEGRADED window is exactly the 120s quarantine cooldown: the steal hop fails, the only candidate bot is quarantined, no spare exists, and the channel waits the full cooldown to recover. **Why it needs its own fix.** This is a voice-join-reliability / timing problem (the leave→join hop is racy under load), distinct from the allocation-bookkeeping bug just fixed (cross-channel overflow re-promotion). Fixing it safely needs **concurrent-load reproduction** to tell a transient hop timeout from a genuinely stuck bot. **Candidate directions (not yet validated):** - Distinguish a deliberate *hop* timeout (steal: leave→join of a healthy bot) from a genuinely stuck bot, and retry the hop with a short first backoff instead of the full 120s. - Ensure the `manager.remove` (leave) fully settles before `manager.join` (re-join) — the hop may be joining too soon after leaving. - Consider a shorter first-retry cooldown with backoff, rather than a flat 120s. **Related:** the cross-channel re-promotion fix (same session) and the earlier `puppet-channel-hop-join-timeout` quarantine+failover work. _Filed from a live investigation; see the steal path `steal_for_activation` and `on_join_failed`/`recover_bot` in `crates/discord/src/puppet.rs`._

robocub referenced this issue from a commit

2026-06-17 23:23:39 +00:00

puppet: re-promote overflow speakers cross-channel when a bot frees

robocub referenced this issue

2026-06-17 23:28:28 +00:00

Live-verify shared-pool cross-channel re-promotion under concurrent STEAL load #24

robocub referenced this issue from a commit

2026-06-18 20:01:10 +00:00

puppet: exponential backoff for join-timeout quarantine (#23)

robocub commented

2026-06-18 20:06:40 +00:00

Author

Collaborator

Fixed in 4733459 — replaced the flat 120s quarantine cooldown with per-bot exponential backoff (15→30→60→120s cap, reset on a successful join). Live-verified via fault injection (forced manager.join timeouts): one bot cycled 15→30→60→120→120s, each interval accurate to ~2ms, then reset on success. Closing.

Fixed in `4733459` — replaced the flat 120s quarantine cooldown with per-bot exponential backoff (15→30→60→120s cap, reset on a successful join). Live-verified via fault injection (forced `manager.join` timeouts): one bot cycled 15→30→60→120→120s, each interval accurate to ~2ms, then reset on success. Closing.

robocub closed this issue

2026-06-18 20:06:40 +00:00