Live-verify shared-pool cross-channel re-promotion under concurrent STEAL load

robocub commented

2026-06-17 23:28:28 +00:00

Collaborator

The cross-channel overflow re-promotion fix is merged to master (258eef7) and deployed live, but its real-world path has only been unit-verified (incl. a negative control) — it still needs a concurrent multi-channel STEAL scenario with several real speakers to confirm end-to-end.

The fix (background). When a channel activates and the pool is full, it steals a bot from a donor channel, demoting one of the donor's dedicated speakers into the overflow mix. Previously, when the stealing channel later freed that bot, the donor's demoted speaker was stranded in overflow forever (the "single overflow bot" symptom). 258eef7 adds promote_overflow_into_free_bots / rebalance in crates/discord/src/puppet.rs: whenever a bot returns to the pool, any already-anchored channel with overflow speakers and a guild-capable free bot gets its oldest overflow speaker promoted back to a dedicated puppet (FCFS, guild-scoped).

Live test to run (needs ~4+ speakers across 2 channels):

Fill one channel (e.g. Light Voice) with enough speakers to consume the shared pool.
Activate a second channel so it steals a bot → a donor speaker is demoted to overflow (one shared overflow bot).
Empty the stealing channel so its bot frees back to the pool.
Expect: the donor's demoted speaker is re-promoted to its own named, individually-mutable puppet — log line promoted overflow speaker into freed pool bot (cross-channel) — instead of lingering in overflow.

Also confirm guild isolation: a freed bot is only used for channels in a guild it's a member of.

Note: during the steal hop you may still see ~2 min of channel DEGRADED — that is the separate quarantine issue #23, not a regression of this fix.

Stop the bridge with SIGINT/Ctrl-C only (zombie-voice gotcha).

The cross-channel overflow **re-promotion** fix is merged to master (`258eef7`) and deployed live, but its real-world path has only been **unit-verified** (incl. a negative control) — it still needs a **concurrent multi-channel STEAL** scenario with several real speakers to confirm end-to-end. **The fix (background).** When a channel activates and the pool is full, it steals a bot from a donor channel, demoting one of the donor's dedicated speakers into the overflow mix. Previously, when the stealing channel later freed that bot, the donor's demoted speaker was stranded in overflow forever (the "single overflow bot" symptom). `258eef7` adds `promote_overflow_into_free_bots` / `rebalance` in `crates/discord/src/puppet.rs`: whenever a bot returns to the pool, any already-anchored channel with overflow speakers and a guild-capable free bot gets its oldest overflow speaker promoted back to a dedicated puppet (FCFS, guild-scoped). **Live test to run (needs ~4+ speakers across 2 channels):** 1. Fill one channel (e.g. Light Voice) with enough speakers to consume the shared pool. 2. Activate a second channel so it **steals** a bot → a donor speaker is demoted to overflow (one shared overflow bot). 3. **Empty the stealing channel** so its bot frees back to the pool. 4. **Expect:** the donor's demoted speaker is re-promoted to its own named, individually-mutable puppet — log line `promoted overflow speaker into freed pool bot (cross-channel)` — instead of lingering in overflow. Also confirm guild isolation: a freed bot is only used for channels in a guild it's a member of. **Note:** during the steal hop you may still see ~2 min of `channel DEGRADED` — that is the **separate** quarantine issue #23, not a regression of this fix. _Stop the bridge with SIGINT/Ctrl-C only (zombie-voice gotcha)._

dark added the

bug

label

2026-06-19 15:27:57 +00:00

robocub commented

2026-07-05 15:31:40 +00:00

Author

Collaborator

Still pending live confirmation — and the target moved with #46 (v0.3.0-alpha.5): allocation is now per-(bot, guild) and steals are same-guild-only, so the concurrent-STEAL scenario specifically needs two active bridged channels in the same guild (cross-guild pressure can no longer steal, by design). The overflow/re-promotion logic survived the #46 refactor with its unit coverage grown (the 34-test allocator suite exercises steal + re-promotion paths).

The multi-guild live session being set up for #46 could double as this test if two same-guild channels go active with real speakers.

Still pending live confirmation — and the target moved with #46 (v0.3.0-alpha.5): allocation is now per-`(bot, guild)` and **steals are same-guild-only**, so the concurrent-STEAL scenario specifically needs **two active bridged channels in the same guild** (cross-guild pressure can no longer steal, by design). The overflow/re-promotion logic survived the #46 refactor with its unit coverage grown (the 34-test allocator suite exercises steal + re-promotion paths). The multi-guild live session being set up for #46 could double as this test if two same-guild channels go active with real speakers.

robocub referenced this issue

2026-07-05 18:02:06 +00:00

Per-guild puppet allocation & multiplexing #46

robocub referenced this issue from a commit

2026-07-06 19:36:32 +00:00

test: Tier-2 named lifecycle scenarios (#38 Phase C2)

robocub referenced this issue from a commit

2026-07-07 00:42:33 +00:00

test: Tier-2 named lifecycle scenarios (#38 Phase C2)

robocub referenced this issue from a commit

2026-07-07 01:16:20 +00:00

test: Tier-2 named lifecycle scenarios (#38 Phase C2)

robocub referenced this issue

2026-07-11 22:33:36 +00:00

Automatic Testing #38

robocub referenced this issue from a commit

2026-07-12 00:09:53 +00:00

feat(e2e): add #6 steal_release_repromotion — live cross-channel steal/re-promotion (#24)

robocub referenced this issue from a commit

2026-07-12 00:55:31 +00:00

feat(e2e): add #6 steal_release_repromotion — live cross-channel steal/re-promotion (#24)

robocub commented

2026-07-12 00:57:28 +00:00

Author

Collaborator

Live-verified and merged to master (6fb61f5).

The e2e suite now has the live variant of this scenario: steal_release_repromotion (catalogue #6), composing #7's phantom pattern with #15's two-tenant setup on a restricted 2-bot pool. E2E run #194 (compressed profile) passed first try in 26.5s, hitting exactly the sequence this issue asked for:

00:22:29.9 assigned participant to puppet   bridge="Auto Test 01" key=@nvbtest_alice idx=0
00:22:31.4 assigned participant to puppet   bridge="Auto Test 01" key=@nvbtest_bob   idx=1
00:22:33.0 stole bot from donor to activate channel  donor="Auto Test 01" for_channel="Auto Test 03" idx=1 demoted=@nvbtest_bob
00:22:33.0 stole bot to activate channel anchor      bridge="Auto Test 03" key=@nvbtest_carol idx=1
00:22:36.7 promoted overflow speaker into freed pool bot (cross-channel)  channel="Auto Test 01" promoted=@nvbtest_bob idx=1

The scenario also asserts Discord voice presence per-channel at every step (2 → 1 → 2 bots in the donor channel, 0 → 1 → 0 in the thief channel) via the read-only gateway probe, and that the demoted/promoted participant is the donor's newest non-anchor speaker (FCFS). Regression run #195 confirmed overflow_fcfs_promotion and multi_bridge_isolation still pass alongside it. It runs in the nightly --all sweep from now on.

The guild-isolation half (a freed bot only serves guilds it is a member of) stays hermetic-only — the live env has a single test guild — covered by puppet.rs::freed_bot_only_repromotes_guild_capable_channel.

(The channel DEGRADED quarantine noise mentioned in the description did not occur; the steal hop completed cleanly.)

**Live-verified and merged to master (`6fb61f5`).** The e2e suite now has the live variant of this scenario: `steal_release_repromotion` (catalogue #6), composing #7's phantom pattern with #15's two-tenant setup on a restricted 2-bot pool. E2E run #194 (compressed profile) passed first try in 26.5s, hitting exactly the sequence this issue asked for: ``` 00:22:29.9 assigned participant to puppet bridge="Auto Test 01" key=@nvbtest_alice idx=0 00:22:31.4 assigned participant to puppet bridge="Auto Test 01" key=@nvbtest_bob idx=1 00:22:33.0 stole bot from donor to activate channel donor="Auto Test 01" for_channel="Auto Test 03" idx=1 demoted=@nvbtest_bob 00:22:33.0 stole bot to activate channel anchor bridge="Auto Test 03" key=@nvbtest_carol idx=1 00:22:36.7 promoted overflow speaker into freed pool bot (cross-channel) channel="Auto Test 01" promoted=@nvbtest_bob idx=1 ``` The scenario also asserts Discord voice presence per-channel at every step (2 → 1 → 2 bots in the donor channel, 0 → 1 → 0 in the thief channel) via the read-only gateway probe, and that the demoted/promoted participant is the donor's newest non-anchor speaker (FCFS). Regression run #195 confirmed `overflow_fcfs_promotion` and `multi_bridge_isolation` still pass alongside it. It runs in the nightly `--all` sweep from now on. The guild-isolation half (a freed bot only serves guilds it is a member of) stays hermetic-only — the live env has a single test guild — covered by `puppet.rs::freed_bot_only_repromotes_guild_capable_channel`. (The `channel DEGRADED` quarantine noise mentioned in the description did not occur; the steal hop completed cleanly.)

robocub closed this issue

2026-07-12 00:57:28 +00:00

robocub referenced this issue

2026-07-12 02:05:11 +00:00

mgmt: operator observability + messaging + instance notice (status/stats/msg/announce/instance_notice) #94

robocub referenced this issue

2026-07-19 22:12:58 +00:00

e2e nightly RED 8 days — real multi-channel/encrypted/recovery audio failure cluster (NOT contention); env-vs-code TBD #124

robocub referenced this issue

2026-07-26 03:53:20 +00:00

fix(discord): external displacement during puppet media-startup breaks recovery — rejoin succeeds but playback stays silent (deterministic; was masked by voice-state lag) #127

robocub referenced this issue from a commit