mgmt: detect, notify, and (with consent) self-heal room call-permission breakage; optional bot-created rooms #95

New issue

Open

opened 2026-07-12 01:32:06 +00:00 by robocub · 1 comment

robocub commented

2026-07-12 01:32:06 +00:00

Collaborator

Operator idea (2026-07-12), prompted by the operator himself hitting the m.call.member PL trap in a prod room tonight (ghost 403 retry loop, Discord side invisible in the call, fixed by hand mid-call). Docs + an ops-room alert now cover it reactively (415f9c5/7efb53b + nvb-watch 🔧 alert); this issue is the in-product fix.

Tier 1 — detect. The ghost already logs the exact failure (ghost post_membership failed … 403 … org.matrix.msc3401.call.member (50)). Hook it: first 403 per room per session raises a room-misconfigured event. Also add the PL probe to the periodic mgmt maintenance sweep so regressions (room upgrades, permission edits) are caught for ALL tenants without waiting for someone to join a call — the link-time probe only runs once.

Tier 2 — notify. On detection: (a) post a plain chat message into the bridged Matrix room (the bot is a member and events_default is 0 even when state is locked, so chat still works): what broke, what it looks like, exactly how to fix; (b) DM the tenant owner (the #94 create_dm path exists now). Rate-limit to once per room per N hours.

Tier 3 — consented self-fix (existing rooms). New command (fixroom <name> or a reply-keyword on the in-room notice): the bot attempts the m.room.power_levels write itself. Honest constraint: writing power_levels typically requires PL ≥ events["m.room.power_levels"] (usually 100), which the bot normally lacks — on 403 it should reply with the exact deficit ("I have PL 0, this room requires 100 — promote me first or apply the fix by hand"). Never auto-write without an explicit human ask (state-mutation on someone else's room).

Tier 4 — bot-created rooms (the mautrix pattern, sidesteps the wall). Optional link variant where the BRIDGE creates the Matrix room: as creator it holds PL 100, so it configures call.member = 0 (and optionally encryption) perfectly, invites the requester and promotes them to admin, then links. Guaranteed-correct rooms for users who don't have one yet — the direct analogue of mautrix-discord creating portal rooms from the Discord side. Notes: room lands on the bridge's homeserver; nether.im instance implications (room count, who owns the room long-term) need a think; probably operator-gated at first.

Suggested order: 1+2 (small, high value, no permission questions) → 4 (feature) → 3 (needs the promote-the-bot UX story).

Operator idea (2026-07-12), prompted by the operator himself hitting the `m.call.member` PL trap in a prod room tonight (ghost 403 retry loop, Discord side invisible in the call, fixed by hand mid-call). Docs + an ops-room alert now cover it reactively (`415f9c5`/`7efb53b` + nvb-watch 🔧 alert); this issue is the in-product fix. **Tier 1 — detect.** The ghost already logs the exact failure (`ghost post_membership failed … 403 … org.matrix.msc3401.call.member (50)`). Hook it: first 403 per room per session raises a room-misconfigured event. Also add the PL probe to the periodic mgmt maintenance sweep so regressions (room upgrades, permission edits) are caught for ALL tenants without waiting for someone to join a call — the link-time probe only runs once. **Tier 2 — notify.** On detection: (a) post a plain chat message into the bridged Matrix room (the bot is a member and `events_default` is 0 even when state is locked, so chat still works): what broke, what it looks like, exactly how to fix; (b) DM the tenant owner (the #94 `create_dm` path exists now). Rate-limit to once per room per N hours. **Tier 3 — consented self-fix (existing rooms).** New command (`fixroom <name>` or a reply-keyword on the in-room notice): the bot attempts the `m.room.power_levels` write itself. Honest constraint: writing power_levels typically requires PL ≥ `events["m.room.power_levels"]` (usually 100), which the bot normally lacks — on 403 it should reply with the exact deficit ("I have PL 0, this room requires 100 — promote me first or apply the fix by hand"). Never auto-write without an explicit human ask (state-mutation on someone else's room). **Tier 4 — bot-created rooms (the mautrix pattern, sidesteps the wall).** Optional `link` variant where the BRIDGE creates the Matrix room: as creator it holds PL 100, so it configures `call.member = 0` (and optionally encryption) perfectly, invites the requester and promotes them to admin, then links. Guaranteed-correct rooms for users who don't have one yet — the direct analogue of mautrix-discord creating portal rooms from the Discord side. Notes: room lands on the bridge's homeserver; nether.im instance implications (room count, who owns the room long-term) need a think; probably operator-gated at first. Suggested order: 1+2 (small, high value, no permission questions) → 4 (feature) → 3 (needs the promote-the-bot UX story).

robocub referenced this issue from a commit

2026-07-12 02:09:18 +00:00

feat(mgmt): #95 tiers 1+2 — detect + notify bridged-room call-permission breakage

robocub referenced this issue from a commit

2026-07-12 02:19:34 +00:00

test(e2e): room_misconfig_notice — live drill for the #95 detect+notify sweep

robocub referenced this issue from a commit

2026-07-12 02:41:25 +00:00

fix(mgmt): #95 defer notification until the room is LOCALLY Joined

robocub commented

2026-07-12 04:00:06 +00:00

Author

Collaborator

Tiers 1+2 SHIPPED in v0.3.0-alpha.24, deployed to prod 2026-07-12 (issue stays open for tier 3 fixroom + tier 4 bot-created rooms).

Maintenance sweep now runs for EVERY deployment (reap duties stay open-reg-gated inside) and checks each bridged room's m.room.power_levels per tick via the existing pure evaluate_call_member_power. On failure: in-room fix-it notice + owner DM + room_misconfigured audit event + bridged room misconfigured WARN (nvb-watch 🔧 alert matches it), once per room per 24h. NVB_MAINTENANCE_INTERVAL_SECS (min 5s) for drills.
Permanent e2e scenario room_misconfig_notice (suite → 20): operator-created fresh room = misconfigured by construction; asserts notice/WARN/audit/cooldown; teardown leaves collected by the #89b reap.
The live drill caught two real startup races before prod did: (1) the first sweep can beat the initial sync, and (2) the appservice raw join leaves the LOCAL SDK state Invited so room.send() refuses — either would have burned the 24h cooldown silently. Fix: defer the whole notification until the room is locally Joined, BEFORE any cooldown stamping.
Verification: runner clippy + full-workspace --locked green; drill LIVE PASS 43.4s; mgmt_self_serve_link regression PASS; prod deploy clean (deb sha-verified, 0 ERROR, management bot ready), first prod sweep: zero false positives across all 9 bridges.
Ops docs: operations-runbook.md documents the new behavior; alpha.24 is also the first release with the #49 dual-arch assets (all 7 present).

**Tiers 1+2 SHIPPED in v0.3.0-alpha.24, deployed to prod 2026-07-12** (issue stays open for tier 3 `fixroom` + tier 4 bot-created rooms). - Maintenance sweep now runs for EVERY deployment (reap duties stay open-reg-gated inside) and checks each bridged room's `m.room.power_levels` per tick via the existing pure `evaluate_call_member_power`. On failure: in-room fix-it notice + owner DM + `room_misconfigured` audit event + `bridged room misconfigured` WARN (nvb-watch 🔧 alert matches it), once per room per 24h. `NVB_MAINTENANCE_INTERVAL_SECS` (min 5s) for drills. - Permanent e2e scenario `room_misconfig_notice` (suite → 20): operator-created fresh room = misconfigured by construction; asserts notice/WARN/audit/cooldown; teardown leaves collected by the #89b reap. - The live drill caught two real startup races before prod did: (1) the first sweep can beat the initial sync, and (2) the appservice raw join leaves the LOCAL SDK state `Invited` so `room.send()` refuses — either would have burned the 24h cooldown silently. Fix: defer the whole notification until the room is locally Joined, BEFORE any cooldown stamping. - Verification: runner clippy + full-workspace `--locked` green; drill LIVE PASS 43.4s; `mgmt_self_serve_link` regression PASS; prod deploy clean (deb sha-verified, 0 ERROR, `management bot ready`), first prod sweep: zero false positives across all 9 bridges. - Ops docs: operations-runbook.md documents the new behavior; alpha.24 is also the first release with the #49 dual-arch assets (all 7 present).