Multi-tenant hardening for a hosted instance #47
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Roadmap: Stage 5 — the hardening work that makes the public instance (#43) safe to actually run. Depends on the lean ghosts (#44), runtime provisioning (Stage 2), and per-guild multiplexing (Stage 3).
org.matrix.msc3401.call.member, and whether the tenant's homeserver actually runs MatrixRTC / Element Call (no EC support => no call to join).Background analysis in
docs/nether-voicebridge-public-instance-feasibility.md(Tier 2).Slice 1 shipped:
linknow probes room prerequisites before writing (merged to master,34bed3b+ docsad30670).The management bot's
linkno longer replies with an advisory checklist — it verifies up front, and a failed probe writes nothing:@voicebridge_asfirst, then runlinkagain".m.room.power_levels, evaluatesevents["org.matrix.msc3401.call.member"]→state_defaultfallback → spec default 50, againstusers_default(what ghosts join at). The error explains the Element Call / devtools fix and warns that Element's "Roles & Permissions" UI label is unreliable. Absent event fails safe..well-knownRTC-foci discovery, exactly what the spawned bridge will do.ControlMsg::GuildCoverageanswered by the supervisor from pool memory; warn-only because bots learn guilds at gatewayready, so a fresh process can't tell "not invited" from "still connecting".Live-tested on staging 2026-07-05 (all four scenarios): positive link against Auto Test 01 → all ✓ and bridge up; fresh default-PL room → the PL error verbatim,
tenants.tomluntouched (bonus: Tuwunel's createRoom seeds an explicitevents[call.member]=50, so the events-override path got exercised live); restricted room without invite → M_FORBIDDEN with the invite hint, nothing written; bogus guild → link succeeds with the ⚠️ coverage warning. Zero errors in the log; unit tests cover the PL evaluation matrix (override / fallback / absent / string-encoded legacy levels).Remaining #47 scope: Discord ToS research doc (slice 2), per-tenant isolation design doc (slice 3); link authority model + quotas still deferred.
Slice 3 (per-tenant isolation) — first hardening shipped: per-bridge panic boundary, merged to master
2b56935(unreleased, rides the next release).Finding: every bridge future and the shared puppet-pool future are polled on ONE task (they're
!Send), and nothing caught panics — so a panic in a single bridge'snvb_matrix_rtc::rununwound the whole supervisor task and took down every co-resident tenant plus the pool. Error isolation was already per-bridge and bounded; panic isolation was absent. That's both a fault-containment hole and a trivial whole-service DoS for a hosted instance.Fix:
guard_bridge_panic(AssertUnwindSafe+catch_unwind) wraps each bridge future at the supervisor push site. A caught panic logs and resolves to the bridge label exactly like a normal exit, so the existingon_bridge_completeteardown (unregister_channel, WAV finalize) runs for just that tenant and its neighbours keep running. The pool future is deliberately left fail-fast (a pool panic degrades everyone anyway — clean crash + restart beats limping on). Relies on the defaultpanic=unwind; documented in-code.Verification: full workspace
cargo test196/196 green +clippy --workspace --all-targets -- -D warningsclean, incl. 3 new unit tests (pass-through, panic containment, and a FuturesUnordered driving-loop-survives test mirroring the realrun()seam). No live smoke (unit tests replicate the unwinding semantics faithfully).Remaining slice-3/hardening work, in order: admission budgets/quotas at the
reconcile_tenants/add_tenant_entrychokepoints (max ghosts per tenant is currently unbounded), owner records + audit log + ban list, two-sided link authority; process-per-tenant (broker/worker split) stays gated on the ToS verdict (#61) + pool economics.Two-sided link auth shipped: merged to master
c6c02ef(unreleased, rides the next release).Two optional
[management]knobs, both default off (existing deployments unchanged):require_room_authority— Matrix side: thelinksender must be a joined member of the target room with PL ≥ itsstate_default(a room moderator), not just an allowlisted operator. Evaluated inside the slice-1 preflight probe, reusing the fetchedm.room.power_levelsplus onem.room.memberGET; fail-safe (absent PL event ⇒ required 50).require_discord_approval— Discord side: alinkthat passes every probe is held pending (nothing written) under a one-timeNVB-XXXXXXcode (16-entry cap,approval_timeout_secsTTL, default 600 s). Someone with Manage Server in the target guild must run/bridge approve code:<code>— a global slash command registered by the pool bot atapproval_bot_index(default 0), hidden from non-managers viadefault_member_permissionsand permission-re-checked server-side from the interaction payload. Approval re-checks the ban list fail-closed, commits through the same tenants.toml + reconcile path as a direct link, and notifies both sides. Audit gainslink_pending/approve_rejected(with the real refusal reason);link_okrecordsapproved_by.Layering:
nvb-discordandnvb-matrix-rtcstay decoupled — the binary adapts the pool'sLinkApprovalinto the bot'sDiscordApproval(reply oneshot passes through).get_room_stategained astate_keyparam; serenity gained thebuilderfeature (dm_permissionis deprecated in 0.12.5 →contexts(Guild)).Live-tested on staging 2026-07-06 (prod down 21 min, restored clean), all five scenarios exact:
/bridge approve→ ephemeral success + owner-recorded tenants.toml entry +link_pending→link_ok(approved_by)audit chain + ops-DM notification + bridge fully spawned.Runner-verified: 230 workspace tests +
clippy --workspace --all-targets -D warningsclean (rebased onto the #38 Phase A/B commits; zero overlap).Deployment gotcha (now in the README): the bot-invite OAuth URL must join scopes with
%20, not+— the+form silently dropsapplications.commandsand the slash command never appears client-side (registration still succeeds server-side, which makes it confusing). Also: Discord clients cache the command list — Ctrl+R after scope changes.Remaining #47 scope: process-per-tenant isolation stays gated on the ToS follow-ups (privacy policy, disclosure/consent UX, 100-server verification plan). Known v1 gaps, deliberate: no
pendinglist/cancel command (codes expire on their own),unlinkauthority unchanged (allowlist-gated), a pending link doesn't reserve its tenant name (duplicate caught cleanly at commit).