Multi-SFU MatrixRTC (MSC4195): remote-homeserver users unbridged and reaped alive; Commet hears no ghosts — per-SFU fan-in + elected-SFU ghosts #139
Labels
No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
nether/nether-voicebridge#139
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Symptom
A user on another homeserver with a current client joins a bridged room's call: the bridge joins, ghosts appear, but no audio crosses in either direction, and the bridge's stale-membership reaper (#25) keeps emptying the user's live
m.call.member. Hit by tenant📞┃Limited-VC(@therealfame:matrix.org, Element Call) on 2026-09-28 (84 reaps in one evening) and silently bybsch(@eezra:matrix.org) since 2026-08-13.Root cause
Element Call 0.21 (July 2026) made MSC4195 multi-SFU the default: each participant publishes on its own homeserver's LiveKit SFU (
focus_selection: multi_sfu,foci_preferred[0]= e.g.livekit-jwt.call.matrix.org). The bridge assumed the whole call lived on the SFU from its own.well-knownand only ever connected there — it never heard remote-homeserver users, and judged their absence from our SFU as "phantom".A second, independent failure was found live with Commet (legacy
oldest_membershipelection): Commet listens only on the SFU elected by the oldest membership, so a ghost publishing on our SFU while a matrix.org Commet user is oldest is inaudible. Commet also skips memberships whosefocus_selectionit doesn't know, so advertisingmulti_sfuon ghosts made them invisible to it (it then fell back to its own SFU, and our resolver mis-placed and reaped it).Fix (branch
feat/multi-sfu-fanin)rtc::focus: per-member SFU resolution mirroring matrix-js-sdkgetTransport();elected_focus= oldest active membership's transport (ghost-only ⇒ ours).rtc::run::ForeignSessions: one subscribe-only LiveKit session per SFU that any human's resolved focus or ownfoci_preferred[0]names (JWT from their lk-jwt via our OpenID token); same router; E2EE shares the primary key provider + #73 heal gate.created_ts), and advertise the spec's legacyoldest_membershipnaming that SFU — valid for Commet's election and resolves correctly for multi-SFU Element Call.Verification
nvb-matrix-rtc); workspace clippy-D warningsgreen.--allaudio e2e dispatch; then merge + release.Follow-ups
focus_selection(so it can't hear multi-SFU Element Call users on other homeservers), sorts byorigin_server_tsnotcreated_ts, never re-elects mid-call.Released as v0.3.8 and deployed to prod 2026-09-30 ~03:05 UTC.
--allcompressed e2e on the merged branch (run 834): 27 pass + 1 known_fail (fed_cold_outreach_utd, #104, pre-existing).416ddd2, releaseca23e2e, tagv0.3.8.nether-voicebridge_0.3.8_amd64.debinstalled (sha256d9533f85…6dfc), unit restarted, 23/23 bridges alive, 0 ERROR / 0 WARN after restart, every bridge published its call-focus election.Release-asset gap (CI infra, not this fix): the tag's
image-arm64job failed withNo space left on deviceon the arm64 (Oracle Ampere) runner, somanifestandpackageswere skipped and the v0.3.8 release currently has no assets. The amd64 image was built and pushed by CI; the prod deb was packaged from that exact image with the repo'spackaging/nfpm.yamlvianfpmv2.46.3 — the same steps thepackagesjob runs. Once disk is freed on the arm64 runner, re-dispatchingdocker.ymlon refv0.3.8will publish the full asset set.Live verification and the Commet findings are in the issue body. Closing.
Follow-up regression fixed in v0.3.9 (tag pushed 2026-09-30 ~16:50 UTC; prod deploy follows the packages job).
Pinning
created_tsacross ghost membership re-posts (0.3.8) kept the election stable but also froze the membership's expiry (expiresis relative tocreated_ts): a Discord user present > 4 h before a call started was invisible to Commet/Element although their audio flowed — the active-phase re-post carried identical content and the server deduplicated it (Light Voice, huxley, created 04:41 / expired 08:41 / re-post 14:39 changed nothing).Fix (
7fcfef4, mergedbcf5506):expiresis computed at every post as(now − created_ts) + 4 h, and a keeper task re-posts hourly for the ghost's lifetime (presence, active, post-migration), socreated_tsstays fixed for the election while the membership never ages out. Teardown fires the ghost's cancel before awaiting the keeper so the join-before-leave ordering holds.Verification: unit tests for the arithmetic and wire shape; full
--alle2e on the branch = 25 pass + #104 known_fail + 2 audio-oracle fails (audio_d2m_enc,multi_bridge_isolation) that both passed on isolated re-runs (66 s / 96 s vs master's 95 s) under lower host load — the failing probes measure the Matrix→Discord puppet path the diff does not touch.Known remaining gap (low priority): the E2EE subscriber's own membership (
rtc/membership.rs) is posted once per call with the default 4 h expiry; a single encrypted call longer than 4 h would drop the bot from the roster.v0.3.9 deployed to prod 2026-09-30 ~17:25 UTC (official deb, 23/23 bridges, 0 ERROR/WARN; all Light Voice ghost memberships fresh with the hourly refresh armed).
The remaining gap is closed in v0.3.10 (tag pushed ~17:50 UTC, deploy follows the packages job): the bridge bot's own E2EE roster membership now uses the same keeper — fixed
created_tsper activation (kept across a disconnect-recovery rejoin),expiresrecomputed on every post, hourly refresh until the call ends. Helpers moved tortc::membershipand shared with the ghosts. Verified: unit tests; push CI; isolated e2eaudio_m2d_enc,audio_d2m_enc,audio_m2d_enc_freshjoin,fresh_store_rekey,ghost_session_rotationall pass (run 858).