rtc: Commet joiners in encrypted rooms hear Discord ghosts only after seconds — 3 of 4 ghosts re-keyed the new device ~17 s after its membership (bridge-side recipient lag) #144

New issue

Open

opened 2026-10-01 23:47:34 +00:00 by robocub · 1 comment

robocub commented

2026-10-01 23:47:34 +00:00

Member

Symptom (operator, 2026-10-01): joining an encrypted bridged room's call from Commet, the Discord side (ghost audio) takes several seconds to become audible.

Evidence — prod journal, Purple Gaming, Commet device @dark:nether.im:inMftF8sdw (identified as Commet: its io.element.call.encryption_keys has member_id=-, cf. #135):

t (UTC)	event
23:44:42.18	Commet membership gone → leave-grace timer
23:44:45.67	membership re-appeared within grace
23:44:46.12	Commet's own key reaches the bridge (Commet→bridge direction is fine, ~0.5 s)
23:44:49.31	ghost `…1314350124666847294` re-keys Commet (`re-key delivered after membership change latency_ms=29`) — +3.6 s
23:45:02.64–.73	ghosts `…1467364551388299467`, `…996161727068643380`, `…868334831937937438` re-key Commet (`latency_ms=28/30/156`) — +17 s

Between 23:44:49 and 23:45:02 the three late ghosts' ~3 s key heartbeat did not include inMftF8sdw — only …1314…'s did. So the recipient set for 3 of 4 ghosts lagged the membership change by ~17 s, and all three caught up in the same 100 ms (looks like a shared periodic refresh, not the event-driven path). The latency_ms metric hides this: it is measured from each ghost's own detection, not from the membership event.

So at least part of the delay is bridge-side (per-ghost recipient refresh), on top of whatever Commet itself needs (key install / decryptor setup — not measured; we can't see Commet's side).

Unknowns / next steps (evidence first, no pre-fix):

Find what drives a ghost's recipient refresh after a membership change (rtc::ghost re-key trigger vs. the periodic path) and why one ghost got the event and three didn't. Is it the grace-window re-appear (membership "re-appeared within grace" may not fan out a members_gen bump to all ghosts — cf. #51 RefreshMode::Force + members_gen)?
Measure from the membership event, not from per-ghost detection (fix the metric so this is visible in logs/nvb-watch).
Reproduce with a fresh Commet join (not a grace re-appear) and with Element Call, to separate a Commet-only cost from the bridge lag. Compare with #42/#51 baselines (~1–3 s on Element).
If Commet adds its own seconds after the key lands, note it on #136 (upstream).

Related: #42, #51, #55, #135, #136.

**Symptom (operator, 2026-10-01):** joining an *encrypted* bridged room's call from **Commet**, the Discord side (ghost audio) takes several seconds to become audible. **Evidence — prod journal, Purple Gaming, Commet device `@dark:nether.im:inMftF8sdw`** (identified as Commet: its `io.element.call.encryption_keys` has `member_id=-`, cf. #135): | t (UTC) | event | |---|---| | 23:44:42.18 | Commet membership gone → leave-grace timer | | 23:44:45.67 | membership re-appeared within grace | | 23:44:46.12 | Commet's own key reaches the bridge (Commet→bridge direction is fine, ~0.5 s) | | **23:44:49.31** | ghost `…1314350124666847294` re-keys Commet (`re-key delivered after membership change latency_ms=29`) — **+3.6 s** | | **23:45:02.64–.73** | ghosts `…1467364551388299467`, `…996161727068643380`, `…868334831937937438` re-key Commet (`latency_ms=28/30/156`) — **+17 s** | Between 23:44:49 and 23:45:02 the three late ghosts' ~3 s key heartbeat did **not** include `inMftF8sdw` — only `…1314…`'s did. So the recipient set for 3 of 4 ghosts lagged the membership change by ~17 s, and all three caught up in the same 100 ms (looks like a shared periodic refresh, not the event-driven path). The `latency_ms` metric hides this: it is measured from each ghost's *own* detection, not from the membership event. **So at least part of the delay is bridge-side** (per-ghost recipient refresh), on top of whatever Commet itself needs (key install / decryptor setup — not measured; we can't see Commet's side). **Unknowns / next steps (evidence first, no pre-fix):** 1. Find what drives a ghost's recipient refresh after a membership change (`rtc::ghost` re-key trigger vs. the periodic path) and why one ghost got the event and three didn't. Is it the grace-window re-appear (membership "re-appeared within grace" may not fan out a members_gen bump to all ghosts — cf. #51 `RefreshMode::Force` + `members_gen`)? 2. Measure from the membership event, not from per-ghost detection (fix the metric so this is visible in logs/nvb-watch). 3. Reproduce with a fresh Commet join (not a grace re-appear) and with Element Call, to separate a Commet-only cost from the bridge lag. Compare with #42/#51 baselines (~1–3 s on Element). 4. If Commet adds its own seconds after the key lands, note it on #136 (upstream). Related: #42, #51, #55, #135, #136.

robocub referenced this issue from a commit

2026-10-02 01:52:16 +00:00

fix(matrix): re-key on every new call membership, not only a new MXID (#144)

robocub commented

2026-10-02 02:24:42 +00:00

Author

Member

Root cause found (crates/matrix-rtc/src/matrix.rs::apply_member_event): the ghosts' key loops are only woken (members_gen bump) when a new MXID appears. Two kinds of new key recipient never bumped it, so they waited for the 60 s key heartbeat:

a quick rejoin inside the leave grace ("membership re-appeared within grace" returned early with "do NOT … re-key"), although the rejoined client is a fresh session that needs the keys again (the Commet case above);
a second device of a member already in the call.

Fix on branch fix/144-rekey-on-new-device (5783a5c, CI test green): the participant-set update is now a pure, unit-tested function that reports whether the (user, state_key) membership is new; every new one wakes the key loops. Periodic membership refreshes still don't. Not merged/deployed yet — needs a live check (Commet rejoin: all ghosts should key the new device within ~1 s).

**Root cause found** (`crates/matrix-rtc/src/matrix.rs::apply_member_event`): the ghosts' key loops are only woken (`members_gen` bump) when a **new MXID** appears. Two kinds of new key recipient never bumped it, so they waited for the 60 s key heartbeat: 1. a quick **rejoin inside the leave grace** ("membership re-appeared within grace" returned early with "do NOT … re-key"), although the rejoined client is a fresh session that needs the keys again (the Commet case above); 2. a **second device** of a member already in the call. Fix on branch `fix/144-rekey-on-new-device` (5783a5c, CI test green): the participant-set update is now a pure, unit-tested function that reports whether the (user, state_key) membership is new; every new one wakes the key loops. Periodic membership refreshes still don't. Not merged/deployed yet — needs a live check (Commet rejoin: all ghosts should key the new device within ~1 s).

robocub referenced this issue

2026-10-02 16:53:22 +00:00

Evaluate element-hq/matrix-rust-rtc as the MatrixRTC layer for crates/matrix-rtc (#41) #146