Slow/uneven media-key re-delivery when a Matrix listener rejoins an encrypted call (follow-up to #44) #51

New issue

Closed

opened 2026-06-28 18:57:01 +00:00 by robocub · 1 comment

robocub commented

2026-06-28 18:57:01 +00:00

Collaborator

Symptom

When a Matrix listener leaves an encrypted call and rejoins, Discord speakers
become audible again only slowly and unevenly — some ghosts re-key the
returning listener within a few seconds, others take ~30-35s, so the listener
hears static/partial audio first and the main speaker can be last to come back.

Live-observed on v0.3.0-alpha.1 (Dark Voice, encrypted), listener @dark
device ihjNlBVX90, same device across the rejoin:

~T+0s    listener rejoins
T+~13s   ghosts 325541945 / 1169803 / 1221495  -> media key delivered=1 (fast)
T+~35s   ghost 1010121  -> delivered=1, after a /keys/claim round
T+~40s   ghost 909365   -> delivered=1, same delayed pattern

The slow ghosts logged, just before finally delivering:

receive_keys_claim_response: Tried to create new Olm sessions, but the signed
one-time key was missing for some devices
missing_devices_by_user={"@dark:nether.im": {WwRz4rI5q7, fw6bHG5CYL, l4esNRFzn1,
wk18E4QVw2, yVlHOeMTCZ}}  -> Established new Olm sessions sessions={}

The audio core stayed clean throughout (jitter_dropped=0, capture_errs=0), so
this is purely media-key re-delivery latency, not audio loss.

Likely mechanism

Fast ghosts are the ones whose cached OlmMachine ([GhostCryptoCache], #44)
still held a warm Olm session to the listener's device, so re-keying was a single
encrypt + /sendToDevice. The slow ghosts had to rebuild the session on demand
(update_tracked_users -> /keys/query -> get_missing_sessions -> /keys/claim
in key_transport.rs::ensure_tracked_and_sessions), and that claim first wrestled
with the listener's stale offline devices (no OTKs) before reaching the live
device — all gated by distribute_key_loop's burst/heartbeat cadence
(ghost.rs).

Suspected #44 side effect (UNCONFIRMED): before #44, each ghost ran a perpetual
/sync that kept recipient device-lists and Olm sessions warm, so a rejoin re-keyed
near-instantly. #44 replaced that with on-demand query/claim (the whole point — it
retired the FD-exhaustion /sync fan-out), which is slower to re-warm a lapsed
session. This needs an A/B against master to confirm it's actually a regression
— we only ever tested a Discord-user rejoin on master, never a listener rejoin,
so master's listener-rejoin latency is unknown and may also have been slow. Do not
assume; measure first.

Fix directions (to investigate)

On a recipient (re)join (members_gen bump), proactively burst-key all
active ghosts immediately and force a session re-establish, rather than letting
each ghost's loop reach it on its own cadence.
Restrict the per-recipient /keys/claim to the live in-call device so a
rejoin doesn't spend time on the listener's dead offline devices' missing OTKs
(we already target the in-call device for the send; the claim/session step may
still fan out wider).
Optionally a per-room shared device-tracker (already noted as an optional
follow-up in #44) to warm device-lists once per room instead of per ghost.

Notes

This is an alpha (v0.3.0-alpha.1); not a blocker, but it degrades the
rejoin UX on encrypted calls.
Separate from #50 (decouple enumeration from the bot Client) and from the
fidelity question (debug-build + double-transcode/mono, not corruption).

## Symptom When a **Matrix listener leaves an encrypted call and rejoins**, Discord speakers become audible again only **slowly and unevenly** — some ghosts re-key the returning listener within a few seconds, others take ~30-35s, so the listener hears static/partial audio first and the main speaker can be last to come back. Live-observed on `v0.3.0-alpha.1` (Dark Voice, encrypted), listener `@dark` device `ihjNlBVX90`, same device across the rejoin: ``` ~T+0s listener rejoins T+~13s ghosts 325541945 / 1169803 / 1221495 -> media key delivered=1 (fast) T+~35s ghost 1010121 -> delivered=1, after a /keys/claim round T+~40s ghost 909365 -> delivered=1, same delayed pattern ``` The slow ghosts logged, just before finally delivering: ``` receive_keys_claim_response: Tried to create new Olm sessions, but the signed one-time key was missing for some devices missing_devices_by_user={"@dark:nether.im": {WwRz4rI5q7, fw6bHG5CYL, l4esNRFzn1, wk18E4QVw2, yVlHOeMTCZ}} -> Established new Olm sessions sessions={} ``` The audio core stayed clean throughout (`jitter_dropped=0`, `capture_errs=0`), so this is purely **media-key re-delivery latency**, not audio loss. ## Likely mechanism Fast ghosts are the ones whose cached `OlmMachine` ([`GhostCryptoCache`], #44) still held a warm Olm session to the listener's device, so re-keying was a single encrypt + `/sendToDevice`. The slow ghosts had to rebuild the session on demand (`update_tracked_users` -> `/keys/query` -> `get_missing_sessions` -> `/keys/claim` in `key_transport.rs::ensure_tracked_and_sessions`), and that claim first wrestled with the listener's **stale offline devices** (no OTKs) before reaching the live device — all gated by `distribute_key_loop`'s burst/heartbeat cadence (`ghost.rs`). **Suspected #44 side effect (UNCONFIRMED):** before #44, each ghost ran a perpetual `/sync` that kept recipient device-lists and Olm sessions warm, so a rejoin re-keyed near-instantly. #44 replaced that with on-demand query/claim (the whole point — it retired the FD-exhaustion `/sync` fan-out), which is slower to re-warm a lapsed session. **This needs an A/B against master to confirm it's actually a regression** — we only ever tested a *Discord-user* rejoin on master, never a *listener* rejoin, so master's listener-rejoin latency is unknown and may also have been slow. Do not assume; measure first. ## Fix directions (to investigate) - On a **recipient (re)join** (members_gen bump), proactively burst-key *all* active ghosts immediately and force a session re-establish, rather than letting each ghost's loop reach it on its own cadence. - Restrict the per-recipient `/keys/claim` to the **live in-call device** so a rejoin doesn't spend time on the listener's dead offline devices' missing OTKs (we already target the in-call device for the *send*; the claim/session step may still fan out wider). - Optionally a per-room shared device-tracker (already noted as an optional follow-up in #44) to warm device-lists once per room instead of per ghost. ## Notes - This is an **alpha** (`v0.3.0-alpha.1`); not a blocker, but it degrades the rejoin UX on encrypted calls. - Separate from #50 (decouple enumeration from the bot Client) and from the fidelity question (debug-build + double-transcode/mono, not corruption).

robocub added the

bug

label

2026-06-28 18:57:25 +00:00

robocub referenced this issue from a commit

2026-07-03 02:47:06 +00:00

feat(matrix-rtc): E2EE decrypt-state handler + media-key fingerprints

robocub referenced this issue from a commit

2026-07-03 02:47:06 +00:00

fix(matrix-rtc): stable ghost media-key delivery — rejoin re-key (#51) + persistent identity (#35)

robocub referenced this issue

2026-07-04 00:37:00 +00:00

Ghost to-device receive path / Olm wedge self-heal #52

robocub referenced this issue

2026-07-05 15:31:36 +00:00

perf(matrix-rtc): replace per-ghost full crypto Client with a lean send-only OlmMachine (MemoryStore) to retire FD exhaustion #44

robocub commented

2026-07-05 15:31:38 +00:00

Author

Collaborator

Fixed in v0.3.0-alpha.2 (master 1bf9723). Two changes: the first key-send to a device uses RefreshMode::Force (bypasses the SDK's stale device-list cache for that recipient), and rejoin bumps a membership generation so every ghost re-sends promptly. Measured on the original repro: a rejoining listener is re-keyed by all ghosts in ~244 ms, down from the 30–35 s uneven straggle described here.

The Grace/L.Ron "flicker" thread of this report was proven client-side via key-fingerprint logging — the bridge sent byte-identical keys through the flaps (the fp logging is kept for future triage).

The deeper receive-path problem (a listener whose inbound Olm session dies can't be detected by a send-only ghost) grew into #52 — max-age session rotation, shipped and closed with definitive live proof 2026-07-05 — and #55 (reactive re-key via MSC4203, blocked on Tuwunel). Closing.

**Fixed in v0.3.0-alpha.2** (master `1bf9723`). Two changes: the first key-send to a device uses `RefreshMode::Force` (bypasses the SDK's stale device-list cache for that recipient), and rejoin bumps a membership generation so every ghost re-sends promptly. Measured on the original repro: a rejoining listener is re-keyed by **all** ghosts in ~244 ms, down from the 30–35 s uneven straggle described here. The Grace/L.Ron "flicker" thread of this report was proven **client-side** via key-fingerprint logging — the bridge sent byte-identical keys through the flaps (the fp logging is kept for future triage). The deeper receive-path problem (a listener whose *inbound* Olm session dies can't be detected by a send-only ghost) grew into #52 — max-age session rotation, shipped and closed with definitive live proof 2026-07-05 — and #55 (reactive re-key via MSC4203, blocked on Tuwunel). Closing.

robocub closed this issue

2026-07-05 15:31:38 +00:00

robocub referenced this issue

2026-07-07 14:41:28 +00:00

Decouple ghost key-recipient enumeration from the shared bot Client (follow-up to #44) #50