dark/nether-voicebridge

Fork 1

Ghost media-key delivery latency: ~30s of silence before Discord speakers are audible in Matrix (stale-device bloat + OTK claim failures) #42

New issue

Closed

opened 2026-06-24 01:03:25 +00:00 by robocub · 2 comments

robocub commented

2026-06-24 01:03:25 +00:00

Collaborator

Symptom

When a Matrix user joins a bridged call, Discord speakers (ghosts) appear in the
Element roster staggered ("one by one") and then remain silent for ~30 seconds
after appearing, before their audio becomes audible. Audio itself is correct once it
starts; the delay is entirely in per-device Olm media-key delivery (Discord→Matrix).

Live-observed 2026-06-24 on the VPS staging bridge (encrypted "Dark Voice" room).

Evidence (timings, user = `@dark`)

Ghost jackalope:

ghost publishing Discord speaker audio at 00:56:19.7
first ghost media key delivered at 00:56:49.8 → ~30 s of silence
at 00:57:07 even logged ghost media key reached NO recipient devices (transient, unreachable=12)

Olm session establishment to the receiving user:

00:56:48 Tried to create new Olm sessions, but the signed one-time key was missing for some devices missing_devices_by_user={"@dark": {WwRz4rI5q7, fw6bHG5CYL, l4esNRFzn1, wk18E4QVw2, yVlHOeMTCZ}} → Established new Olm sessions sessions={} (claim returned nothing)
00:56:54 retry finally established a session to the live device

Root causes (compounding)

Stale-device bloat. @dark has ~12 devices; 5 are permanently dead (no
one-time keys on the homeserver) — WwRz4rI5q7, fw6bHG5CYL, l4esNRFzn1, wk18E4QVw2, yVlHOeMTCZ. Every ghost key-send wastes a failing /keys/claim on these 5
(the constant unreachable=5). Likely accumulated via per-store device minting
(#30) across staging/prod/dev/wiped stores.
One-time-key availability on the homeserver (Tuwunel). Even the live device's
first claim returned no OTKs and had to be retried, adding seconds.
Fan-out cost. ~5 ghosts × ~12 devices ≈ 60 Olm sessions to establish, bursted/
serialized, so the first key to the live device lands ~30 s after audio starts.

Proposed bridge-side optimizations

Fast-path already-keyed devices: deliver the media key to existing Olm sessions
immediately rather than blocking the send behind new /keys/claim round-trips for
all devices (so the active device hears audio ASAP, slow/new devices catch up).
Cache known-OTK-less devices: remember devices whose claim repeatedly fails and
stop re-claiming them every burst (cuts wasted round-trips + the NO recipient noise).
Pre-distribute on presence: send the ghost's media key when the ghost is brought
online (presence) instead of waiting for first speech, so the key is in place before
audio flows.

Environmental mitigation (separate from the bridge)

Clean up dead devices on heavy-use accounts (sign out stale sessions); investigate
Tuwunel one-time-key replenishment.

Notes

Part of the deferred "crypto/sync latency" cluster surfaced alongside the puppet
DriverDisconnect recovery work (2026-06-24). The reached NO recipient devices
WARN and /sync errors observed in the same window are related symptoms of this
environment. Bridge correctness is fine; this is a latency/UX issue.

## Symptom When a Matrix user joins a bridged call, Discord speakers (ghosts) appear in the Element roster staggered ("one by one") and then remain **silent for ~30 seconds** after appearing, before their audio becomes audible. Audio itself is correct once it starts; the delay is entirely in **per-device Olm media-key delivery** (Discord→Matrix). Live-observed 2026-06-24 on the VPS staging bridge (encrypted "Dark Voice" room). ## Evidence (timings, user = `@dark`) Ghost `jackalope`: - `ghost publishing Discord speaker audio` at **00:56:19.7** - first `ghost media key delivered` at **00:56:49.8** → **~30 s of silence** - at **00:57:07** even logged `ghost media key reached NO recipient devices` (transient, `unreachable=12`) Olm session establishment to the receiving user: - `00:56:48` `Tried to create new Olm sessions, but the signed one-time key was missing for some devices missing_devices_by_user={"@dark": {WwRz4rI5q7, fw6bHG5CYL, l4esNRFzn1, wk18E4QVw2, yVlHOeMTCZ}}` → `Established new Olm sessions sessions={}` (claim returned nothing) - `00:56:54` retry finally established a session to the live device ## Root causes (compounding) 1. **Stale-device bloat.** `@dark` has ~12 devices; **5 are permanently dead** (no one-time keys on the homeserver) — `WwRz4rI5q7, fw6bHG5CYL, l4esNRFzn1, wk18E4QVw2, yVlHOeMTCZ`. Every ghost key-send wastes a failing `/keys/claim` on these 5 (the constant `unreachable=5`). Likely accumulated via per-store device minting (#30) across staging/prod/dev/wiped stores. 2. **One-time-key availability on the homeserver (Tuwunel).** Even the *live* device's first claim returned no OTKs and had to be retried, adding seconds. 3. **Fan-out cost.** ~5 ghosts × ~12 devices ≈ 60 Olm sessions to establish, bursted/ serialized, so the *first* key to the live device lands ~30 s after audio starts. ## Proposed bridge-side optimizations - **Fast-path already-keyed devices:** deliver the media key to existing Olm sessions immediately rather than blocking the send behind new `/keys/claim` round-trips for all devices (so the active device hears audio ASAP, slow/new devices catch up). - **Cache known-OTK-less devices:** remember devices whose claim repeatedly fails and stop re-claiming them every burst (cuts wasted round-trips + the `NO recipient` noise). - **Pre-distribute on presence:** send the ghost's media key when the ghost is brought online (presence) instead of waiting for first speech, so the key is in place before audio flows. ## Environmental mitigation (separate from the bridge) - Clean up dead devices on heavy-use accounts (sign out stale sessions); investigate Tuwunel one-time-key replenishment. ## Notes Part of the deferred "crypto/sync latency" cluster surfaced alongside the puppet `DriverDisconnect` recovery work (2026-06-24). The `reached NO recipient devices` WARN and `/sync` errors observed in the same window are related symptoms of this environment. Bridge correctness is fine; this is a latency/UX issue.

robocub added the

bug

label

2026-06-24 01:03:25 +00:00

robocub commented

2026-06-24 01:31:37 +00:00

Author

Collaborator

Update: device-scoped key targeting is the real fix (supersedes the failure-cache idea above)

A closer pass at the code changed the recommendation.

Also observed: the same root cause causes mid-call dropouts, not just join latency

A Matrix speaker (@dark) had an 8s E2eeStateChanged: MissingKey window
(00:57:13→00:57:21) — bridge couldn't decrypt them → Discord side heard them drop —
correlating exactly with a new participant joining (Victoria Lady at 00:57:10). Plus
~1s blips during a leave/rejoin. A Discord speaker reported as dropping (Mustapha Mond)
had continuous ghost audio health frames — capture never stalled; the loss was
receiver-side (missing his rotated key). Every join/leave triggers an EC media-key
rotation that must re-propagate over the same slow Olm path.

Root cause is narrower than "stale devices" — it's over-broad fan-out

enumerate_key_recipients (ghost.rs:1437) reads m.call.member but dedups to user,
discarding the device; the Olm transport (key_transport.rs:219) then fans out to every
device via get_user_devices. But m.call.member is per-device (ruma-events 0.34
active_memberships() exposes device_id; state key {user}_{device}), and the receive
side already works in device terms (keys.rs claimed_device_id). Evidence: delivered=7 unreachable=5 for @dark = the bridge keys all ~12 of their devices when only one is in
the call. The 5 "unreachable" are verified-but-offline sessions (OTKs exhausted; fallback
only partial — fallback_key_used true 6× / false 13×), claimed and failed 45× in one
session. These users' extra sessions are legitimate, not junk — so "delete stale devices" is
not the answer.

Recommended fix

Send each ghost's media key only to the device(s) actually in the call (from
active_memberships()), not all of a user's devices:

enumerate_key_recipients → return (user, device_id) pairs.
Olm path → target those exact devices (encryption().get_device(...)) instead of
get_user_devices(...).devices().

For @dark this is ~12 devices → 1: it removes the doomed dead-device claims and the
needless deliveries at the root, shrinks the initial ~30s silence (1 session to establish,
not 12), kills the NO recipient windows, and scales with "many users × many devices." This
supersedes the earlier failure-cache idea (dead devices simply aren't targeted anymore).

Out of scope / honest limits

Receive-side gaps (the @dark 8s window = subscriber waiting for a remote client's
rotated key) are a separate concern (subscriber sync responsiveness).
EC's rotation cadence on every join/leave is Element behavior, not ours.
This is E2EE-correctness-critical: must live-verify every participant (incl. a
multi-device user) still hears all ghosts after narrowing; fall back to per-user fan-out
if a real client needs the key on a non-membership device.

## Update: device-scoped key targeting is the real fix (supersedes the failure-cache idea above) A closer pass at the code changed the recommendation. ### Also observed: the same root cause causes **mid-call dropouts**, not just join latency - A Matrix speaker (`@dark`) had an **8s `E2eeStateChanged: MissingKey`** window (`00:57:13`→`00:57:21`) — bridge couldn't decrypt them → Discord side heard them drop — correlating exactly with a new participant joining (`Victoria Lady` at `00:57:10`). Plus ~1s blips during a leave/rejoin. A Discord speaker reported as dropping (`Mustapha Mond`) had **continuous** `ghost audio health` frames — capture never stalled; the loss was receiver-side (missing his *rotated* key). Every join/leave triggers an EC media-key rotation that must re-propagate over the same slow Olm path. ### Root cause is narrower than "stale devices" — it's over-broad fan-out `enumerate_key_recipients` (`ghost.rs:1437`) reads `m.call.member` but **dedups to user**, discarding the device; the Olm transport (`key_transport.rs:219`) then fans out to **every** device via `get_user_devices`. But `m.call.member` is **per-device** (ruma-events 0.34 `active_memberships()` exposes `device_id`; state key `{user}_{device}`), and the receive side already works in device terms (`keys.rs` `claimed_device_id`). Evidence: `delivered=7 unreachable=5` for `@dark` = the bridge keys **all ~12 of their devices when only one is in the call**. The 5 "unreachable" are verified-but-offline sessions (OTKs exhausted; fallback only partial — `fallback_key_used` true 6× / false 13×), claimed and failed 45× in one session. These users' extra sessions are legitimate, not junk — so "delete stale devices" is not the answer. ### Recommended fix Send each ghost's media key **only to the device(s) actually in the call** (from `active_memberships()`), not all of a user's devices: - `enumerate_key_recipients` → return `(user, device_id)` pairs. - Olm path → target those exact devices (`encryption().get_device(...)`) instead of `get_user_devices(...).devices()`. For `@dark` this is ~12 devices → 1: it removes the doomed dead-device claims **and** the needless deliveries at the root, shrinks the initial ~30s silence (1 session to establish, not 12), kills the `NO recipient` windows, and scales with "many users × many devices." This **supersedes** the earlier failure-cache idea (dead devices simply aren't targeted anymore). ### Out of scope / honest limits - **Receive-side** gaps (the `@dark` 8s window = subscriber waiting for a remote client's rotated key) are a separate concern (subscriber sync responsiveness). - EC's rotation cadence on every join/leave is Element behavior, not ours. - This is **E2EE-correctness-critical**: must live-verify every participant (incl. a multi-device user) still hears all ghosts after narrowing; fall back to per-user fan-out if a real client needs the key on a non-membership device.

robocub referenced this issue from a commit

2026-06-24 02:32:18 +00:00

release: v0.2.4

robocub closed this issue

2026-06-24 02:32:18 +00:00

robocub commented

2026-06-24 02:34:28 +00:00

Author

Collaborator

Resolved in v0.2.4 by three changes to ghost media-key delivery, all live-verified on staging:

Device-scoped targeting — a ghost now keys only the device(s) actually in the call (from m.call.member), not all of a listener's sessions. @dark went from 12 devices → 1 (delivered=7 unreachable=5 → delivered=1 unreachable=0); the doomed OTK claims against offline/stale devices and the "nobody can decrypt" windows are gone. This makes the "stale device bloat" moot at the root — no device cleanup required.
Lean ghost /sync — filtered bootstrap (timeline/ephemeral/presence/account-data dropped, membership+encryption kept), ~halving each bootstrap and lightening homeserver load.
Crypto warmed during presence — the bootstrap runs while a ghost idles in presence instead of all ghosts stampeding the homeserver the instant a human joins.

Result: Discord speakers audible in Matrix in ~1–3s (was ~20–37s), and no more multi-second dropouts on every membership-driven key rotation. Correctness held throughout (every participant still hears all ghosts).

Out of scope (separate): receive-side rotation gaps (the bridge subscriber waiting for a remote human's rotated key) and Element Call's rotation cadence.

Resolved in **v0.2.4** by three changes to ghost media-key delivery, all live-verified on staging: 1. **Device-scoped targeting** — a ghost now keys only the device(s) actually in the call (from `m.call.member`), not all of a listener's sessions. `@dark` went from 12 devices → 1 (`delivered=7 unreachable=5` → `delivered=1 unreachable=0`); the doomed OTK claims against offline/stale devices and the "nobody can decrypt" windows are gone. This makes the "stale device bloat" moot at the root — no device cleanup required. 2. **Lean ghost `/sync`** — filtered bootstrap (timeline/ephemeral/presence/account-data dropped, membership+encryption kept), ~halving each bootstrap and lightening homeserver load. 3. **Crypto warmed during presence** — the bootstrap runs while a ghost idles in presence instead of all ghosts stampeding the homeserver the instant a human joins. **Result:** Discord speakers audible in Matrix in **~1–3s** (was ~20–37s), and no more multi-second dropouts on every membership-driven key rotation. Correctness held throughout (every participant still hears all ghosts). Out of scope (separate): receive-side rotation gaps (the bridge subscriber waiting for a remote human's rotated key) and Element Call's rotation cadence.