dark/nether-voicebridge

Fork 1

Subscriber inbound-frame decryptor wedges permanently when frames arrive before the peer key (encrypted-room Matrix→Discord silence) #73

New issue

Closed

opened 2026-07-10 15:22:23 +00:00 by robocub · 3 comments

robocub commented

2026-07-10 15:22:23 +00:00

Collaborator

P1, live-reproduced 2026-07-07, receive-side. In an encrypted room, when a human publisher's media frames reach the subscriber (rtc::session playback consumer) before that publisher's media key is installed into the KeyProvider, libwebrtc's inbound frame decryptor latches into a permanently-silent state: LiveKit reports E2EE state=Ok, the frame counter keeps advancing, but every output frame is digital silence (peak_db=-96.0), and later key installs / rotations do not heal it. Result: that Matrix speaker is inaudible on Discord for the rest of the call. The listener direction (ghost→Matrix) is unaffected.

This is the receive-side cousin of the #52 Ruffle wedge (which was send-side / ghost-outbound). #52's rotation fix does not apply here — this is the bridge's own inbound decryptor.

Trigger / race

The subscriber's playback_consumer starts and pulls the first frame the instant it subscribes to the track. The peer key arrives via a separate to-device event. When frames win that race, the decryptor sees ciphertext with no key → wedges. A subscriber already in the call when the publisher joins usually wins the key race (key known first) and is fine — so it manifests specifically on fresh joins (bridge (re)connects, or a Discord/Matrix user joins an encrypted call mid-session).

Live evidence (stock v0.3.0-alpha.13, encrypted Dark Voice, fresh join)

20:28:02 playback: first frame from LiveKit track=@dark:nether.im ...
20:28:02 E2EE state changed identity=@dark:...:Fc97QROPUK state=MissingKey
20:28:04 installed peer media key peer_identity=@dark:...:Fc97QROPUK key_index=3 installed=true
20:28:04 E2EE state changed identity=@dark:...:Fc97QROPUK state=Ok
20:28:07 inbound LiveKit level track=@dark:nether.im frames=500  peak_db=-96.0
20:28:12 inbound LiveKit level track=@dark:nether.im frames=1000 peak_db=-96.0
   ... -96.0 dB for the rest of the call; index-4 key install later also does not heal ...

State goes MissingKey -> Ok, yet audio stays at the -96 dB silence floor permanently. A pre-block run where the subscriber was already keyed before the publisher's track arrived played real audio (-24..-55 dB) from the same code — confirming the race, not a config problem.

Bisect note

Found while live-testing #66. Reproduced identically on stock alpha.13 with a fresh-join protocol, which exonerated the #66 branch (every earlier A/B flip tracked join-order, not the binary). So this is a pre-existing bug, unrelated to #66.

Likely fix directions (to investigate)

Hold/delay the playback_consumer's frame intake (or the LiveKit subscribe) until the peer key for that identity is installed — i.e. gate first-frame decode on key-present.
Or, on a MissingKey -> Ok transition (or on each install_peer_key for an already-subscribed identity), force the decryptor to re-initialize for that participant so buffered/subsequent frames decode (equivalent of the #52 rotation but receive-side).
Investigate whether livekit-rust's failure_tolerance / key-ring behavior latches a participant's cryptor after N failures such that a later set_key doesn't re-arm it (KeyProviderOptions is failure_tolerance:-1, ratchet_window_size:16 in both rtc::session and rtc::ghost).

Impact

Encrypted-room Matrix→Discord audio is unreliable on any fresh join that loses the key race. Also blocks the encrypted-room live-validation of #66 (whose own logic is media-E2EE-agnostic and whose peer-key replay path is unit-tested, so #66 is not blocked by this).

Filed from the #66 live-test session.

**P1, live-reproduced 2026-07-07, receive-side.** In an encrypted room, when a human publisher's media frames reach the subscriber (`rtc::session` playback consumer) **before** that publisher's media key is installed into the `KeyProvider`, libwebrtc's inbound frame decryptor latches into a permanently-silent state: LiveKit reports E2EE `state=Ok`, the frame counter keeps advancing, but every output frame is digital silence (`peak_db=-96.0`), and **later key installs / rotations do not heal it**. Result: that Matrix speaker is inaudible on Discord for the rest of the call. The listener direction (ghost→Matrix) is unaffected. This is the *receive-side* cousin of the #52 Ruffle wedge (which was send-side / ghost-outbound). #52's rotation fix does not apply here — this is the bridge's own inbound decryptor. ### Trigger / race The subscriber's `playback_consumer` starts and pulls the first frame the instant it subscribes to the track. The peer key arrives via a separate to-device event. When frames win that race, the decryptor sees ciphertext with no key → wedges. A subscriber already in the call when the publisher joins usually wins the key race (key known first) and is fine — so it manifests specifically on **fresh joins** (bridge (re)connects, or a Discord/Matrix user joins an encrypted call mid-session). ### Live evidence (stock v0.3.0-alpha.13, encrypted Dark Voice, fresh join) ``` 20:28:02 playback: first frame from LiveKit track=@dark:nether.im ... 20:28:02 E2EE state changed identity=@dark:...:Fc97QROPUK state=MissingKey 20:28:04 installed peer media key peer_identity=@dark:...:Fc97QROPUK key_index=3 installed=true 20:28:04 E2EE state changed identity=@dark:...:Fc97QROPUK state=Ok 20:28:07 inbound LiveKit level track=@dark:nether.im frames=500 peak_db=-96.0 20:28:12 inbound LiveKit level track=@dark:nether.im frames=1000 peak_db=-96.0 ... -96.0 dB for the rest of the call; index-4 key install later also does not heal ... ``` State goes `MissingKey -> Ok`, yet audio stays at the -96 dB silence floor permanently. A pre-block run where the subscriber was *already keyed before the publisher's track arrived* played real audio (-24..-55 dB) from the same code — confirming the race, not a config problem. ### Bisect note Found while live-testing #66. Reproduced identically on **stock alpha.13** with a fresh-join protocol, which **exonerated the #66 branch** (every earlier A/B flip tracked join-order, not the binary). So this is a pre-existing bug, unrelated to #66. ### Likely fix directions (to investigate) - Hold/delay the `playback_consumer`'s frame intake (or the LiveKit subscribe) until the peer key for that identity is installed — i.e. gate first-frame decode on key-present. - Or, on a `MissingKey -> Ok` transition (or on each `install_peer_key` for an already-subscribed identity), force the decryptor to re-initialize for that participant so buffered/subsequent frames decode (equivalent of the #52 rotation but receive-side). - Investigate whether livekit-rust's `failure_tolerance` / key-ring behavior latches a participant's cryptor after N failures such that a later `set_key` doesn't re-arm it (KeyProviderOptions is `failure_tolerance:-1`, `ratchet_window_size:16` in both `rtc::session` and `rtc::ghost`). ### Impact Encrypted-room Matrix→Discord audio is unreliable on any fresh join that loses the key race. Also blocks the encrypted-room live-validation of #66 (whose own logic is media-E2EE-agnostic and whose peer-key replay path is unit-tested, so #66 is not blocked by this). Filed from the #66 live-test session.

robocub referenced this issue

2026-07-10 15:35:15 +00:00

[review] matrix-rtc: LiveKit RoomEvent::Disconnected is swallowed with no recovery (subscriber reaper can wipe real participants) #66

robocub referenced this issue from a commit

2026-07-10 16:21:35 +00:00

fix(matrix-rtc): heal inbound decryptor wedged by frames-before-key (#73)

robocub commented

2026-07-10 16:31:38 +00:00

Author

Collaborator

Fix implemented on branch fix/issue73-decryptor-wedge (a70dc4c) — runner-green (workspace tests + clippy -D warnings), staging binary pre-built on the VPS. Awaiting the fresh-join live test on an encrypted room before merge.

Root-cause findings (upstream source archaeology)

Our livekit 0.7.45 ships a prebuilt libwebrtc built from webrtc-sdk/webrtc @ m144_release (via rust-sdks release tag webrtc-51ef663). Reading the actual decrypt path (api/crypto/frame_crypto_transformer.cc):

With our failure_tolerance: -1, DecryptionFailure() always returns false → after the first successful decrypt flips state to Ok, subsequent per-frame decrypt failures emit NO state change ever again — which is exactly why the logs show state=Ok while audio is dead. The state event is untrustworthy for detecting this wedge.
Failed frames are dropped before the decoder (silence, not noise), matching the observed peak_db=-96.0 at a healthy frame cadence.
The key ring itself is NOT the latch: SetKey → SetKeyFromMaterial + SetHasValidKey resets has_valid_key_ and the failure counter, and the control run (key-before-frames) works with an identical post-install ring. So the wedged state lives in the per-receiver chain (transformer instance / receiver / decode pipeline) — inside the prebuilt libwebrtc, not patchable from Rust.

Fix (Rust-layer, two parts)

Heal-on-key-install (the main fix). The session's room-event loop now tracks, per publisher identity, whether that peer's media key was already delivered at TrackSubscribed time (HealGate). When the first key for a keyless-subscribed identity is installed (to-device handler → session notifier channel), the track is unsubscribed → resubscribed once: the SDK mints a fresh RtpReceiver + FrameCryptor against the now-keyed provider, i.e. the proven-good "key before frames" join order. The existing TrackUnsubscribed/TrackSubscribed handling restarts the playback consumer cleanly; puppets are untouched (membership owns release). Later installs/rotations don't re-trigger (the resubscribe re-records the identity as keyed).
#66-rejoin window closed. The key provider is now built, shared with the to-device handler, and re-seeded from the PeerKeyCache before Room::connect (previously replay ran post-connect, leaving a frames-before-key window on every disconnect-recovery rejoin).

Unencrypted bridges: zero behavior change (no provider → gate inert).

Expected live-test signature (fresh join, encrypted room, human already publishing)

track subscribed before its peer media key — will re-subscribe when the key lands (#73)
installed peer media key ...
peer media key installed for a track subscribed keyless — re-subscribing to rebuild its decryptor (#73)
LiveKit: unsubscribed from remote participant track
LiveKit: subscribed to remote participant track
... inbound LiveKit level ... peak_db > -90 while speaking

**Fix implemented on branch `fix/issue73-decryptor-wedge`** (`a70dc4c`) — runner-green (workspace tests + clippy `-D warnings`), staging binary pre-built on the VPS. **Awaiting the fresh-join live test on an encrypted room before merge.** ### Root-cause findings (upstream source archaeology) Our livekit 0.7.45 ships a prebuilt libwebrtc built from `webrtc-sdk/webrtc @ m144_release` (via rust-sdks release tag `webrtc-51ef663`). Reading the actual decrypt path (`api/crypto/frame_crypto_transformer.cc`): - With our `failure_tolerance: -1`, `DecryptionFailure()` always returns `false` → after the first successful decrypt flips state to `Ok`, **subsequent per-frame decrypt failures emit NO state change ever again** — which is exactly why the logs show `state=Ok` while audio is dead. The state event is untrustworthy for detecting this wedge. - Failed frames are dropped before the decoder (silence, not noise), matching the observed `peak_db=-96.0` at a healthy frame cadence. - The key ring itself is NOT the latch: `SetKey` → `SetKeyFromMaterial` + `SetHasValidKey` resets `has_valid_key_` and the failure counter, and the control run (key-before-frames) works with an identical post-install ring. So the wedged state lives in the **per-receiver chain** (transformer instance / receiver / decode pipeline) — inside the prebuilt libwebrtc, not patchable from Rust. ### Fix (Rust-layer, two parts) 1. **Heal-on-key-install (the main fix).** The session's room-event loop now tracks, per publisher identity, whether that peer's media key was already delivered at `TrackSubscribed` time (`HealGate`). When the first key for a keyless-subscribed identity is installed (to-device handler → session notifier channel), the track is **unsubscribed → resubscribed once**: the SDK mints a fresh RtpReceiver + FrameCryptor against the now-keyed provider, i.e. the proven-good "key before frames" join order. The existing `TrackUnsubscribed`/`TrackSubscribed` handling restarts the playback consumer cleanly; puppets are untouched (membership owns release). Later installs/rotations don't re-trigger (the resubscribe re-records the identity as keyed). 2. **#66-rejoin window closed.** The key provider is now built, shared with the to-device handler, and re-seeded from the `PeerKeyCache` **before** `Room::connect` (previously replay ran post-connect, leaving a frames-before-key window on every disconnect-recovery rejoin). Unencrypted bridges: zero behavior change (no provider → gate inert). ### Expected live-test signature (fresh join, encrypted room, human already publishing) ``` track subscribed before its peer media key — will re-subscribe when the key lands (#73) installed peer media key ... peer media key installed for a track subscribed keyless — re-subscribing to rebuild its decryptor (#73) LiveKit: unsubscribed from remote participant track LiveKit: subscribed to remote participant track ... inbound LiveKit level ... peak_db > -90 while speaking ```

robocub referenced this issue from a commit

2026-07-10 16:38:03 +00:00

e2e: audio_m2d_enc_freshjoin — the #73 wedge-ordering regression scenario

robocub commented

2026-07-10 16:48:18 +00:00

Author

Collaborator

Live-validated on staging 2026-07-10 — automated, via a new #38 e2e scenario. Prod stayed up throughout (dedicated harness bots + Auto Test rooms).

New regression scenario: `audio_m2d_enc_freshjoin` (branch commit `3868f16`)

The existing m2d scenarios only speak after the key is installed, so they never exercised the frames-before-key race. The new scenario makes the #73 ordering deterministic: the Matrix speaker joins encrypted Auto Test 02 and streams continuously before the bridge process even starts, so the subscriber's first RTP deterministically beats the Olm-delivered key. It then asserts the full heal chain from the bridge log and oracles a post-heal probe on a Discord listener.

Result: PASS (74.2s)

Bridge log sequence, exactly as designed:

16:43:11.838 track subscribed before its peer media key — will re-subscribe when the key lands (#73)
16:43:11.969 playback consumer started            (wedged pre-key chain)
16:43:13.821 peer media key installed for a track subscribed keyless — re-subscribing to rebuild its decryptor (#73)
16:43:14.286 playback consumer started            (fresh keyed chain)

Exactly one heal (later key installs / heartbeat re-sends didn't re-trigger), zero bridge ERRORs, and the post-heal audio oracle: mean NCC 0.927, 7/8 good cycles (0.977–0.992; live-healthy band). On stock alpha.13 this exact ordering live-produced permanent -96 dB silence — the heal converts the wedge case into the working case end-to-end.

Regression check: audio_m2d_enc also passes on the fixed binary (mean NCC 0.871 — its historical band). One earlier run of it scored 0.126, fully explained by the documented shared-host gotcha (a CI build was pegging the cores during that exact window; the bridge-side log for that run shows a clean decrypt chain and state=Ok throughout — re-run on a quiet host passed).

Branch is runner-green (workspace tests + clippy -D warnings) with the scenario included. Ready to merge on go-ahead.

**Live-validated on staging 2026-07-10 — automated, via a new #38 e2e scenario.** Prod stayed up throughout (dedicated harness bots + Auto Test rooms). ### New regression scenario: `audio_m2d_enc_freshjoin` (branch commit `3868f16`) The existing m2d scenarios only speak after the key is installed, so they never exercised the frames-before-key race. The new scenario makes the #73 ordering deterministic: the Matrix speaker joins encrypted Auto Test 02 and **streams continuously before the bridge process even starts**, so the subscriber's first RTP deterministically beats the Olm-delivered key. It then asserts the full heal chain from the bridge log and oracles a post-heal probe on a Discord listener. ### Result: PASS (74.2s) Bridge log sequence, exactly as designed: ``` 16:43:11.838 track subscribed before its peer media key — will re-subscribe when the key lands (#73) 16:43:11.969 playback consumer started (wedged pre-key chain) 16:43:13.821 peer media key installed for a track subscribed keyless — re-subscribing to rebuild its decryptor (#73) 16:43:14.286 playback consumer started (fresh keyed chain) ``` Exactly **one** heal (later key installs / heartbeat re-sends didn't re-trigger), zero bridge ERRORs, and the post-heal audio oracle: **mean NCC 0.927, 7/8 good cycles** (0.977–0.992; live-healthy band). On stock alpha.13 this exact ordering live-produced permanent `-96 dB` silence — the heal converts the wedge case into the working case end-to-end. Regression check: `audio_m2d_enc` also passes on the fixed binary (mean NCC 0.871 — its historical band). One earlier run of it scored 0.126, fully explained by the documented shared-host gotcha (a CI build was pegging the cores during that exact window; the bridge-side log for that run shows a clean decrypt chain and `state=Ok` throughout — re-run on a quiet host passed). Branch is runner-green (workspace tests + clippy `-D warnings`) with the scenario included. Ready to merge on go-ahead.

robocub commented

2026-07-10 17:30:15 +00:00

Author

Collaborator

Shipped. Merged to master (a70dc4c fix + 3868f16 regression scenario, ff), released in v0.3.0-alpha.14 and deployed to prod 2026-07-10 (deb upgrade alpha.13 → alpha.14; service active, all 5 bridges up, 0 errors).

Validation trail:

runner-green (workspace tests + clippy -D warnings) on both commits
live staging: audio_m2d_enc_freshjoin PASS (deterministic frames-before-key ordering; exactly one heal re-subscribe; post-heal oracle mean NCC 0.927) — comment 717
audio_m2d_enc regression PASS (0.871)

The scenario stays in the e2e suite, so any future decryptor-wedge regression fails loudly. Closing.

**Shipped.** Merged to master (`a70dc4c` fix + `3868f16` regression scenario, ff), **released in v0.3.0-alpha.14** and **deployed to prod** 2026-07-10 (deb upgrade alpha.13 → alpha.14; service active, all 5 bridges up, 0 errors). Validation trail: - runner-green (workspace tests + clippy `-D warnings`) on both commits - live staging: `audio_m2d_enc_freshjoin` PASS (deterministic frames-before-key ordering; exactly one heal re-subscribe; post-heal oracle mean NCC 0.927) — comment 717 - `audio_m2d_enc` regression PASS (0.871) The scenario stays in the e2e suite, so any future decryptor-wedge regression fails loudly. Closing.

robocub closed this issue

2026-07-10 17:30:15 +00:00

robocub referenced this issue from a commit

2026-07-11 02:21:37 +00:00

test(e2e): decrypting MatrixListener — true end-to-end oracle for encrypted D->M (#38 D3 Phase D)

robocub referenced this issue

2026-07-11 22:33:36 +00:00

Automatic Testing #38