dark/nether-voicebridge

Fork 1

Reactive ghost re-key on client unwedge via MSC4203 to-device push (blocked upstream on Tuwunel) #55

New issue

Open

opened 2026-07-05 04:10:40 +00:00 by robocub · 5 comments

robocub commented

2026-07-05 04:10:40 +00:00

Collaborator

Summary

When Tuwunel gains MSC4203 (to-device messages pushed to appservices in transactions — Jason has confirmed it's doable; Tuwunel tracking issue to be linked here once filed), the bridge should consume that push and reactively re-key a ghost the moment a client tries to recover a dead Olm session, instead of waiting for the blind periodic rotation (#52) to come around.

Today's behavior (the gap this closes)

#52 gave us blind outbound session rotation: membership-triggered (heals active calls in ≤90s) and max-age (heals static calls in ≤15 min). It works — definitively live-proven 2026-07-05 (#52, comment 458: one speaker inaudible on a static call, healed at the 15-minute rotation, no restart). But it's open-loop: the bridge never sees the client's side of the conversation.

What actually happens during a wedge: the client detects it can't decrypt a ghost's key message, claims fresh one-time keys for the ghost's device, and sends m.dummy to-device to establish a new session. Today those messages land in a /sync inbox no appservice reads. The client did everything right; we're just deaf.

Design sketch

Receive path. Consume to_device from the MSC4203 appservice transaction push for ghost-namespaced recipients. Note: the bridge currently drives everything through per-bridge client /sync — receiving AS transactions means wiring (or adding) the appservice HTTP endpoint (PUT /_matrix/app/v1/transactions/{txnId}) and routing to-device events to the owning ghost's crypto.
Reactive trigger. An inbound m.room.encrypted (Olm) or decrypted m.dummy addressed to a ghost from a call participant's device = that device just re-established a session with us → immediately rotate/re-key that (ghost → device) pair and burst the current media key. Target heal time: seconds. This should bypass the 90s min-interval (it's client-initiated, not flap noise) — rate-limit per (ghost, device) instead.
Prerequisite check — can clients even unwedge against a ghost? Establishing a session toward a ghost requires the ghost's device to have claimable one-time keys. Ghost OlmMachines are send-only with in-memory stores; verify whether they publish OTKs at all, and whether counts get replenished (server-side OTK counts in AS transactions are MSC3202, also missing from Tuwunel — the natural companion). If ghosts publish no OTKs, clients' unwedge attempts fail silently today and item 2 never fires; that may need fixing first and is valuable even before MSC4203 lands.

Dependencies / blocked on

Upstream: Tuwunel MSC4203 implementation (to_device: Vec::new(), // TODO in send_events_dest_appservice, src/service/sending/sender.rs). MSC3202 for OTK counts is the companion.
Item 3 (ghost OTK publication audit) is not blocked and can be done now.

References

#52 — rotation mechanism, corrected research (rust-sdk #3427, unwedge throttle semantics, MSC4203/MSC3202/MSC4190 status on Tuwunel), and the live wedge-heal proof (comment 458).
element-hq/element-meta#2356 — senders never re-session on their own; recovery is always receiver-initiated, which is exactly the signal this issue makes visible.

## Summary When Tuwunel gains MSC4203 (to-device messages pushed to appservices in transactions — Jason has confirmed it's doable; Tuwunel tracking issue to be linked here once filed), the bridge should consume that push and **reactively re-key a ghost the moment a client tries to recover a dead Olm session**, instead of waiting for the blind periodic rotation (#52) to come around. ## Today's behavior (the gap this closes) #52 gave us blind outbound session rotation: membership-triggered (heals active calls in ≤90s) and max-age (heals static calls in ≤15 min). It works — definitively live-proven 2026-07-05 (#52, comment 458: one speaker inaudible on a static call, healed at the 15-minute rotation, no restart). But it's open-loop: the bridge never *sees* the client's side of the conversation. What actually happens during a wedge: the client detects it can't decrypt a ghost's key message, claims fresh one-time keys for the ghost's device, and sends `m.dummy` to-device to establish a new session. Today those messages land in a `/sync` inbox no appservice reads. The client did everything right; we're just deaf. ## Design sketch 1. **Receive path.** Consume `to_device` from the MSC4203 appservice transaction push for ghost-namespaced recipients. Note: the bridge currently drives everything through per-bridge client `/sync` — receiving AS transactions means wiring (or adding) the appservice HTTP endpoint (`PUT /_matrix/app/v1/transactions/{txnId}`) and routing to-device events to the owning ghost's crypto. 2. **Reactive trigger.** An inbound `m.room.encrypted` (Olm) or decrypted `m.dummy` addressed to a ghost from a call participant's device = that device just re-established a session with us → immediately rotate/re-key that (ghost → device) pair and burst the current media key. Target heal time: seconds. This should bypass the 90s min-interval (it's client-initiated, not flap noise) — rate-limit per (ghost, device) instead. 3. **Prerequisite check — can clients even unwedge against a ghost?** Establishing a session *toward* a ghost requires the ghost's device to have claimable one-time keys. Ghost OlmMachines are send-only with in-memory stores; verify whether they publish OTKs at all, and whether counts get replenished (server-side OTK counts in AS transactions are MSC3202, also missing from Tuwunel — the natural companion). If ghosts publish no OTKs, clients' unwedge attempts fail silently today and item 2 never fires; that may need fixing first and is valuable even before MSC4203 lands. ## Dependencies / blocked on - **Upstream:** Tuwunel MSC4203 implementation (`to_device: Vec::new(), // TODO` in `send_events_dest_appservice`, `src/service/sending/sender.rs`). MSC3202 for OTK counts is the companion. - Item 3 (ghost OTK publication audit) is **not blocked** and can be done now. ## References - #52 — rotation mechanism, corrected research (rust-sdk #3427, unwedge throttle semantics, MSC4203/MSC3202/MSC4190 status on Tuwunel), and the live wedge-heal proof (comment 458). - element-hq/element-meta#2356 — senders never re-session on their own; recovery is always receiver-initiated, which is exactly the signal this issue makes visible.

robocub referenced this issue

2026-07-05 15:31:36 +00:00

perf(matrix-rtc): replace per-ghost full crypto Client with a lean send-only OlmMachine (MemoryStore) to retire FD exhaustion #44

robocub referenced this issue

2026-07-05 15:31:38 +00:00

Slow/uneven media-key re-delivery when a Matrix listener rejoins an encrypted call (follow-up to #44) #51

robocub commented

2026-07-05 16:18:14 +00:00

Author

Collaborator

OTK-publication audit (the NOT-blocked sub-item) — done 2026-07-05

Question: do the send-only ghost OlmMachines (#44) publish one-time keys at all, so a client's unwedge-claim toward a ghost device can succeed?

Answer: yes at bootstrap, and the pool is fully restored on every process restart and every #52 rotation — but it is never replenished within a run, cache-hit respawns don't re-upload, and no fallback key is ever published. So the pool is a finite ~50 that drains between restore events, with no backstop.

Code facts (`crates/matrix-rtc/src/rtc/key_transport.rs`, matrix-sdk-crypto 0.18)

Account::new pre-generates max OTKs, and the bootstrap drain_outgoing (:408) uploads them — dispatch (:711) handles KeysUpload/KeysQuery/KeysClaim/ToDeviceRequest.
The rotation pickle (account_pickle_json) is serialized before the initial upload (:375), so the pickled OTKs stay flagged unpublished forever → every rotate_sessions rebuild (:467-495) and every restart re-uploads the same original key ids.
No in-run replenishment: ghosts have no /sync, so the only count feedback is the upload echo (= max) — generate_one_time_keys_if_needed never re-fires, and there is no periodic pump (drains happen only at bootstrap, rotation, and per key-send).
No fallback key, ever: fallback generation is only reachable via receive_sync_changes, which a send-only machine never calls; the fallback_keys map in our uploads is always empty.
GhostCryptoCache hit = no re-upload: a ghost respawn inside one process reuses the cached machine and skips bootstrap (ghost.rs:1287-1299).
The comment at key_transport.rs:404 ("OTKs are not strictly needed — the ghost is a pure sender") is stale under this issue: an unwedging client claims the ghost's OTK, so the ghost is an Olm responder for recovery purposes.

Live verification (nether.im, prod, 2026-07-05 ~07:00 UTC)

A fresh ghost device (uploaded 06:23) had claimable OTKs — and real clients had already consumed ~3 ids within 40 minutes. The standing drain is megolm room-key sharing: every client sharing a chat room key Olm-sessions all devices in the encrypted room, ghosts included. This runs 24/7 in a chatty room, independent of calls.
A reused device (2 days old, process restart 05:15) served claims sequentially from key id 0 — proof the restart re-upload restored the full original batch including previously-claimed ids (Tuwunel re-adds re-uploaded ids). Same batch every time, per the pickle behavior above.
Every claim returned fallback: false, as predicted.
Side observation: one ghost MXID has 4 accumulated crypto devices (prune soft-fails on this Tuwunel build, as documented in #53). The stale devices' pools are permanently empty, so clients' room-key shares fail claims against them forever — client-side noise/backoff, no audio impact.
(The probes consumed ~8 OTKs across two devices — harmless; pools restore on the next restart/rotation.)

Severity

Today this is latent: an empty ghost pool does not break audio (media-key delivery is ghost→client outbound; the ghost claims client keys, not vice versa), and pre-MSC4203 the ghost can't receive the unwedge m.dummy anyway. What it costs today is failed claims/backoff noise on clients. Post-MSC4203 it becomes load-bearing: an unwedging client that hits an empty pool (long uptime + chatty room + no recent call ⇒ no rotations) fails silently — exactly the gap this issue exists to close.

Recommended fix (small, worth doing with — or before — the receive path)

Publish a fallback key at bootstrap and on rotation. One fallback key answers unlimited claims, which kills the drain problem outright: feed the machine a synthetic receive_sync_changes with unused_fallback_key_types = Some(vec![]) (no fallback in use) right after construction — the machine then generates and bundles a fallback key into the next /keys/upload that drain_outgoing already sends. ~10 lines in bootstrap + rotate_sessions, plus fixing the stale :404 comment. Optionally the same call can carry a synthetic low OTK count to force regeneration, but with a fallback key that's belt-and-braces.

## OTK-publication audit (the NOT-blocked sub-item) — done 2026-07-05 **Question:** do the send-only ghost OlmMachines (#44) publish one-time keys at all, so a client's unwedge-claim toward a ghost device can succeed? **Answer: yes at bootstrap, and the pool is fully restored on every process restart and every #52 rotation — but it is never replenished within a run, cache-hit respawns don't re-upload, and no fallback key is ever published.** So the pool is a finite ~50 that drains between restore events, with no backstop. ### Code facts (`crates/matrix-rtc/src/rtc/key_transport.rs`, matrix-sdk-crypto 0.18) - `Account::new` pre-generates max OTKs, and the bootstrap `drain_outgoing` (`:408`) uploads them — `dispatch` (`:711`) handles `KeysUpload`/`KeysQuery`/`KeysClaim`/`ToDeviceRequest`. - The rotation pickle (`account_pickle_json`) is serialized **before** the initial upload (`:375`), so the pickled OTKs stay flagged unpublished forever → every `rotate_sessions` rebuild (`:467-495`) and every restart re-uploads the **same original key ids**. - **No in-run replenishment:** ghosts have no `/sync`, so the only count feedback is the upload echo (= max) — `generate_one_time_keys_if_needed` never re-fires, and there is no periodic pump (drains happen only at bootstrap, rotation, and per key-send). - **No fallback key, ever:** fallback generation is only reachable via `receive_sync_changes`, which a send-only machine never calls; the `fallback_keys` map in our uploads is always empty. - **`GhostCryptoCache` hit = no re-upload:** a ghost respawn inside one process reuses the cached machine and skips bootstrap (`ghost.rs:1287-1299`). - The comment at `key_transport.rs:404` ("OTKs are not strictly needed — the ghost is a pure sender") is stale under this issue: an unwedging client claims the *ghost's* OTK, so the ghost is an Olm responder for recovery purposes. ### Live verification (nether.im, prod, 2026-07-05 ~07:00 UTC) - A **fresh** ghost device (uploaded 06:23) had claimable OTKs — and real clients had already consumed ~3 ids within 40 minutes. The standing drain is megolm room-key sharing: every client sharing a chat room key Olm-sessions **all** devices in the encrypted room, ghosts included. This runs 24/7 in a chatty room, independent of calls. - A **reused** device (2 days old, process restart 05:15) served claims sequentially **from key id 0** — proof the restart re-upload restored the full original batch *including previously-claimed ids* (Tuwunel re-adds re-uploaded ids). Same batch every time, per the pickle behavior above. - Every claim returned `fallback: false`, as predicted. - Side observation: one ghost MXID has **4 accumulated crypto devices** (prune soft-fails on this Tuwunel build, as documented in #53). The stale devices' pools are permanently empty, so clients' room-key shares fail claims against them forever — client-side noise/backoff, no audio impact. - (The probes consumed ~8 OTKs across two devices — harmless; pools restore on the next restart/rotation.) ### Severity Today this is **latent**: an empty ghost pool does not break audio (media-key delivery is ghost→client outbound; the ghost claims *client* keys, not vice versa), and pre-MSC4203 the ghost can't receive the unwedge `m.dummy` anyway. What it costs today is failed claims/backoff noise on clients. **Post-MSC4203 it becomes load-bearing**: an unwedging client that hits an empty pool (long uptime + chatty room + no recent call ⇒ no rotations) fails silently — exactly the gap this issue exists to close. ### Recommended fix (small, worth doing with — or before — the receive path) **Publish a fallback key at bootstrap and on rotation.** One fallback key answers unlimited claims, which kills the drain problem outright: feed the machine a synthetic `receive_sync_changes` with `unused_fallback_key_types = Some(vec![])` (no fallback in use) right after construction — the machine then generates and bundles a fallback key into the next `/keys/upload` that `drain_outgoing` already sends. ~10 lines in `bootstrap` + `rotate_sessions`, plus fixing the stale `:404` comment. Optionally the same call can carry a synthetic low OTK count to force regeneration, but with a fallback key that's belt-and-braces.

robocub referenced this issue from a commit

2026-07-05 16:29:13 +00:00

feat(matrix-rtc): publish a ghost fallback key at bootstrap + rotation (#55)

robocub commented

2026-07-05 16:29:31 +00:00

Author

Collaborator

Fallback-key fix from the audit above is built: branch feat/issue-55-ghost-fallback-key (9e43518) — prime_fallback_key feeds the machine a synthetic "no unused fallback keys" sync signal after construction in both bootstrap and rotate_sessions, so the existing drain_outgoing uploads a fallback key that answers unlimited claims; priming failure is non-fatal (falls back to pre-fix OTK-only behavior). Unit test pins that the next /keys/upload carries the fallback key without disturbing the OTK batch; clippy + full workspace green.

Live acceptance when it ships: after a deploy, drain-test a ghost device with repeated /keys/claim — past the ~50 OTKs it should keep answering with the same fallback: true key instead of going empty.

Not merged yet (gated on live testing as usual). The MSC4203 receive path remains this issue's blocked main item.

Fallback-key fix from the audit above is **built**: branch `feat/issue-55-ghost-fallback-key` (`9e43518`) — `prime_fallback_key` feeds the machine a synthetic "no unused fallback keys" sync signal after construction in both `bootstrap` and `rotate_sessions`, so the existing `drain_outgoing` uploads a fallback key that answers unlimited claims; priming failure is non-fatal (falls back to pre-fix OTK-only behavior). Unit test pins that the next `/keys/upload` carries the fallback key without disturbing the OTK batch; clippy + full workspace green. Live acceptance when it ships: after a deploy, drain-test a ghost device with repeated `/keys/claim` — past the ~50 OTKs it should keep answering with the same `fallback: true` key instead of going empty. Not merged yet (gated on live testing as usual). The MSC4203 receive path remains this issue's blocked main item.

robocub commented

2026-07-05 16:50:29 +00:00

Author

Collaborator

Live acceptance PASSED (staging on the prod VPS, 2026-07-05 ~16:48 UTC, prod stopped for the window and restored after): fresh ghost crypto device Cpntst58y on the encrypted Purple Voice bridge, drain-probed via /keys/claim — served exactly 50 normal OTKs, then every further claim returned the same fallback: true key (signed_curve25519:AAAAAAAAAAA), verified stable across repeated claims. Zero failed to prime warnings, zero errors in the staging run. Pre-fix behavior at claim 51 was a permanently empty response.

Branch feat/issue-55-ghost-fallback-key (9e43518) is ready to merge.

**Live acceptance PASSED** (staging on the prod VPS, 2026-07-05 ~16:48 UTC, prod stopped for the window and restored after): fresh ghost crypto device `Cpntst58y` on the encrypted Purple Voice bridge, drain-probed via `/keys/claim` — served exactly **50 normal OTKs**, then every further claim returned the **same `fallback: true` key** (`signed_curve25519:AAAAAAAAAAA`), verified stable across repeated claims. Zero `failed to prime` warnings, zero errors in the staging run. Pre-fix behavior at claim 51 was a permanently empty response. Branch `feat/issue-55-ghost-fallback-key` (`9e43518`) is ready to merge.

dark commented

2026-07-07 15:33:48 +00:00

Owner

Two issues opened on tuwunel's github:

Two issues opened on tuwunel's github: 1. https://github.com/matrix-construct/tuwunel/issues/501 2. https://github.com/matrix-construct/tuwunel/issues/502

robocub commented

2026-07-19 21:43:59 +00:00

Author

Collaborator

Unblocked upstream by Tuwunel v1.8.2 (released 2026-07-17): appservice transaction extensions now deliver to-device events (MSC4203) — plus device-list changes and one-time-key counts (MSC3202) — to appservices. That's the exact capability this was waiting on.

Prod (nether.im) is already running 1.8.2, so the mechanism is live for us now. Re-scoping this from "blocked upstream" → ready to implement: consume the to-device push and reactively re-key a ghost the moment a client tries to recover a dead Olm session, instead of waiting for the blind #52 periodic rotation.

Release: https://github.com/matrix-construct/tuwunel/releases/tag/v1.8.2 — suggest retitling to drop "(blocked upstream on Tuwunel)". Implementation is separate, larger work.

**Unblocked upstream by Tuwunel v1.8.2** (released 2026-07-17): appservice transaction extensions now deliver **to-device events (MSC4203)** — plus device-list changes and one-time-key counts (MSC3202) — to appservices. That's the exact capability this was waiting on. **Prod (`nether.im`) is already running 1.8.2**, so the mechanism is live for us now. Re-scoping this from "blocked upstream" → ready to implement: consume the to-device push and reactively re-key a ghost the moment a client tries to recover a dead Olm session, instead of waiting for the blind #52 periodic rotation. Release: https://github.com/matrix-construct/tuwunel/releases/tag/v1.8.2 — suggest retitling to drop "(blocked upstream on Tuwunel)". Implementation is separate, larger work.