Ghost to-device receive path / Olm wedge self-heal #52
Labels
No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
dark/nether-voicebridge#52
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Give ghosts a to-device receive path so their Olm sessions with listeners can
self-heal when they wedge. Today ghosts are strictly send-only (#44), which means a
wedged ghost→listener Olm session can never recover on its own → that Discord speaker is
permanently silent for that listener until the ghost's device changes (respawn/restart).
Deferred by design (2026-07-02): the immediate root cause of observed wedging is stale
device accumulation (#34) + device-id churn (#35); hygiene removes the dominant trigger
while keeping ghosts light. This issue tracks the residual recovery mechanism — build it
only if wedging still occurs after hygiene lands.
Background: how a ghost session wedges
A 1:1 Olm session starts when the sender claims one of the receiver's single-use one-time
keys (OTKs) and sends a pre-key message. A send-only ghost never receives a reply, so
it keeps re-sending pre-key messages reusing the same OTK forever. If the listener ever
fails to build its inbound side (OTK already consumed/discarded, e.g. under OTK exhaustion
or a stale-device signature failure), it hits
MissingOneTimeKeyand can never rebuild— every retry references the now-gone OTK. Element Call then marks the frame key invalid and
drops that publisher's audio.
Live-proven from a listener's Element Desktop crypto console (2026-07-02): repeated
Failed to create a new Olm session from a pre-key message: MissingOneTimeKey(...),chain_indexclimbing 14→15→16, for one specific ghost while others were fine. Bridge logsshowed
delivered=1(the Olm PUT succeeded) — misleading; the listener couldn't decrypt it.Why send-only can't self-heal
Matrix's standard recovery ("m.dummy" unwedge) is receiver-initiated: the receiver
builds a new session to the sender and sends an
m.dummy; the sender adopts it (newestsession wins for outbound) — but only if the sender receives it. A send-only ghost never
does. Two hard limits (matrix-sdk-crypto 0.18 + spec):
pub(crate)),so we can't rebuild a wedged session in place.
MissingOneTimeKey(gated on an existing >1h-old session — matrix-rust-sdk#3110). So a receive path repairs
the later (>1h existing-session) wedge class, not the first-contact one (that's what
hygiene #34 prevents).
Proposed architecture (light, for the hosted multi-tenant goal)
The receiver is
OlmMachine::receive_sync_changes(public; feed it to-device events +device-list/OTK counts). It auto-adopts a peer's new session for outbound. The question is
how to deliver to-device events to N in-memory ghost machines cheaply.
Preferred: single shared appservice to-device push (MSC2409). One HS→bridge stream for
ALL ghosts; demux each to-device event by target
device_id→ owningGhostCrypto→receive_sync_changes. Ghost cost stays O(1) in connections. Not buildable as-is — thebridge has no inbound HTTP server, registration
urlisnull, nohs_tokenhandling,matrix-sdkis built without appservice support, and Tuwunel's MSC2409 support isunconfirmed. Work: add an inbound HTTP server (new dep), wire
url+hs_token+thede.sorunome.msc2409.push_ephemeralregistration flag, and the device→ghost demux. Firststep: confirm Tuwunel pushes to-device to appservices.
Fallback: per-ghost filtered
/sync(to-device only, timeline disabled) on each ghost'sexisting token. Trivial to add (reuses the outbound reqwest pattern) but re-imposes the
O(ghosts)
/syncfan-out #44 deleted — wrong for a light, multi-tenant service. Onlyconsider for small/self-hosted deployments.
Also on the table: send-only self-heal by periodic crypto-device rotation (fresh
device = fresh session; #34 prunes the old one) — keeps ghosts strictly send-only but is
blunt (re-establishes everything) and carries the #44 rejoin-churn risk.
Gate
Land device hygiene (#34) + stable device (#35) first, then measure whether wedging
still occurs. Build the appservice-push receive path only if it does. Related: #34, #35, #44.
Clean live reproduction tonight (2026-07-04), severity upgrade candidate.
During #53 live-testing the bridge process was restarted several times (each restart drops all ghosts' in-memory Olm sessions and establishes fresh ones on the next key send). Afterward, on the prod alpha.2 service, the listener (@dark, device
ihjNlBVX90) could hear five of six Discord speakers — everyone except Ruffle's ghost (@nether-voicebridge_941211140774563870, crypto deviceCwbc3t2wx, stable identity minted that evening).Bridge side was proven clean end-to-end via the #51 fingerprint methodology:
generated == frame-encrypt == sentkey fp (b20e23b0) for every heartbeat,transport=olm delivered=1, zero warnings, no unreachable devices.⇒ the listener's client held a dead inbound Olm session for exactly this one ghost and could not decrypt its key messages;
delivered=1cannot see that. There is no recovery: the ghost's send-only OlmMachine reuses its established outbound session indefinitely (a dead peer session is undetectable to it), and the client has no key-request path for to-device Olm.Remediation that confirmed the diagnosis: restarting the bridge (fresh MemoryStore ⇒ fresh outbound session ⇒ new one-time-key claim) made Ruffle audible immediately (new key fp
c2a7158c).Design note for the fix: any periodic/opportunistic outbound-session rotation (e.g. rotate the Olm session on every media-key ratchet, or after N sends without any observable client activity) would bound the wedge duration without needing a receive path.
Implemented on branch
feat/issue-52-session-rotation(tipf26fa2e; unit-tested, NOT yet live-verified or merged): opportunistic outbound-session rotation.Since matrix-sdk-crypto 0.18 has no public session-discard (store mutators are
pub(crate), and the machine's internal session cache masks store-level deletion), rotation = rebuild the ghost'sOlmMachinefrom its persisted account pickle with a freshMemoryStore— exactly what a bridge restart does, which is the remediation that fixed the 2026-07-04 Ruffle wedge instantly. Same device id + identity keys (noSigningKeyChanged; the re-uploaded device keys are byte-identical), all sessions dropped, next send claims fresh OTKs.Triggers: each membership change (rate-limited to 1/90s against EC membership flaps) + a 15-min max-age checked on the key heartbeat. Expected worst-case one-speaker silence: ~90s in an active call (a listener's rejoin now actually heals them), ~16 min in a fully static call — vs unbounded today. Config:
rotate_ghost_sessions(bridge/server profile, default on, Olm transport only). Ghosts stay send-only, zero FDs (#44 preserved).Research corrections to this issue's framing (verified 2026-07-04 from matrix-sdk-crypto 0.18 source, upstream trackers, and Tuwunel source):
creation_time, ignores MSC1719), not #3110 (an unrelated multi-process cache bug, fixed 2024).UNWEDGING_INTERVALis a throttle, not a wait. A listener unwedges immediately on failed decrypt if its newest stored session with the device is >1h old; it's blocked only for young pairs (<1h), and never fires with no stored session (#3427 — plausibly the exact Ruffle case: that ghost's crypto identity was freshly minted the same evening). Element Web/Desktop runs exactly this code (Rust crypto WASM; legacy removed matrix-js-sdk v37, 2025-02).m.dummy— into the void. Our wedge persists because a send-only ghost discards the recovery message, not because of the gates. Sender-side never re-sessions on its own (element-meta#2356).to_device: Vec::new(), // TODO(src/service/sending/sender.rs:873, current git 2026-07-02); its spec-compliance campaign targets the published spec, so MSC4203 won't arrive as a side effect. MSC3202's transaction extensions (OTK counts / device-list changes) are also missing. The MSC2409 "70/70 partial" on Tuwunel's compliance page is the EDU half only (typing+receipts; not presence, not to-device).Live verification on prod (2026-07-05, alpha.6 running): PASSED — closing.
Rotation fired during real encrypted calls on the Purple Voice bridge, exactly as designed:
reason="membership", logged asrotated Olm outbound sessions (fresh machine, same device identity). Same-device-identity means noSigningKeyChangedexposure to peers.delivered=1to the listener's live device, burst 6→1 counting down normally, then the 60s heartbeat). Zero WARN/ERROR in either window; the only WARN anywhere is the normalClientInitiatedLiveKit disconnect at call end.Not yet observed in the wild: the max-age (15 min) trigger — all encrypted calls since deploy were shorter than 15 minutes, so it legitimately never armed. The path shares the same rotate mechanism as the membership trigger (only the scheduling differs) and is covered by the rotation-policy unit tests; it will self-evidence in the journal during the next long encrypted call.
The membership trigger is the fast path that heals an active-call wedge (the Ruffle incident class) within ~90s of any join/leave, and it is now proven live end-to-end. The full receive-path self-heal remains blocked on Tuwunel gaining MSC4203 (to-device appservice push) — tracked in the issue body's research notes.
Definitive live confirmation, ~1 hour after closing — a real wedge, healed by max-age rotation, witnessed end-to-end.
During a static encrypted call on Purple Voice (2026-07-05 ~03:37–03:53 UTC), the listener reported one Discord speaker (Spooki) inaudible while every other participant was fine — the classic single-ghost wedge:
capture_errs=0, live RMS levels, and his ghost's keydelivered=1to the listener's device on every heartbeat.delivered=1 ≠ decryptable— dead inbound Olm session on the client, exactly the Ruffle incident class.reason="max-age", keys re-delivered on the fresh sessions ~200ms later.Also observed in the same call: a new Discord speaker joined at ~03:50 and its ghost's membership-triggered rotation fired at 03:52:38 — so both trigger paths are now proven against live traffic, and the max-age path is proven against a real wedge, not just a clean rotation.
This closes the last open verification item. The static-call worst-case heal time (~15–16 min) behaved exactly as designed; anyone hitting it sooner can leave+rejoin the call (membership rotation heals in seconds). Receive-path reactive healing remains blocked on Tuwunel MSC4203.