perf(matrix-rtc): replace per-ghost full crypto Client with a lean send-only OlmMachine (MemoryStore) to retire FD exhaustion #44
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
In an encrypted call, each Discord speaker's ghost spins up a full
matrix_sdk::Clientpurely to deliver its media key — own SQLite crypto store, a bounded initialsync_once, and a perpetual/syncloop (crates/matrix-rtc/src/rtc/ghost.rs:1260,build_ghost_crypto_client; sync spawned at:1322). A 20-speaker encrypted call therefore runs 20 crypto stores, 20 homeserver/synclong-polls, and ~40 long-lived tasks.This is the real cause of the file-descriptor exhaustion seen under load (currently worked around with
LimitNOFILE=65536).SqliteCryptoStoreopens a deadpool connection pool sizedmax(2, physical_cores × 4)— ~32 connections per ghost store on an 8-core box, not the ~3 one might assume — so a couple dozen ghosts blow past the 1024 softnofilelimit, crypto sync dies, and those speakers go inaudible. The perpetual per-ghost sync is also an O(ghosts) long-poll fan-out on the homeserver.Key observation: the ghost is send-only
The ghost crypto client never receives or decrypts anything — its
/syncexists only to keep recipient device-lists fresh, and the sync results are discarded (ghost.rs:1314). Its entire job is to Olm-encrypt one media key to the in-call recipient devices (OlmTransport::send_key,key_transport.rs~211–280; recipients enumerated fromm.call.memberbyenumerate_key_recipients,ghost.rs:1480).Constraint that bounds the fix
We cannot collapse all ghosts onto one shared crypto identity. Element Call attributes a received media key to the authenticated to-device sender (
event.getSender()) and ignores themember.idin the payload — verified in matrix-js-sdk (ToDeviceKeyTransport/RTCEncryptionManager, released tag v41.8.0). So N distinct attributable E2EE participants require N distinct sending identities. We can make each identity cheap; we can't make them one.Proposed change
Replace the per-ghost full
Clientwith a standalonematrix-sdk-crypto::OlmMachinebacked by an in-memoryMemoryStore, driven by a small hand-rolled HTTP loop:outgoing_requests()→ raw/keys/uploadPUT →mark_request_as_sent(). (OTK upload is likely skippable — the ghost is a pure sender, never an Olm recipient; confirm.)update_tracked_users()+ a/keys/querybefore a key send, instead of a perpetual/sync.get_missing_sessions()→/keys/claim.encrypt_content_for_devices()(behind theexperimental-send-custom-to-devicefeature the bridge already enables) → raw/sendToDevicePUT. This is exactly whatencrypt_and_send_raw_to_devicewraps today, so it's a lower-level rewiring of the same operation, not new crypto.Touch points:
build_ghost_crypto_client(ghost.rs:1260) and the Olm path inkey_transport.rs. Thedistribute_key_loopburst+heartbeat policy (ghost.rs:1354) and recipient enumeration are unchanged.What this removes
/synctask → replaced by on-demand/keys/query(removes the O(ghosts) homeserver fan-out).MemoryStoreuses zero, making FD cost independent of ghost count. This is the proper fix;LimitNOFILE=65536is a stopgap.What stays (inherent)
O(ghosts × in-call recipient devices) pairwise Olm sessions per call — bounded by call size, now in-memory and cheap. Same reason EC's own key distribution is O(n²).
Tradeoffs / decisions
MemoryStore's "keys lost on drop" fits — nothing worth persisting. But a fresh in-memory account = a new device per ghost lifecycle → re-key on rejoin. Peer-safe per the #30 device-identity notes, but a deliberate change from today's persistent-store-reused-across-rejoins behaviour. (Decision: in-memory vs. a leaner persistent store.)OlmMachinemeans hand-rolling the HTTP/retry/lock logicClientgave for free (the SDK tutorial warns to lock aroundoutgoing_requeststo avoid duplicate sends). Pin the experimental feature flag.Optional follow-up (separate)
A per-room shared device-tracker: every ghost currently queries the same recipient device-list independently; one shared
/keys/queryper room feeding all ghosts collapses device discovery from O(ghosts) to O(1) per room (Olm sessions stay per-ghost). Track separately if pursued.Acceptance
/sync; device-lists refreshed on demand.Context and full analysis:
docs/nether-voicebridge-public-instance-feasibility.md("Scaling encrypted callers"), and the scaling discussion on #43.Shipped in v0.3.0-alpha.1 (master
00eff03). Each ghost's fullmatrix_sdk::Clientis replaced by a lean send-onlyOlmMachineon aMemoryStore— no per-ghost SQLite crypto store, no per-ghost/syncloop. FD usage is now flat with speaker count (the alpha.1 verification runs showed no per-ghost FD growth), retiring the exhaustion this issue was filed for (theLimitNOFILE=65536unit setting stays as belt-and-braces).One regression was caught during verification and fixed before release: ghosts stopped re-keying a rejoining listener; fixed via the
GhostCryptoCache(respawned ghosts reuse cached crypto + re-send keys).Follow-ups spun out of this rework:
Client(deferred, low priority)Closing.