dark/nether-voicebridge

Fork 1

Stale m.call.member ('waiting for media') after unclean client disconnect — blocked on homeserver MSC4140 (delayed events) #25

New issue

Closed

opened 2026-06-17 23:59:09 +00:00 by robocub · 2 comments

robocub commented

2026-06-17 23:59:09 +00:00

Collaborator

Symptom. A call participant appears stuck as a perpetually-loading tile / "waiting for media from " in Matrix clients (seen in Commet and Element on the Light Voice call), even though they are not transmitting anything.

Diagnosis (live-traced 2026-06-17). The example was @victor:nether.im — a real Matrix user, not a bridge ghost. From the bridge log + the room's m.call.member state:

His client (device nwhe0pUtdu) joined LiveKit at 23:26:22, published audio, then disconnected from the SFU at 23:26:32 (remote participant disconnected / TrackUnpublished … Disconnected).
His m.call.member was never retracted — still active 20+ min later, expires: 14400000 (4h). So other clients keep him in the roster but receive no track → "waiting for media." It only clears at the 4h expiry or on rejoin.

Root cause = no MSC4140 (Delayed Events) on the homeserver. Delayed events let a client arm a server-side "leave" that auto-fires if it stops heartbeating — the crashed/unclean-disconnect cleanup that is missing here. On any unclean exit (crash, killed tab, network drop), the membership lingers ~4h.

This is blocked on the homeserver (Tuwunel), not on a client and not on this bridge:

Verified live: Tuwunel 1.7.1 advertises no org.matrix.msc4140 flag in /_matrix/client/versions. Its MatrixRTC docs (docs/calls/matrix_rtc.md) and issue tracker have no MSC4140 / delayed-events entry — so it is not on a visible roadmap and not imminent. The conduwuit family tracks it unimplemented (sibling fork continuwuity issue #903). MSC4140 itself is still an unstable MSC (PR #4140, not in the stable spec).
Client side is ready on both ends: Element / Element Call via matrix-js-sdk (UNSTABLE_MSC4140_DELAYED_EVENTS); Commet via the famedly matrix-dart-sdk msc_4140_delayed_events library. So we are not waiting on Commet or Element.
Not a bridge bug: the bridge never posts another user's m.call.member and should not reap another user's state. The only fix for what humans see is upstream (homeserver delayed events, the 4h expiry, or the owning client rejoining).

Lever to move it: if we want this fixed, file a feature request on matrix-construct/tuwunel (none exists today) and/or wait for Tuwunel to prioritize Element Call membership reliability.

Downstream effect on the bridge. Because the bridge keys Discord-puppet presence off m.call.member (deliberately, to absorb Element Call's normal ~90s membership flapping — LEAVE_GRACE in crates/matrix-rtc/src/matrix.rs), a stale human membership also keeps a zombie [Matrix] <name> Discord puppet alive until expiry. Root cause is still upstream.

Bridge-side mitigations available if we ever want them (NOT doing now):

Defensive puppet reaping — drop a participant's Discord puppet when their LiveKit media has been gone for a long while despite a lingering membership. Must not regress the LEAVE_GRACE flap-absorption.
Shorter ghost expires + heartbeat refresh — so the bridge's own ghost memberships self-clean faster on a bridge crash without needing delayed events (crates/matrix-rtc/src/rtc/membership.rs currently defaults to the 4h expiry).
Arm MSC4140 for our own ghost/subscriber memberships — also blocked on Tuwunel (the server must honor the delay); revisit when Tuwunel ships it.

Re-check the gate any time: curl -s https://nether.im/_matrix/client/versions → look for org.matrix.msc4140 in unstable_features (absent today). When it appears, revisit mitigation #3.

Filed from a live investigation; tracking-only — no code change yet.

**Symptom.** A call participant appears stuck as a perpetually-loading tile / "waiting for media from <name>" in Matrix clients (seen in Commet and Element on the Light Voice call), even though they are not transmitting anything. **Diagnosis (live-traced 2026-06-17).** The example was `@victor:nether.im` — a **real Matrix user, not a bridge ghost**. From the bridge log + the room's `m.call.member` state: - His client (device `nwhe0pUtdu`) joined LiveKit at 23:26:22, published audio, then **disconnected from the SFU at 23:26:32** (`remote participant disconnected` / `TrackUnpublished … Disconnected`). - **His `m.call.member` was never retracted** — still active 20+ min later, `expires: 14400000` (4h). So other clients keep him in the roster but receive no track → "waiting for media." It only clears at the 4h expiry or on rejoin. **Root cause = no MSC4140 (Delayed Events) on the homeserver.** Delayed events let a client arm a server-side "leave" that auto-fires if it stops heartbeating — the crashed/unclean-disconnect cleanup that is missing here. On any unclean exit (crash, killed tab, network drop), the membership lingers ~4h. **This is blocked on the homeserver (Tuwunel), not on a client and not on this bridge:** - Verified live: **Tuwunel 1.7.1** advertises **no** `org.matrix.msc4140` flag in `/_matrix/client/versions`. Its MatrixRTC docs (`docs/calls/matrix_rtc.md`) and issue tracker have **no** MSC4140 / delayed-events entry — so it is not on a visible roadmap and not imminent. The conduwuit family tracks it unimplemented (sibling fork continuwuity issue #903). MSC4140 itself is still an unstable MSC (PR #4140, not in the stable spec). - **Client side is ready on both ends:** Element / Element Call via matrix-js-sdk (`UNSTABLE_MSC4140_DELAYED_EVENTS`); **Commet** via the famedly matrix-dart-sdk `msc_4140_delayed_events` library. So we are not waiting on Commet or Element. - **Not a bridge bug:** the bridge never posts another user's `m.call.member` and should not reap another user's state. The only fix for what humans see is upstream (homeserver delayed events, the 4h expiry, or the owning client rejoining). **Lever to move it:** if we want this fixed, file a feature request on `matrix-construct/tuwunel` (none exists today) and/or wait for Tuwunel to prioritize Element Call membership reliability. **Downstream effect on the bridge.** Because the bridge keys Discord-puppet presence off `m.call.member` (deliberately, to absorb Element Call's normal ~90s membership flapping — `LEAVE_GRACE` in `crates/matrix-rtc/src/matrix.rs`), a stale *human* membership also keeps a zombie `[Matrix] <name>` Discord puppet alive until expiry. Root cause is still upstream. **Bridge-side mitigations available if we ever want them (NOT doing now):** 1. *Defensive puppet reaping* — drop a participant's Discord puppet when their LiveKit media has been gone for a long while despite a lingering membership. Must not regress the `LEAVE_GRACE` flap-absorption. 2. *Shorter ghost `expires` + heartbeat refresh* — so the bridge's **own** ghost memberships self-clean faster on a bridge crash without needing delayed events (`crates/matrix-rtc/src/rtc/membership.rs` currently defaults to the 4h expiry). 3. *Arm MSC4140 for our own ghost/subscriber memberships* — **also blocked on Tuwunel** (the server must honor the delay); revisit when Tuwunel ships it. **Re-check the gate any time:** `curl -s https://nether.im/_matrix/client/versions` → look for `org.matrix.msc4140` in `unstable_features` (absent today). When it appears, revisit mitigation #3. _Filed from a live investigation; tracking-only — no code change yet._

dark added the

bug

label

2026-06-19 15:27:50 +00:00

shadowdrake commented

2026-06-19 15:47:13 +00:00

Collaborator

I wanted to add some extra context to this one too since I'm familiar with MSC4140 - this isn't a bridge-solvable issue.

Element Call utilizes the scheduled events feature to make clients auto-kick themselves in case of disconnect/crash. It's used as a heartbeat.

The client schedules an event in the near future (10 seconds? I forget the exact duration but it's not long). The scheduled event is that the client leaves the call. While the client is active/alive and has a working connection, once every few seconds (2-5) it updates the scheduled event to delay the event ~10 seconds from current time. This continues perpetually as long as the client is alive.

Once the client stops doing this due to network problems or otherwise, the server eventually executes the action and processes that the dead client has left the room.

The server doesn't do any other processing or tracking of participant health... at least on Synapse. I can't speak to Tuwunel or Continuwuity, etc etc... my understanding is these (largely) don't support MSC4140 at all so all livekit calls will be plagued by phantom participants.

I'm not sure I'd try any mitigations for this as on the bridge they will all have down-sides. But of course, this is just an opinion/voice.

I wanted to add some extra context to this one too since I'm familiar with MSC4140 - this isn't a bridge-solvable issue. Element Call utilizes the scheduled events feature to make clients auto-kick themselves in case of disconnect/crash. It's used as a heartbeat. The client schedules an event in the near future (10 seconds? I forget the exact duration but it's not long). The scheduled event is that the client leaves the call. While the client is active/alive and has a working connection, once every few seconds (2-5) it updates the scheduled event to delay the event ~10 seconds from current time. This continues perpetually as long as the client is alive. Once the client stops doing this due to network problems or otherwise, the server eventually executes the action and processes that the dead client has left the room. The server doesn't do any other processing or tracking of participant health... at least on Synapse. I can't speak to Tuwunel or Continuwuity, etc etc... my understanding is these (largely) don't support MSC4140 at all so all livekit calls will be plagued by phantom participants. I'm not sure I'd try any mitigations for this as on the bridge they will all have down-sides. But of course, this is just an opinion/voice.

robocub referenced this issue from a commit

2026-06-19 21:29:17 +00:00

feat(matrixrtc): reap stale m.call.member phantoms (#25)

robocub referenced this issue

2026-07-05 01:43:24 +00:00

Reaped phantom m.call.member never releases the Discord puppet #54

robocub referenced this issue from a commit

2026-07-05 01:49:35 +00:00

fix(matrixrtc): attribute foreign m.call.member retractions to the member, not the sender (#54)

robocub commented

2026-07-05 15:31:39 +00:00

Author

Collaborator

Closing as mitigated bridge-side; the real fix stays upstream, as @shadowdrake laid out.

What shipped since this was filed:

The stale-membership reaper: expired/orphaned m.call.member entries are reaped, and the #54 fix (v0.3.0-alpha.6) made reaping also release the Discord puppet — live-verified ~75 s from client kill to full release.
#52 max-age session rotation (alpha.7-era, closed with live proof) covers the encrypted-audio side of unclean disconnects.

So a crashed client's tile no longer lingers indefinitely and no longer wedges anything. The proper fix — the client's own MSC4140 delayed-event heartbeat kicking its membership — is Tuwunel's to implement; nothing further is actionable in the bridge. Reopen if a stale tile ever survives the reaper.

Closing as **mitigated bridge-side**; the real fix stays upstream, as @shadowdrake laid out. What shipped since this was filed: - The **stale-membership reaper**: expired/orphaned `m.call.member` entries are reaped, and the #54 fix (v0.3.0-alpha.6) made reaping also release the Discord puppet — live-verified ~75 s from client kill to full release. - **#52 max-age session rotation** (alpha.7-era, closed with live proof) covers the encrypted-audio side of unclean disconnects. So a crashed client's tile no longer lingers indefinitely and no longer wedges anything. The *proper* fix — the client's own MSC4140 delayed-event heartbeat kicking its membership — is Tuwunel's to implement; nothing further is actionable in the bridge. Reopen if a stale tile ever survives the reaper.

robocub closed this issue

2026-07-05 15:31:39 +00:00

robocub referenced this issue from a commit

2026-07-11 03:32:38 +00:00

test(e2e): real-timing reaper + leave-grace debounce scenarios (#38 D3)

robocub referenced this issue from a commit

2026-07-26 04:38:07 +00:00

release: 0.3.1 — one-way-deafness fix; e2e: hold the reaper off the steal scenario's phantoms (#127, #24)

robocub referenced this issue

2026-07-26 04:38:55 +00:00

fix(discord): external displacement during puppet media-startup breaks recovery — rejoin succeeds but playback stays silent (deterministic; was masked by voice-state lag) #127