Encrypted room: no ghosts and no puppet on a fresh-store deploy (unencrypted works) #30

New issue

Closed

opened 2026-06-19 03:35:32 +00:00 by robocub · 1 comment

robocub commented

2026-06-19 03:35:32 +00:00

Collaborator

Symptom

On a fresh-store deploy (v0.2.0-rc4 .deb on the VPS, systemd service, /var/lib/nether-voicebridge wiped before first start), the encrypted bridge does not function in either direction:

A Matrix user who joins the encrypted call sees no ghost users (Discord speakers are not represented).
Discord users do not see the Matrix user's puppet bot (the Matrix→Discord side is dead too).

The unencrypted bridge (Light Voice) works normally on the same process — ghosts are assigned, post m.call.member, mirror mute, etc.

Important context

This is after the fresh-store E2EE auto-detection fix (a6efd71, shipped in rc4), which made detect_room_encryption authoritative via Room::latest_encryption_state(). So detection is no longer the problem — the room is now correctly detected as encrypted, but the encrypted media/membership path itself fails to come up on a fresh store.

Pattern mirrors the detection bug it followed: the encrypted bridge works locally with an established store (verified earlier the same day) but fails on a freshly-provisioned store. Strongly suggests an encrypted-path bootstrap step that silently depends on pre-existing crypto/store state and doesn't establish itself on first run.

Environment

v0.2.0-rc4 .deb, installed on VPS, run as the nether-voicebridge systemd service.
/var/lib/nether-voicebridge wiped before start → genuine fresh crypto store (new per-bridge devices, new per-ghost crypto clients).
Encrypted room reached via the appservice path (@voicebridge_as, self-provisioned device).

Evidence so far

Only unencrypted (Light Voice) logs were captured, and they are healthy:

bridge{name=Light Voice}:ghost{...}: ghost assigned key="504045051567931396" display_name="Lokilao"
bridge{name=Light Voice}:ghost{...}: ghost present (membership posting; no media until a Matrix user joins the call)
bridge{name=Light Voice}:ghost{...}: ghost call membership posted — visible in call
bridge{name=Light Voice}:ghost{...}: ghost track mute mirrored from Discord muted=false

Dark Voice (encrypted) logs were NOT yet captured — that's the first task.

Next steps (tomorrow)

Collect the encrypted bridge's logs from the VPS:
```
journalctl -u nether-voicebridge | grep -iE "Dark Voice|resolved media E2EE|encryption state|m.call.member|membership|ghost|key|olm|MissingKey|recipient|unreachable|error|warn"
```
Confirm: (a) Dark resolved detected_encrypted=true … Olm; (b) whether the subscriber posts its own m.call.member (it MUST be a visible member in E2EE); (c) whether ghosts post membership in the encrypted room; (d) any crypto/Olm/SQLite-store errors during fresh-device provisioning; (e) key-distribution warnings (reached NO recipient devices, unreachable).
Reproduce locally with a FRESH store (NVB_DATA_DIR=$(mktemp -d)) and actually join the encrypted call. The earlier fresh-store test only checked the detection log line, not live call participation — so the bug likely reproduces locally this way. (Do NOT run a local instance while the VPS instance is live — both would post ghosts/subscriber to the same rooms and conflict.)
Hypotheses to check (see also the encrypted-room E2EE findings in repo history / memory):
- Subscriber and/or ghost m.call.member not posting on a fresh crypto device → invisible in Element Call (EC builds its roster from membership, not LiveKit presence).
- Per-ghost Olm appservice-login / crypto-client provisioning failing or racing on a fresh store (each ghost becomes its own crypto client).
- Membership self-heal-retry not firing on first run.
- Room power level for org.matrix.msc3401.call.member not at 0 on the VPS's encrypted room (verify the RAW m.room.power_levels — Element's friendly UI label is unreliable). A wrong PL silently blocks ghost/subscriber membership → exactly this symptom.

Severity

Blocks encrypted-room bridging on any fresh/new deployment — i.e., every new install. Gates the final v0.2.0 release. Unencrypted bridging is unaffected.

## Symptom On a **fresh-store** deploy (v0.2.0-rc4 `.deb` on the VPS, systemd service, `/var/lib/nether-voicebridge` wiped before first start), the **encrypted** bridge does not function in either direction: - A Matrix user who joins the encrypted call sees **no ghost users** (Discord speakers are not represented). - Discord users do **not** see the Matrix user's puppet bot (the Matrix→Discord side is dead too). The **unencrypted** bridge (Light Voice) works normally on the same process — ghosts are assigned, post `m.call.member`, mirror mute, etc. ## Important context This is **after** the fresh-store E2EE auto-detection fix (`a6efd71`, shipped in rc4), which made `detect_room_encryption` authoritative via `Room::latest_encryption_state()`. So detection is no longer the problem — the room is now correctly detected as encrypted, but the **encrypted media/membership path itself fails to come up on a fresh store**. Pattern mirrors the detection bug it followed: the encrypted bridge **works locally with an established store** (verified earlier the same day) but fails on a freshly-provisioned store. Strongly suggests an encrypted-path bootstrap step that silently depends on pre-existing crypto/store state and doesn't establish itself on first run. ## Environment - v0.2.0-rc4 `.deb`, installed on VPS, run as the `nether-voicebridge` systemd service. - `/var/lib/nether-voicebridge` wiped before start → genuine fresh crypto store (new per-bridge devices, new per-ghost crypto clients). - Encrypted room reached via the appservice path (`@voicebridge_as`, self-provisioned device). ## Evidence so far Only **unencrypted (Light Voice)** logs were captured, and they are healthy: ``` bridge{name=Light Voice}:ghost{...}: ghost assigned key="504045051567931396" display_name="Lokilao" bridge{name=Light Voice}:ghost{...}: ghost present (membership posting; no media until a Matrix user joins the call) bridge{name=Light Voice}:ghost{...}: ghost call membership posted — visible in call bridge{name=Light Voice}:ghost{...}: ghost track mute mirrored from Discord muted=false ``` **Dark Voice (encrypted) logs were NOT yet captured** — that's the first task. ## Next steps (tomorrow) 1. **Collect the encrypted bridge's logs from the VPS:** ``` journalctl -u nether-voicebridge | grep -iE "Dark Voice|resolved media E2EE|encryption state|m.call.member|membership|ghost|key|olm|MissingKey|recipient|unreachable|error|warn" ``` Confirm: (a) Dark resolved `detected_encrypted=true … Olm`; (b) whether the **subscriber** posts its own `m.call.member` (it MUST be a visible member in E2EE); (c) whether **ghosts** post membership in the encrypted room; (d) any crypto/Olm/SQLite-store errors during fresh-device provisioning; (e) key-distribution warnings (`reached NO recipient devices`, `unreachable`). 2. **Reproduce locally with a FRESH store** (`NVB_DATA_DIR=$(mktemp -d)`) and actually join the encrypted call. The earlier fresh-store test only checked the *detection log line*, not live call participation — so the bug likely reproduces locally this way. (Do NOT run a local instance while the VPS instance is live — both would post ghosts/subscriber to the same rooms and conflict.) 3. **Hypotheses to check** (see also the encrypted-room E2EE findings in repo history / memory): - Subscriber and/or ghost `m.call.member` not posting on a **fresh crypto device** → invisible in Element Call (EC builds its roster from membership, not LiveKit presence). - Per-ghost **Olm** appservice-login / crypto-client provisioning failing or racing on a fresh store (each ghost becomes its own crypto client). - Membership **self-heal-retry** not firing on first run. - Room power level for `org.matrix.msc3401.call.member` not at **0** on the VPS's encrypted room (verify the RAW `m.room.power_levels` — Element's friendly UI label is unreliable). A wrong PL silently blocks ghost/subscriber membership → exactly this symptom. ## Severity Blocks encrypted-room bridging on any fresh/new deployment — i.e., every new install. Gates the final v0.2.0 release. Unencrypted bridging is unaffected.

dark added the

bug

label

2026-06-19 15:28:19 +00:00

robocub referenced this issue from a commit

2026-06-19 16:58:00 +00:00

fix(e2ee): pair Matrix device identity with the crypto store (#30)

robocub commented

2026-06-19 16:58:54 +00:00

Author

Collaborator

Root cause found and fixed (live-verified on Tuwunel; commit 84b0a0c).

The "no ghosts / no puppet" framing was partly off: on a fresh store the encrypted bridge comes up fine and Discord→Matrix works. The real failure is receive-only — Discord users can't hear Matrix speakers.

The bot's device_id is derived from the bridge name, not the crypto store, so a wiped store re-registers the same device_id with new identity keys. Peers that cached the old identity reject the change (SigningKeyChanged) and never deliver the per-call media key (io.element.call.encryption_keys) to the subscriber → it can't decrypt the human's frames (LiveKit E2eeState stuck at MissingKey → −96 dB silence). The send direction works because there the bridge encrypts to the human's own stable devices — hence "I can hear them, they can't hear me."

Fix (84b0a0c): pair the device identity with the store — persist a <store_path>/device_id marker. A genuinely fresh store mints a unique id (peers see a brand-new device and open Olm normally); a pre-existing store adopts its configured id (back-compat, no re-key); a marker present is reused on restart. Appservice path only (the access_token path keeps its token-pinned device).

Verified live: fresh store → @dark track MissingKey→Ok, media key received, clear audio both directions; warm store adopts dark-voice unchanged. Build + new resolver unit tests + clippy all green.

Non-blocking follow-ups (candidates for separate issues): the superseded device lingers on the homeserver after each wipe (device-list growth, cosmetic); the per-ghost crypto clients use the same reuse pattern (harmless — ghosts are senders — but it's the source of the SigningKeyChanged log churn).

**Root cause found and fixed** (live-verified on Tuwunel; commit `84b0a0c`). The "no ghosts / no puppet" framing was partly off: on a fresh store the encrypted bridge comes up fine and **Discord→Matrix works**. The real failure is **receive-only** — Discord users can't hear Matrix speakers. The bot's `device_id` is derived from the bridge **name**, not the crypto store, so a wiped store re-registers the **same** `device_id` with **new** identity keys. Peers that cached the old identity reject the change (`SigningKeyChanged`) and never deliver the per-call media key (`io.element.call.encryption_keys`) to the subscriber → it can't decrypt the human's frames (LiveKit `E2eeState` stuck at `MissingKey` → −96 dB silence). The send direction works because there the bridge encrypts to the human's **own** stable devices — hence "I can hear them, they can't hear me." **Fix (`84b0a0c`):** pair the device identity with the store — persist a `<store_path>/device_id` marker. A genuinely fresh store **mints a unique id** (peers see a brand-new device and open Olm normally); a pre-existing store **adopts** its configured id (back-compat, no re-key); a marker present is **reused** on restart. Appservice path only (the `access_token` path keeps its token-pinned device). **Verified live:** fresh store → `@dark` track `MissingKey→Ok`, media key received, clear audio **both directions**; warm store adopts `dark-voice` unchanged. Build + new resolver unit tests + clippy all green. **Non-blocking follow-ups** (candidates for separate issues): the superseded device lingers on the homeserver after each wipe (device-list growth, cosmetic); the per-ghost crypto clients use the same reuse pattern (harmless — ghosts are senders — but it's the source of the `SigningKeyChanged` log churn).

robocub closed this issue

2026-06-19 16:59:07 +00:00

robocub referenced this issue

2026-06-19 17:01:58 +00:00

Prune stale superseded bot devices on the homeserver after a fresh-store wipe #34

robocub referenced this issue

2026-06-19 17:01:58 +00:00

Apply store-paired device_id to per-ghost crypto clients (kill SigningKeyChanged churn) #35

robocub referenced this issue from a commit

2026-06-20 00:40:09 +00:00

fix(e2ee): per-store unique ghost device ids (#30 follow-up)

robocub referenced this issue from a commit