Startup self-prune of stale bridge/ghost devices (device hygiene automation) #53

New issue

Closed

opened 2026-07-03 21:38:25 +00:00 by robocub · 1 comment

robocub commented

2026-07-03 21:38:25 +00:00

Collaborator

Motivation (root-caused 2026-07-03)

Encrypted Matrix→Discord went silent on all bridges: the manual stale-device purge (tuwunel-cleanup.txt, 138 !admin users delete-device commands) included the bridges' LIVE bot devices (dark-voice-v3wsney6, light-voice-y5qnpmvj, light-gaming-gh4bf4g2).

Chain of failure:

Deleting a live device makes every client drop it from its crypto/device store.
On the next service restart the bridge resurrects the same device id with identical keys (store + marker were untouched), and Tuwunel does not re-announce a resurrection whose keys are unchanged.
Clients therefore permanently refuse to Olm-encrypt the call media key to the subscriber → MissingKey forever → Matrix speakers inaudible on Discord. No errors anywhere on the bridge, client, or homeserver.

Fix applied (operational): delete the store's device_id marker → resolve_store_device_id mints a fresh unique id + resets the bot crypto store → clients see a genuinely new device and key it within ~1s. Verified live on all three bridges.

Proposed hardening

Automate device hygiene so manual cleanup lists are never needed again:

Bot: at startup, after login, list the AS user's devices (GET /_matrix/client/v3/devices), delete any device matching this bridge's <bridge-slug>- prefix that is not the current minted id.
Ghosts: analogously via as_token masquerade — delete stale D<discord-id>-* devices that don't match the current per-store salt-derived id.
Rate-limit / batch deletes; log a summary line.

Rules for any remaining MANUAL cleanup (until this ships)

Never delete an id present in any store's device_id marker (<data_dir>/<bridge>/device_id).
Never delete the live ghost device ids (D<discord-id>-<current-salt>, salt in <bridge>/ghost_device_salt).
A safe orphan list for the current deployment is in tuwunel-orphan-devices-20260703.txt.

Notes

Since #35 (stable persisted crypto identities) normal operation no longer churns devices, so accumulation is slow — this is hosted-grade hygiene, not urgent.
Related: #34 (prune, superseded by #35), #35, #44.

## Motivation (root-caused 2026-07-03) Encrypted Matrix→Discord went silent on all bridges: the manual stale-device purge (`tuwunel-cleanup.txt`, 138 `!admin users delete-device` commands) **included the bridges' LIVE bot devices** (`dark-voice-v3wsney6`, `light-voice-y5qnpmvj`, `light-gaming-gh4bf4g2`). Chain of failure: 1. Deleting a live device makes every client drop it from its crypto/device store. 2. On the next service restart the bridge **resurrects the same device id with identical keys** (store + marker were untouched), and Tuwunel does **not** re-announce a resurrection whose keys are unchanged. 3. Clients therefore permanently refuse to Olm-encrypt the call media key to the subscriber → `MissingKey` forever → Matrix speakers inaudible on Discord. No errors anywhere on the bridge, client, or homeserver. Fix applied (operational): delete the store's `device_id` marker → `resolve_store_device_id` mints a fresh unique id + resets the bot crypto store → clients see a genuinely new device and key it within ~1s. Verified live on all three bridges. ## Proposed hardening Automate device hygiene so manual cleanup lists are never needed again: - **Bot:** at startup, after login, list the AS user's devices (`GET /_matrix/client/v3/devices`), delete any device matching this bridge's `<bridge-slug>-` prefix that is not the current minted id. - **Ghosts:** analogously via `as_token` masquerade — delete stale `D<discord-id>-*` devices that don't match the current per-store salt-derived id. - Rate-limit / batch deletes; log a summary line. ## Rules for any remaining MANUAL cleanup (until this ships) - **Never delete an id present in any store's `device_id` marker** (`<data_dir>/<bridge>/device_id`). - **Never delete the live ghost device ids** (`D<discord-id>-<current-salt>`, salt in `<bridge>/ghost_device_salt`). - A safe orphan list for the current deployment is in `tuwunel-orphan-devices-20260703.txt`. ## Notes - Since #35 (stable persisted crypto identities) normal operation no longer churns devices, so accumulation is slow — this is hosted-grade hygiene, not urgent. - Related: #34 (prune, superseded by #35), #35, #44.

robocub referenced this issue

2026-07-04 00:37:00 +00:00

Ghost to-device receive path / Olm wedge self-heal #52

robocub referenced this issue from a commit

2026-07-04 00:41:52 +00:00

feat(matrix-rtc): device hygiene — history-scoped self-prune + deletion self-heal (#53)

robocub referenced this issue from a commit

2026-07-04 00:41:52 +00:00

release: 0.3.0-alpha.3

robocub commented

2026-07-04 00:42:12 +00:00

Author

Collaborator

Implemented and shipped in v0.3.0-alpha.3 (master 436a5af; feature commits e22ca71 + 2bc1a8c).

Two mechanisms, both live-verified end-to-end on the production deployment (2026-07-04):

Deletion self-heal — before reusing a persisted identity (bot via resolve_store_device_id, ghost via its pickled identity blob), the bridge checks /keys/query; if the device keys were deleted server-side while offline it re-provisions a fresh device instead of resurrecting the old id. Verified by deleting the live Dark Voice bot device in the admin room with the bridge stopped: restart detected it, re-keyed, the listener's client keyed the new device in ~1 s, audio confirmed both directions. Fails open on transient errors.
History-scoped pruning — every re-mint records the abandoned id (bot: <store>/retired_device_ids; ghost: retired field in the identity blob); startup deletes exactly those ids. A store can only delete ids it itself retired, so parallel deployments (dev box + VPS on one appservice) can never prune each other's live devices — the failure mode that motivated this issue is structurally impossible. Config: prune_stale_devices (default on).

Tuwunel caveat (live-probed): no client-API device deletion exists for appservice users today — masqueraded DELETE is 403, and even a self-delete by the device's own session gets a 401 with empty UIA flows (no MSC4190). Pruning therefore soft-fails there with one WARN and retains the history; it flushes automatically if/when the homeserver allows deletion. Until then, stale-device removal remains an admin-room operation.

Operator leftover from testing: !admin users delete-device @voicebridge_as:nether.im dark-voice-prunetest0

Implemented and shipped in **v0.3.0-alpha.3** (master `436a5af`; feature commits `e22ca71` + `2bc1a8c`). Two mechanisms, both **live-verified end-to-end on the production deployment** (2026-07-04): 1. **Deletion self-heal** — before reusing a persisted identity (bot via `resolve_store_device_id`, ghost via its pickled identity blob), the bridge checks `/keys/query`; if the device keys were deleted server-side while offline it re-provisions a fresh device instead of resurrecting the old id. Verified by deleting the live Dark Voice bot device in the admin room with the bridge stopped: restart detected it, re-keyed, the listener's client keyed the new device in ~1 s, audio confirmed both directions. Fails open on transient errors. 2. **History-scoped pruning** — every re-mint records the abandoned id (bot: `<store>/retired_device_ids`; ghost: `retired` field in the identity blob); startup deletes exactly those ids. A store can only delete ids it itself retired, so parallel deployments (dev box + VPS on one appservice) can never prune each other's live devices — the failure mode that motivated this issue is structurally impossible. Config: `prune_stale_devices` (default on). **Tuwunel caveat (live-probed):** no client-API device deletion exists for appservice users today — masqueraded DELETE is 403, and even a self-delete by the device's own session gets a 401 with *empty* UIA flows (no MSC4190). Pruning therefore soft-fails there with one WARN and retains the history; it flushes automatically if/when the homeserver allows deletion. Until then, stale-device removal remains an admin-room operation. Operator leftover from testing: `!admin users delete-device @voicebridge_as:nether.im dark-voice-prunetest0`

robocub closed this issue

2026-07-04 00:42:12 +00:00

robocub referenced this issue

2026-07-05 15:31:37 +00:00

Prune stale superseded bot devices on the homeserver after a fresh-store wipe #34

robocub referenced this issue

2026-07-05 16:18:14 +00:00

Reactive ghost re-key on client unwedge via MSC4203 to-device push (blocked upstream on Tuwunel) #55