Prune stale superseded bot devices on the homeserver after a fresh-store wipe #34

New issue

Closed

opened 2026-06-19 17:01:58 +00:00 by robocub · 3 comments

robocub commented

2026-06-19 17:01:58 +00:00

Collaborator

Follow-up from the #30 fix (84b0a0c).

To fix #30, a fresh-store deploy now mints a new unique appservice-bot device_id (e.g. dark-voice-<rand>) so peers don't reject a reused id whose keys changed. The previously-minted device is never removed, so @voicebridge_as accumulates one stale device per fresh wipe/redeploy.

Impact: cosmetic — device-list growth on the homeserver. Old devices just go idle; no correctness or audio impact (the marker makes restarts reuse the same id, so growth only happens on a genuine store wipe).
Possible fix: when minting a new id, delete the bot's prior device(s) via an as_token-masqueraded DELETE /_matrix/client/v3/devices/{id} (handle UIA), or periodically prune inactive devices.
Priority: low / housekeeping.

Follow-up from the #30 fix (`84b0a0c`). To fix #30, a fresh-store deploy now mints a **new unique** appservice-bot `device_id` (e.g. `dark-voice-<rand>`) so peers don't reject a reused id whose keys changed. The **previously-minted device is never removed**, so `@voicebridge_as` accumulates one stale device per fresh wipe/redeploy. - **Impact:** cosmetic — device-list growth on the homeserver. Old devices just go idle; no correctness or audio impact (the marker makes restarts reuse the same id, so growth only happens on a genuine store wipe). - **Possible fix:** when minting a new id, delete the bot's prior device(s) via an `as_token`-masqueraded `DELETE /_matrix/client/v3/devices/{id}` (handle UIA), or periodically prune inactive devices. - **Priority:** low / housekeeping.

robocub added the

enhancement

label

2026-06-19 17:01:58 +00:00

robocub referenced this issue

2026-07-03 00:30:50 +00:00

Ghost to-device receive path / Olm wedge self-heal #52

robocub commented

2026-07-03 00:31:10 +00:00

Author

Collaborator

Live-proven motivation (2026-07-02). Stale ghost-device accumulation directly causes persistent per-publisher silence on encrypted calls: a listener's device list held ~10 devices for a single ghost MXID (5 C… crypto — one per bridge run, never deleted — + 5 old-salt D… call devices) and ~23 for @voicebridge_as. This poisons the device list (InvalidSignature/SigningKeyChanged on stale devices) and drains the listener's one-time keys → the ghost→listener Olm session wedges on MissingOneTimeKey → that speaker is inaudible until the ghost's device changes. Pruning to one current device per MXID at startup is the fix. See #52 for the wedge mechanism and the deferred self-heal path.

**Live-proven motivation (2026-07-02).** Stale ghost-device accumulation directly causes persistent per-publisher silence on encrypted calls: a listener's device list held ~10 devices for a single ghost MXID (5 `C…` crypto — one per bridge run, never deleted — + 5 old-salt `D…` call devices) and ~23 for `@voicebridge_as`. This poisons the device list (`InvalidSignature`/`SigningKeyChanged` on stale devices) and drains the listener's one-time keys → the ghost→listener Olm session wedges on `MissingOneTimeKey` → that speaker is inaudible until the ghost's device changes. Pruning to one current device per MXID at startup is the fix. See #52 for the wedge mechanism and the deferred self-heal path.

robocub referenced this issue

2026-07-03 00:31:10 +00:00

Apply store-paired device_id to per-ghost crypto clients (kill SigningKeyChanged churn) #35

robocub commented

2026-07-03 00:44:21 +00:00

Author

Collaborator

Spike result (2026-07-02): Tuwunel blocks CS-API device deletion for appservice users — #34-by-deletion is a dead end on this homeserver. Built prune_stale_devices (enumerate own devices, keep current, delete the rest) and tested live: the ghost's own token needs a JSON body on DELETE /devices/{id} (else M_NOT_JSON), and with {} returns 401 with an EMPTY flows:[] — no UIA stage a passwordless appservice user can complete. as_token + ?user_id= masquerade returns 403 M_FORBIDDEN ("user must be authenticated and device identified"). No auth variant deletes. Pivot to #35 (stop minting a fresh crypto device per run so nothing accumulates — needs a light persistent identity). The existing backlog (~hundreds of stale devices already on the HS) will need a one-time Tuwunel admin cleanup, since CS-API deletion is unavailable. See #52 for the wedge mechanism this feeds.

**Spike result (2026-07-02): Tuwunel blocks CS-API device deletion for appservice users — #34-by-deletion is a dead end on this homeserver.** Built `prune_stale_devices` (enumerate own devices, keep current, delete the rest) and tested live: the ghost's own token needs a JSON body on `DELETE /devices/{id}` (else `M_NOT_JSON`), and with `{}` returns **401 with an EMPTY `flows:[]`** — no UIA stage a passwordless appservice user can complete. `as_token` + `?user_id=` masquerade returns **403 M_FORBIDDEN** ("user must be authenticated and device identified"). No auth variant deletes. **Pivot to #35** (stop minting a fresh crypto device per run so nothing accumulates — needs a light persistent identity). The existing backlog (~hundreds of stale devices already on the HS) will need a one-time Tuwunel admin cleanup, since CS-API deletion is unavailable. See #52 for the wedge mechanism this feeds.

robocub referenced this issue

2026-07-03 21:38:25 +00:00

Startup self-prune of stale bridge/ghost devices (device hygiene automation) #53

robocub commented

2026-07-05 15:31:37 +00:00

Author

Collaborator

Done via #53, released v0.3.0-alpha.3 (master 436a5af), deployed to prod 2026-07-04. What shipped is stronger than the original ask:

History-scoped prune: each store records the device ids it minted and prunes only its own superseded devices — safe for dual deployments (a dev box and the VPS can share an MXID without pruning each other's live devices).
Deletion self-heal: if the live device is deleted server-side (the 2026-07-03 cleanup incident — deleting live bot devices made clients refuse to send media keys to the resurrected ids), the store detects it and mints a fresh device instead of resurrecting a dead id.

Caveat, documented in #53: nether.im's current Tuwunel build exposes no device deletion for appservice users, so prunes soft-fail there today (logged, harmless). Latest Tuwunel has MSC4190 — a homeserver upgrade activates the deletions, which is exactly what the code was designed for. Closing.

**Done via #53, released v0.3.0-alpha.3** (master `436a5af`), deployed to prod 2026-07-04. What shipped is stronger than the original ask: - **History-scoped prune**: each store records the device ids *it* minted and prunes only its own superseded devices — safe for dual deployments (a dev box and the VPS can share an MXID without pruning each other's live devices). - **Deletion self-heal**: if the *live* device is deleted server-side (the 2026-07-03 cleanup incident — deleting live bot devices made clients refuse to send media keys to the resurrected ids), the store detects it and mints a fresh device instead of resurrecting a dead id. Caveat, documented in #53: nether.im's current Tuwunel build exposes **no device deletion for appservice users**, so prunes soft-fail there today (logged, harmless). Latest Tuwunel has MSC4190 — a homeserver upgrade activates the deletions, which is exactly what the code was designed for. Closing.

robocub closed this issue

2026-07-05 15:31:37 +00:00