mgmt: first message to a COLD outreach DM is undecryptable for the recipient (megolm not shared to invited members)

robocub commented

2026-07-12 06:40:19 +00:00

Collaborator

Split from #97 (mode 2 of the #100 residuals): an operator msg/announce to a user with NO existing DM creates the room and sends immediately; in the (default-encrypted) fresh DM, the megolm session is shared before/without the invited recipient — they accept and find the first message permanently undecryptable.

Not fixed by member-list loading (tried in 5c6cb63: room.get_member server-backed fetch + retry before the send — instance-2's mgmt_dm_pending_reap drill still reproduces the UTD). Working theory: the SDK does not share megolm to INVITED members (and/or MSC3061 shared-history semantics) — the recipient can only ever decrypt messages sent after their join lands in the bot's view.

Candidate fixes to evaluate:

dm_user waits (bounded) for the recipient to JOIN before the first send — correct-by-construction, adds delivery latency; needs a queue for announce so N cold recipients don't serialize a handler.
Discard + re-share the outbound session once the join is seen, resending the message.
Deep: verify matrix-sdk's shared-history/invited-member share behavior and use it properly.

Scope: operator outreach to cold recipients only. Self-serve users (who initiate the DM themselves) are unaffected — live-proven E2EE DM with a real user 2026-07-12. The e2e scenario mgmt_dm_pending_reap stays KnownFail pointing HERE, so the suite tracks it honestly; #97 itself (invite races + reap, mode 1) is fully fixed and closes with the next release.

Split from #97 (mode 2 of the #100 residuals): an operator `msg`/`announce` to a user with NO existing DM creates the room and sends immediately; in the (default-encrypted) fresh DM, the megolm session is shared before/without the invited recipient — they accept and find the first message permanently undecryptable. **Not fixed by member-list loading** (tried in `5c6cb63`: `room.get_member` server-backed fetch + retry before the send — instance-2's `mgmt_dm_pending_reap` drill still reproduces the UTD). Working theory: the SDK does not share megolm to INVITED members (and/or MSC3061 shared-history semantics) — the recipient can only ever decrypt messages sent after their join lands in the bot's view. Candidate fixes to evaluate: 1. `dm_user` waits (bounded) for the recipient to JOIN before the first send — correct-by-construction, adds delivery latency; needs a queue for `announce` so N cold recipients don't serialize a handler. 2. Discard + re-share the outbound session once the join is seen, resending the message. 3. Deep: verify matrix-sdk's shared-history/invited-member share behavior and use it properly. Scope: operator outreach to cold recipients only. Self-serve users (who initiate the DM themselves) are unaffected — live-proven E2EE DM with a real user 2026-07-12. The e2e scenario `mgmt_dm_pending_reap` stays `KnownFail` pointing HERE, so the suite tracks it honestly; #97 itself (invite races + reap, mode 1) is fully fixed and closes with the next release.

robocub referenced this issue from a commit

2026-07-12 06:40:47 +00:00

test(e2e): re-point mgmt_dm_pending_reap KnownFail #97 → #104

robocub referenced this issue

2026-07-12 07:30:09 +00:00

mgmt: restart-window invite double-processing can auto-DECLINE a legitimate DM invite (#89a cooldown misfire) #97

robocub referenced this issue

2026-07-12 07:30:09 +00:00

mgmt: v12 (domainless) room ids rejected by link shape-check; first-contact joins need a via hint #103

robocub referenced this issue

2026-07-12 07:30:09 +00:00

e2e: #99 implementation — E2EE DM actor + invite-lifecycle scenarios (parallel-instance handoff) #100

robocub referenced this issue

2026-07-12 07:37:10 +00:00

e2e: #99 implementation — E2EE DM actor + invite-lifecycle scenarios (parallel-instance handoff) #100

robocub referenced this issue

2026-07-12 13:34:09 +00:00

e2e: auxiliary federated test homeserver — unlock the federation scenario class #106

robocub referenced this issue

2026-07-12 14:07:55 +00:00

infra: nether.autos incus container — federated-fixture setup runbook (new-session handoff for #106/#108) #109

robocub referenced this issue

2026-07-12 17:49:21 +00:00

infra: nether.autos incus container — federated-fixture setup runbook (new-session handoff for #106/#108) #109

robocub referenced this issue from a commit

2026-07-12 18:04:54 +00:00

feat(e2e): federated scenarios against the nether.autos fixture (#109)

robocub referenced this issue

2026-07-12 18:09:55 +00:00

infra: nether.autos incus container — federated-fixture setup runbook (new-session handoff for #106/#108) #109

robocub referenced this issue

2026-07-12 21:10:32 +00:00

infra: nether.autos incus container — federated-fixture setup runbook (new-session handoff for #106/#108) #109

robocub commented

2026-07-13 01:20:13 +00:00

Author

Collaborator

Root-cause update (2026-07-13): this is NOT "megolm not shared to invited members." The bridge shares the key correctly — the failure is federated to-device (EDU) delivery.

The proposed fix was built (server-authoritative latest_encryption_state + Room::members() fetch, plus request_user_identity to force a synchronous federated /keys/query, before the send — covering both msg and announce) and live-tested against the nether.autos fixture. fed_cold_outreach_utd still failed deterministically. Crypto-level tracing (matrix_sdk_crypto=debug) shows why the fix was aimed at a non-problem:

Bridge side — the key IS shared to the invited remote recipient. share_room_key for the DM room lists the target among the recipients ("Trying to encrypt a room key" → {@…:nether.autos: {<device>}}), a to-device batch is created for that device, and the outbound session is "marked as shared". The recipient's device is discovered and included even while they are merely invited — device discovery was never broken.
Target side — the key never arrives. Capturing the receiving crypto client's log across the full 180 s wait (the actor runs in the nvb-e2e process, so RUST_LOG=nvb_e2e=info,matrix_sdk_crypto=debug captures it with no bridge rebuild), it never logs "Received a new megolm room key" for the DM room. The to-device event carrying the session is not delivered to it.
Same-server works. mgmt_dm_e2ee — identical client and send path, same homeserver — passes. The only differing variable in fed_cold_outreach_utd is the federation hop.

⇒ The federated failure is to-device delivery dropping the room-key EDU to the recipient's homeserver, i.e. a federation/infrastructure gap (same family as the known Tuwunel federation issues), not a bridge defect. The attempted bridge-side fix has been reverted; fed_cold_outreach_utd stays KnownFail{#104}.

Note: request_user_identity does exist in matrix-sdk 0.18 and performs an up-to-date federated /keys/query, but it's irrelevant here since discovery already succeeds.

Next: confirm which hop drops the EDU (sender outbound vs fixture inbound) from server logs, and retarget the scenario as a fixture/federation-delivery check rather than a bridge-behaviour assertion. Suggest re-scoping the title away from "megolm not shared to invited members."

**Root-cause update (2026-07-13): this is NOT "megolm not shared to invited members." The bridge shares the key correctly — the failure is federated to-device (EDU) delivery.** The proposed fix was built (server-authoritative `latest_encryption_state` + `Room::members()` fetch, plus `request_user_identity` to force a synchronous federated `/keys/query`, before the send — covering both `msg` and `announce`) and live-tested against the `nether.autos` fixture. `fed_cold_outreach_utd` still failed deterministically. Crypto-level tracing (`matrix_sdk_crypto=debug`) shows why the fix was aimed at a non-problem: - **Bridge side — the key IS shared to the invited remote recipient.** `share_room_key` for the DM room lists the target among the recipients (`"Trying to encrypt a room key"` → `{@…:nether.autos: {<device>}}`), a to-device batch is created for that device, and the outbound session is `"marked as shared"`. The recipient's device is discovered and included even while they are merely *invited* — device discovery was never broken. - **Target side — the key never arrives.** Capturing the *receiving* crypto client's log across the full 180 s wait (the actor runs in the `nvb-e2e` process, so `RUST_LOG=nvb_e2e=info,matrix_sdk_crypto=debug` captures it with no bridge rebuild), it **never** logs `"Received a new megolm room key"` for the DM room. The to-device event carrying the session is not delivered to it. - **Same-server works.** `mgmt_dm_e2ee` — identical client and send path, same homeserver — passes. The only differing variable in `fed_cold_outreach_utd` is the federation hop. ⇒ The federated failure is **to-device delivery dropping the room-key EDU to the recipient's homeserver**, i.e. a federation/infrastructure gap (same family as the known Tuwunel federation issues), **not a bridge defect**. The attempted bridge-side fix has been reverted; `fed_cold_outreach_utd` stays `KnownFail{#104}`. Note: `request_user_identity` does exist in matrix-sdk 0.18 and performs an up-to-date federated `/keys/query`, but it's irrelevant here since discovery already succeeds. Next: confirm which hop drops the EDU (sender outbound vs fixture inbound) from server logs, and retarget the scenario as a fixture/federation-delivery check rather than a bridge-behaviour assertion. Suggest re-scoping the title away from "megolm not shared to invited members."

robocub referenced this issue from a commit

2026-07-13 17:27:34 +00:00

feat(mgmt): awaiting-invite link resume — self-serve link of invite-only rooms (#111)

robocub referenced this issue

2026-07-13 17:27:57 +00:00

feat(mgmt): awaiting-invite link resume — self-serve link of invite-only rooms (#111) #119

robocub commented

2026-07-19 21:44:21 +00:00

Author

Collaborator

Possible upstream fix in Tuwunel v1.8.2 (2026-07-17). The 07-13 diagnosis of the federated variant (fed_cold_outreach_utd) was that the bridge shares + dispatches the megolm key correctly, but the to-device EDU carrying it never arrives at the remote recipient — i.e. a federated to-device delivery drop, infrastructure rather than bridge code (same-server mgmt_dm_e2ee passes; only the federation hop differs). v1.8.2 carries two changes squarely in that area:

"Federation delivery no longer runs hot against a peer that has come back — the backoff gate now consults full failure history instead of just the current time bucket, preventing re-authorization at timeout boundaries."
"A proxy or CDN answering a federation request with non-JSON is treated as transient rather than evicting the route outright," plus profile queries now return 404 instead of an empty 200 (empty-200 hardening).

Prod nether.im is already on 1.8.2; the fixture nether.autos is being upgraded 1.8.1 → 1.8.2 now. Next: re-run fed_cold_outreach_utd against the upgraded fixture and record the result here — if it flips to Pass, #104's federated failure was this upstream federation-delivery bug, not anything in the bridge.

Release: https://github.com/matrix-construct/tuwunel/releases/tag/v1.8.2

**Possible upstream fix in Tuwunel v1.8.2** (2026-07-17). The 07-13 diagnosis of the *federated* variant (`fed_cold_outreach_utd`) was that the bridge shares + dispatches the megolm key correctly, but the **to-device EDU carrying it never arrives** at the remote recipient — i.e. a federated to-device *delivery* drop, infrastructure rather than bridge code (same-server `mgmt_dm_e2ee` passes; only the federation hop differs). v1.8.2 carries two changes squarely in that area: - *"Federation delivery no longer runs hot against a peer that has come back — the backoff gate now consults full failure history instead of just the current time bucket, preventing re-authorization at timeout boundaries."* - *"A proxy or CDN answering a federation request with non-JSON is treated as transient rather than evicting the route outright,"* plus profile queries now return 404 instead of an empty 200 (empty-200 hardening). Prod `nether.im` is already on 1.8.2; the fixture `nether.autos` is being upgraded 1.8.1 → 1.8.2 now. **Next: re-run `fed_cold_outreach_utd` against the upgraded fixture and record the result here** — if it flips to Pass, #104's federated failure was this upstream federation-delivery bug, not anything in the bridge. Release: https://github.com/matrix-construct/tuwunel/releases/tag/v1.8.2

robocub referenced this issue

2026-07-19 22:12:58 +00:00

e2e nightly RED 8 days — real multi-channel/encrypted/recovery audio failure cluster (NOT contention); env-vs-code TBD #124

robocub commented

2026-07-19 22:14:16 +00:00

Author

Collaborator

v1.8.2 retest — NEGATIVE (not fixed by the version bump). Upgraded the fixture nether.autos 1.8.1 → 1.8.2 (both prod nether.im and the fixture now on 1.8.2, signing key preserved) and re-ran fed_cold_outreach_utd: still known_fail (operator message not decryptable by the remote target within the window). So the v1.8.2 federation-delivery/empty-200 changes did NOT resolve #104s federated to-device drop on their own. Note the same-server twin mgmt_dm_pending_reap now unexpected-passes — worth confirming whether that improvement is 1.8.2-related and, if stable, flipping its expectation. #104 federated variant stays open; next angle is a fresh crypto-trace on the upgraded fixture to see whether the to-device EDU now reaches the target server at all.

**v1.8.2 retest — NEGATIVE (not fixed by the version bump).** Upgraded the fixture `nether.autos` 1.8.1 → 1.8.2 (both prod `nether.im` and the fixture now on 1.8.2, signing key preserved) and re-ran `fed_cold_outreach_utd`: still `known_fail` (operator message not decryptable by the remote target within the window). So the v1.8.2 federation-delivery/empty-200 changes did NOT resolve #104s federated to-device drop on their own. Note the same-server twin `mgmt_dm_pending_reap` now *unexpected-passes* — worth confirming whether that improvement is 1.8.2-related and, if stable, flipping its expectation. #104 federated variant stays open; next angle is a fresh crypto-trace on the upgraded fixture to see whether the to-device EDU now reaches the target server at all.

robocub referenced this issue

2026-08-19 23:47:15 +00:00

e2e nightly RED 8 days — real multi-channel/encrypted/recovery audio failure cluster (NOT contention); env-vs-code TBD #124

robocub referenced this issue

2026-08-20 00:06:33 +00:00

e2e: #99 implementation — E2EE DM actor + invite-lifecycle scenarios (parallel-instance handoff) #100

robocub referenced this issue

2026-08-20 19:10:07 +00:00

e2e nightly RED 8 days — real multi-channel/encrypted/recovery audio failure cluster (NOT contention); env-vs-code TBD #124

robocub commented

2026-08-20 22:41:51 +00:00

Author

Collaborator

2026-08-20 wire-capture diagnostic: the "federated to-device delivery drop" theory is DISPROVEN — the key arrives and is delivered to the client.

Method: tuwunel's packaged binary compiles DEBUG logging out entirely (max_level_info tracing feature — the startup log says so), so server-side log tracing is a dead end on BOTH sides regardless of config. Instead: the fixture's ingress is cloudflared → plaintext :8008, so a tcpdump on the fixture's loopback during a dispatched fed_cold_outreach_utd run (#351) captured full federation transaction bodies.

Captured, in order:

PUT /_matrix/federation/v1/send/... from nether.im containing the m.direct_to_device EDU: sender @nether-voicebridge_e2e_bot:nether.im, target @autotest_cold:nether.autos device YTa7URxYl6, an m.olm.v1.curve25519-aes-sha2 prekey (type 0) ciphertext — i.e. the federated /keys/claim also worked. Fixture answered 200 OK.
A /sync response to the harness's client delivering that exact ciphertext in to_device.events, seconds later — well inside the scenario's decryption window.

Yet the scenario still known_fails ("operator message not decryptable"). So on Tuwunel 1.8.2 the entire server path — device discovery, OTK claim, EDU federation, client delivery — works; the failure is client-side in the receiving actor (or in what it observes). Two candidates for the next session:

the received Olm prekey message failing to process in the actor's OlmMachine (rerun with matrix_sdk_crypto=trace on the harness to see the processing verdict);
simpler: the 📣 is sent while the target is still INVITED, so the event may precede the client's join and never appear in its sync timeline at all (needs /messages backfill to even see it) — which would make part of this a harness-observation artifact and would also reframe the real-user story (a real client backfills and shows UTD-or-not).

The 07-13 "key never arrives at the target" trace predates the 1.8.2 upgrade — v1.8.2's federation fixes may well have cured the transport after all, moving the residual failure somewhere new. Fixture config restored (log level reverted, capture files removed).

**2026-08-20 wire-capture diagnostic: the "federated to-device delivery drop" theory is DISPROVEN — the key arrives and is delivered to the client.** Method: tuwunel's packaged binary compiles DEBUG logging out entirely (`max_level_info` tracing feature — the startup log says so), so server-side log tracing is a dead end on BOTH sides regardless of config. Instead: the fixture's ingress is cloudflared → plaintext `:8008`, so a tcpdump on the fixture's loopback during a dispatched `fed_cold_outreach_utd` run (#351) captured full federation transaction bodies. Captured, in order: 1. `PUT /_matrix/federation/v1/send/...` from `nether.im` containing the **`m.direct_to_device` EDU**: sender `@nether-voicebridge_e2e_bot:nether.im`, target `@autotest_cold:nether.autos` device `YTa7URxYl6`, an `m.olm.v1.curve25519-aes-sha2` **prekey (type 0)** ciphertext — i.e. the federated `/keys/claim` also worked. Fixture answered `200 OK`. 2. A `/sync` response to the harness's client delivering **that exact ciphertext** in `to_device.events`, seconds later — well inside the scenario's decryption window. Yet the scenario still `known_fail`s ("operator message not decryptable"). So on Tuwunel 1.8.2 the entire server path — device discovery, OTK claim, EDU federation, client delivery — works; the failure is **client-side in the receiving actor** (or in what it observes). Two candidates for the next session: - the received Olm prekey message failing to process in the actor's OlmMachine (rerun with `matrix_sdk_crypto=trace` on the harness to see the processing verdict); - simpler: the 📣 is sent while the target is still INVITED, so the event may precede the client's join and never appear in its sync timeline at all (needs `/messages` backfill to even see it) — which would make part of this a harness-observation artifact and would also reframe the real-user story (a real client backfills and shows UTD-or-not). The 07-13 "key never arrives at the target" trace predates the 1.8.2 upgrade — v1.8.2's federation fixes may well have cured the transport after all, moving the residual failure somewhere new. Fixture config restored (log level reverted, capture files removed).

Rows
Columns

mgmt: first message to a COLD outreach DM is undecryptable for the recipient (megolm not shared to invited members) #104