mgmt: restart-window invite double-processing can auto-DECLINE a legitimate DM invite (#89a cooldown misfire)

robocub commented

2026-07-12 04:04:07 +00:00

Collaborator

Observed in prod 2026-07-12 04:00 with the first real self-serve user (@dau:tinier.town), seconds after the alpha.24 restart:

04:00:08 INFO  accepting DM invite from self-serve user (open registration) room_id=!1zw8…:tinier.town inviter=@dau:tinier.town
04:00:10 ERROR failed to join invited room … 403 … Answer from tinier.town: Auth check failed: cannot join a room that is not `public`
04:00:10 INFO  accepting DM invite … (SAME room, SAME inviter — duplicate)
04:00:10 INFO  invite cooldown — declining self-serve invite (#89a) … remaining=28

Two stacked problems:

The same invite is processed twice around a restart — the live stripped-member handler and the startup invited_rooms() sweep both fire for one invite. The first acceptance stamps the #89a per-sender invite cooldown; the duplicate two seconds later then hits its own stamp and takes the DECLINE path — room.leave() on an invited room REJECTS the invite. Net effect: the user watches their invite get refused by the bot. The cooldown was designed to rate-limit distinct invites from one sender, not to fire on a duplicate of the invite we just accepted.
(Independent) the join itself 403'd from the remote server — "cannot join a room that is not public" — most consistent with the inviter having already retracted/left the invite during our restart window (their third attempt after the #96 docs bug; they'd been getting silence for an hour). Not our bug per se, but it's what left the duplicate-processing path exposed.

Fix direction: dedupe invite handling per room (in-flight/handled set keyed by room id — accept-once), and/or only stamp the #89a cooldown after a SUCCESSFUL join, and never take the decline path for a room we're already mid-join on. A failed federated join should also not leave the invite in limbo — either retry with backoff or decline explicitly with a log that says why.

Severity: real users hit this exactly when it hurts most (their retry lands during a deploy restart).

Observed in prod 2026-07-12 04:00 with the first real self-serve user (@dau:tinier.town), seconds after the alpha.24 restart: ``` 04:00:08 INFO accepting DM invite from self-serve user (open registration) room_id=!1zw8…:tinier.town inviter=@dau:tinier.town 04:00:10 ERROR failed to join invited room … 403 … Answer from tinier.town: Auth check failed: cannot join a room that is not `public` 04:00:10 INFO accepting DM invite … (SAME room, SAME inviter — duplicate) 04:00:10 INFO invite cooldown — declining self-serve invite (#89a) … remaining=28 ``` Two stacked problems: 1. **The same invite is processed twice around a restart** — the live stripped-member handler and the startup `invited_rooms()` sweep both fire for one invite. The first acceptance stamps the #89a per-sender invite cooldown; the duplicate two seconds later then hits its own stamp and takes the DECLINE path — `room.leave()` on an invited room REJECTS the invite. Net effect: the user watches their invite get refused by the bot. The cooldown was designed to rate-limit *distinct* invites from one sender, not to fire on a duplicate of the invite we just accepted. 2. (Independent) the join itself 403'd from the remote server — "cannot join a room that is not `public`" — most consistent with the inviter having already retracted/left the invite during our restart window (their third attempt after the #96 docs bug; they'd been getting silence for an hour). Not our bug per se, but it's what left the duplicate-processing path exposed. **Fix direction:** dedupe invite handling per room (in-flight/handled set keyed by room id — accept-once), and/or only stamp the #89a cooldown after a SUCCESSFUL join, and never take the decline path for a room we're already mid-join on. A failed federated join should also not leave the invite in limbo — either retry with backoff or decline explicitly with a log that says why. Severity: real users hit this exactly when it hurts most (their retry lands during a deploy restart).

robocub referenced this issue

2026-07-12 04:17:47 +00:00

e2e: DM/invite lifecycle coverage — E2EE DMs, restart races, cooldown semantics (gaps exposed by #96/#97) #99

robocub commented

2026-07-12 04:43:34 +00:00

Author

Collaborator

Live escalation notes (2026-07-12, ~04:15–04:45):

The double-processing is NOT restart-scoped — it fires on every federated invite. Federated joins take seconds (vs ms locally), so the sync loop redelivers the still-pending invite mid-join; the duplicate hits the #89a stamp from the first pass and DECLINES the just-joined room. Reproduced 3×: 04:19, 04:30, 04:37. This is why local e2e never saw it (#99 tier 4).
Temporary prod mitigation applied: mgmt_cooldown_secs commented out (backup config.toml.bak-pre-97mitigation) — #89a is inert when unset, so duplicates re-accept harmlessly. RESTORE with this fix. Cost while unset: no command cooldown, report un-gated.
Separate homeserver-layer blocker found beneath it: with the decline gone, the federated join itself now fails tinier.town→nether.im — 30s timeout, then 500 M_UNKNOWN: Server returned bad 200 response: Deserialization … missing field event. BOTH servers run Tuwunel 1.8.1, so this is a Tuwunel↔Tuwunel federation invite-join bug (intermittent: 3 joins succeeded earlier tonight). Bridge-side workaround for the stranded user: REVERSED the flow — the bot created the DM on nether.im and invited them (their joins toward nether.im work). Candidate upstream Tuwunel report; evidence in prod journal 04:37.
New sub-bug found during the workaround: bot-initiated DMs race the #89b empty-room reap. A create_dm/createRoom room has ONE joined member until the invitee accepts; the 5-min maintenance sweep reaps ≤1-member non-safe rooms → the invite gets retracted if the human takes >5 min to accept. This also affects #94's msg/announce to owners without an existing DM. Fix alongside the dedupe: exempt rooms with a PENDING invite (or younger than some grace) from the #89b reap.

Live escalation notes (2026-07-12, ~04:15–04:45): 1. **The double-processing is NOT restart-scoped — it fires on every federated invite.** Federated joins take seconds (vs ms locally), so the sync loop redelivers the still-pending invite mid-join; the duplicate hits the #89a stamp from the first pass and DECLINES the just-joined room. Reproduced 3×: 04:19, 04:30, 04:37. This is why local e2e never saw it (#99 tier 4). 2. **Temporary prod mitigation applied:** `mgmt_cooldown_secs` commented out (backup `config.toml.bak-pre-97mitigation`) — #89a is inert when unset, so duplicates re-accept harmlessly. RESTORE with this fix. Cost while unset: no command cooldown, `report` un-gated. 3. **Separate homeserver-layer blocker found beneath it:** with the decline gone, the federated join itself now fails tinier.town→nether.im — 30s timeout, then `500 M_UNKNOWN: Server returned bad 200 response: Deserialization … missing field event`. BOTH servers run Tuwunel 1.8.1, so this is a Tuwunel↔Tuwunel federation invite-join bug (intermittent: 3 joins succeeded earlier tonight). Bridge-side workaround for the stranded user: REVERSED the flow — the bot created the DM on nether.im and invited them (their joins toward nether.im work). Candidate upstream Tuwunel report; evidence in prod journal 04:37. 4. **New sub-bug found during the workaround: bot-initiated DMs race the #89b empty-room reap.** A `create_dm`/createRoom room has ONE joined member until the invitee accepts; the 5-min maintenance sweep reaps ≤1-member non-safe rooms → the invite gets retracted if the human takes >5 min to accept. This also affects #94's `msg`/`announce` to owners without an existing DM. Fix alongside the dedupe: exempt rooms with a PENDING invite (or younger than some grace) from the #89b reap.

robocub referenced this issue

2026-07-12 04:53:45 +00:00

e2e: #99 implementation — E2EE DM actor + invite-lifecycle scenarios (parallel-instance handoff) #100

robocub referenced this issue from a commit

2026-07-12 05:01:59 +00:00

fix(mgmt): #97 — invite dedupe, cooldown-stamp-after-join, reap exemption

robocub referenced this issue from a commit

2026-07-12 05:24:17 +00:00

feat(e2e): #99 invite/DM lifecycle coverage — E2EE DM actor + 3 scenarios

robocub referenced this issue

2026-07-12 05:26:16 +00:00

e2e: #99 implementation — E2EE DM actor + invite-lifecycle scenarios (parallel-instance handoff) #100

robocub referenced this issue

2026-07-12 05:35:10 +00:00

mgmt: invite-flow polish — explain rejected non-direct invites via DM; stretch the greeting window for slow federated joins #101

robocub referenced this issue from a commit

2026-07-12 05:49:15 +00:00

feat(e2e): #99 invite/DM lifecycle coverage — E2EE DM actor + 3 scenarios

robocub referenced this issue from a commit

2026-07-12 05:49:15 +00:00

feat(e2e): flip mgmt_dm_pending_reap to Pass — alpha.25 (#97 fix) merged

robocub referenced this issue from a commit

2026-07-12 06:15:04 +00:00

docs(e2e): record the second #97 residual mode — first-message UTD to cold DM recipients

robocub referenced this issue

2026-07-12 06:16:43 +00:00

e2e: #99 implementation — E2EE DM actor + invite-lifecycle scenarios (parallel-instance handoff) #100

robocub referenced this issue from a commit

2026-07-12 06:28:51 +00:00

fix(mgmt): #97 residuals — server-side pending-invite gate + member-load before first DM send

robocub referenced this issue from a commit

2026-07-12 06:37:29 +00:00

feat(e2e): #99 invite/DM lifecycle coverage — E2EE DM actor + 3 scenarios (#99)

robocub referenced this issue

2026-07-12 06:37:55 +00:00

e2e: #99 implementation — E2EE DM actor + invite-lifecycle scenarios (parallel-instance handoff) #100

robocub referenced this issue from a commit

2026-07-12 06:39:02 +00:00

fix(mgmt): #97 residuals — server-side pending-invite gate + member-load before first DM send

robocub referenced this issue

2026-07-12 06:40:19 +00:00

mgmt: first message to a COLD outreach DM is undecryptable for the recipient (megolm not shared to invited members) #104

robocub referenced this issue from a commit

2026-07-12 06:40:47 +00:00

test(e2e): re-point mgmt_dm_pending_reap KnownFail #97 → #104

robocub commented

2026-07-12 07:30:09 +00:00

Author

Collaborator

Shipped in v0.3.0-beta.1, deployed to prod 2026-07-12 07:29 UTC. #97: invite dedupe + cooldown-stamp-after-join (alpha.25) + the server-side pending-invite reap gate (beta.1; the alpha.25 summary-count gate was proven lagging by the #99 drill — evidence on #100). Prod mgmt_cooldown_secs restored with the deploy; post-restart journal confirms failed joins no longer arm the cooldown. #103: both room-id generations accepted (link/ban/report), via hints on first-contact joins, shape rejections logged. Residual mode 2 tracked as #104 with a live known-fail scenario.

Shipped in **v0.3.0-beta.1**, deployed to prod 2026-07-12 07:29 UTC. #97: invite dedupe + cooldown-stamp-after-join (alpha.25) + the server-side pending-invite reap gate (beta.1; the alpha.25 summary-count gate was proven lagging by the #99 drill — evidence on #100). Prod `mgmt_cooldown_secs` restored with the deploy; post-restart journal confirms failed joins no longer arm the cooldown. #103: both room-id generations accepted (`link`/`ban`/`report`), via hints on first-contact joins, shape rejections logged. Residual mode 2 tracked as #104 with a live known-fail scenario.

robocub closed this issue

2026-07-12 07:30:09 +00:00

robocub referenced this issue

2026-07-12 07:30:09 +00:00

mgmt: v12 (domainless) room ids rejected by link shape-check; first-contact joins need a via hint #103

robocub referenced this issue

2026-07-12 07:30:09 +00:00

e2e: #99 implementation — E2EE DM actor + invite-lifecycle scenarios (parallel-instance handoff) #100

robocub referenced this issue

2026-07-12 13:34:09 +00:00

e2e: auxiliary federated test homeserver — unlock the federation scenario class #106

robocub referenced this issue