fix: classify sync errors structurally + auto-restart failed bridges (#130, #131) #132

Merged

robocub merged 3 commits from fix/130-sync-auth-misclassification into master

2026-08-14 17:41:37 +00:00

robocub commented

2026-08-14 02:06:14 +00:00

Collaborator

Closes #130, closes #131.

The bug

A transient /sync transport error was classified as a fatal auth failure, because the classifier substring-matched the stringified error — which embeds the request URL, and /sync carries ?since=<token>:

if msg.contains("401") || msg.contains("403") || msg.contains("M_UNKNOWN_TOKEN") { /* fatal */ }

On 2026-08-11 the homeserver's sync stream sat at position 5401xxxx, so every bridge's token (54013070, 54014591, 54015213, 54015517) contained 401. Five bridges died in ten minutes. Three — the static config.toml ones — stayed dead for three days, because a fatal verdict had no restart path; the tenant bridges came back only when an unrelated link approval triggered a reconcile.

What changed

auth_error — a pure is_fatal_auth(status, errcode) plus a matrix_sdk::Error adapter. A transport error (no HTTP response at all) is never fatal; 401/M_UNKNOWN_TOKEN/M_MISSING_TOKEN are, and M_FORBIDDEN only alongside a 403. Replaces the substring checks in both sync loops. AppserviceHttpError now carries a typed status so the join path stops matching on "403" too.
Supervisor auto-restart — bridge futures report a BridgeOutcome; a failed one (error or panic) is respawned after bridge_restart_base_ms * 2^n, capped by bridge_restart_max_ms (new [timing] keys, 5 s → 5 min), streak reset after stable uptime. Operator stops, reconcile replacements and shutdown never auto-restart. This also closes the reboot boot-race where a startup 502 killed every bridge permanently.
<data_dir>/bridges.json — a roster naming every bridge alive/restarting/down, refreshed every 60 s as a dead-man's switch.
ops/ — nvb-watch.sh, nvb-backup.sh and a conf example brought into the repo (they lived only on the prod host, unreviewed), with a bridge-liveness section that re-alerts daily while a bridge is down and puts bridges=N/M in the digest. ops/test-nvb-watch.sh drives the real watcher against stubbed journalctl/systemctl/curl.
CI unblocked — generate_config_with_pool_size lost its last non-test caller in ad53ccd, so clippy --all-targets -D warnings has failed on master since the 0.3.1 release (run #289, 19 days). Gated behind cfg(test).

Verification

cargo test --workspace --locked — 359 tests, 0 failures; clippy --all-targets -D warnings clean. CI #311 green.
New unit tests pin the incident: an error with no HTTP status whose message contains since=54015213 classifies as transient, and the four real tokens are asserted to still trip the old substring predicate while the typed one lets them through.
ops/test-nvb-watch.sh: 13/13. It caught two real bugs in the new section before it shipped — grep -v exiting 1 when it filters out every line (latch would have stuck forever), and the dying bridge's own ERROR line clearing its own latch (fixed by comparing log positions).
Live fault injection, isolated on the build host against the test fixture (prod untouched): blocking the homeserver produced Matrix sync error — retrying in 5 s and zero bridge failed, with the roster staying alive and its updated stamp advancing. A corrupted appservice token produced a real 401 Unauthorized → still fatal → then restarts at 5 s → 10 s → 20 s → 40 s → 80 s, with the roster showing restarting / consecutive_failures=5.

Note on scope of the live test: the injected transport error carried token 55219851, which contains neither 401 nor 403, so that run exercised the retry path rather than the misclassification itself — the token case is covered by the unit tests.

Closes #130, closes #131. ## The bug A transient `/sync` transport error was classified as a **fatal auth failure**, because the classifier substring-matched the *stringified* error — which embeds the request URL, and `/sync` carries `?since=<token>`: ```rust if msg.contains("401") || msg.contains("403") || msg.contains("M_UNKNOWN_TOKEN") { /* fatal */ } ``` On 2026-08-11 the homeserver's sync stream sat at position `5401xxxx`, so every bridge's token (`54013070`, `54014591`, `54015213`, `54015517`) contained `401`. Five bridges died in ten minutes. Three — the static `config.toml` ones — stayed dead for **three days**, because a fatal verdict had no restart path; the tenant bridges came back only when an unrelated link approval triggered a reconcile. ## What changed - **`auth_error`** — a pure `is_fatal_auth(status, errcode)` plus a `matrix_sdk::Error` adapter. A transport error (no HTTP response at all) is never fatal; `401`/`M_UNKNOWN_TOKEN`/`M_MISSING_TOKEN` are, and `M_FORBIDDEN` only alongside a 403. Replaces the substring checks in both sync loops. `AppserviceHttpError` now carries a typed status so the join path stops matching on `"403"` too. - **Supervisor auto-restart** — bridge futures report a `BridgeOutcome`; a failed one (error *or* panic) is respawned after `bridge_restart_base_ms * 2^n`, capped by `bridge_restart_max_ms` (new `[timing]` keys, 5 s → 5 min), streak reset after stable uptime. Operator stops, reconcile replacements and shutdown never auto-restart. This also closes the reboot boot-race where a startup 502 killed every bridge permanently. - **`<data_dir>/bridges.json`** — a roster naming every bridge `alive`/`restarting`/`down`, refreshed every 60 s as a dead-man's switch. - **`ops/`** — `nvb-watch.sh`, `nvb-backup.sh` and a conf example brought into the repo (they lived only on the prod host, unreviewed), with a bridge-liveness section that re-alerts **daily** while a bridge is down and puts `bridges=N/M` in the digest. `ops/test-nvb-watch.sh` drives the real watcher against stubbed `journalctl`/`systemctl`/`curl`. - **CI unblocked** — `generate_config_with_pool_size` lost its last non-test caller in `ad53ccd`, so `clippy --all-targets -D warnings` has failed on master since the 0.3.1 release (run #289, 19 days). Gated behind `cfg(test)`. ## Verification - `cargo test --workspace --locked` — 359 tests, 0 failures; `clippy --all-targets -D warnings` clean. CI #311 green. - New unit tests pin the incident: an error with no HTTP status whose message contains `since=54015213` classifies as **transient**, and the four real tokens are asserted to still trip the *old* substring predicate while the typed one lets them through. - `ops/test-nvb-watch.sh`: 13/13. It caught two real bugs in the new section before it shipped — `grep -v` exiting 1 when it filters out every line (latch would have stuck forever), and the dying bridge's own ERROR line clearing its own latch (fixed by comparing log positions). - **Live fault injection**, isolated on the build host against the test fixture (prod untouched): blocking the homeserver produced `Matrix sync error — retrying in 5 s` and **zero** `bridge failed`, with the roster staying `alive` and its `updated` stamp advancing. A corrupted appservice token produced a real `401 Unauthorized` → still fatal → then restarts at **5 s → 10 s → 20 s → 40 s → 80 s**, with the roster showing `restarting` / `consecutive_failures=5`. Note on scope of the live test: the injected transport error carried token `55219851`, which contains neither `401` nor `403`, so that run exercised the retry path rather than the misclassification itself — the token case is covered by the unit tests.

robocub added 2 commits

2026-08-14 02:06:14 +00:00

fix(matrix,bridge): classify sync errors structurally + auto-restart failed bridges (#130 , #131 )

CI / test (push) Failing after 4m33s

Details

49c753ed28

A transient `/sync` transport error was read as a fatal auth failure because the
classifier substring-matched the stringified error for "401"/"403" — and the
string contains the request URL, whose `?since=` token can hold those digits.
On 2026-08-11 the sync stream sat at 5401xxxx and five bridges died in ten
minutes; the three static ones stayed dead for three days, because a fatal
verdict had no restart path (only a tenants reconcile accidentally revived the
others).

- `auth_error`: pure `is_fatal_auth(status, errcode)` + `matrix_sdk::Error`
  adapter. Transport errors (no HTTP response) are never fatal. Replaces the
  substring checks in both sync loops; `AppserviceHttpError` carries a typed
  status so the join path stops matching on "403" too.
- supervisor: bridge futures report `BridgeOutcome`; a Failed one (error or
  panic) is respawned after `bridge_restart_base_ms * 2^n`, capped, streak reset
  after stable uptime. Operator stops/shutdown never restart. Also closes the
  reboot boot-race where a startup 502 killed every bridge permanently.
- `<data_dir>/bridges.json`: roster of every bridge (alive/restarting/down) for
  the ops monitor, refreshed every 60 s as a dead-man's switch.
- ops/: nvb-watch.sh + nvb-backup.sh + conf example brought into the repo, with
  a bridge-liveness section (daily re-alert while down) and an offline test
  harness. The harness caught two real bugs in that section before it shipped.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

fix(e2e): gate the now-test-only pool-size config helper behind cfg(test)

CI / test (push) Successful in 4m30s

Details

CI / test (pull_request) Successful in 4m37s

Details

310d22547e

`generate_config_with_pool_size` lost its only non-test caller in ad53ccd (#127
moved it to the `_timing` variant), so `cargo clippy --all-targets -D warnings`
— the CI gate — fails on dead code in the bin target. Master's CI has been red
since the 0.3.1 release commit on 2026-07-26 (run #289) and nothing noticed:
nvb-watch's Actions guardrail watches e2e.yml only.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

robocub added 1 commit

2026-08-14 02:08:39 +00:00

release: 0.3.2 — sync-error misclassification fix + bridge liveness (#130 , #131 )

CI / test (push) Successful in 8m28s

Details

CI / test (pull_request) Successful in 4m14s

Details

ac2ac45793

A transient /sync transport error was read as a fatal auth failure because the
classifier substring-matched the stringified error, whose URL carries the
`since=` token — on 2026-08-11 the stream sat at 5401xxxx and five bridges died
in ten minutes, three of them staying dead for three days because a fatal
verdict had no restart path and nothing watched bridge-level liveness.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

robocub referenced this pull request

2026-08-14 02:08:55 +00:00

ops(nvb-watch): no per-bridge liveness check — 3 bridges dead for 3 days while daily digests read healthy #131

robocub merged commit 09d5e7aa1e into master

2026-08-14 17:41:37 +00:00

robocub referenced this pull request from a commit

2026-08-14 17:41:38 +00:00

Merge pull request 'fix: classify sync errors structurally + auto-restart failed bridges (#130, #131)' (#132) from fix/130-sync-auth-misclassification into master