fix(supervisor): retry appservice login at startup — a homeserver-unreachable boot race permanently kills every bridge until manual restart #126

New issue

Open

opened 2026-07-26 01:41:52 +00:00 by robocub · 0 comments

robocub commented

2026-07-26 01:41:52 +00:00

Collaborator

Failure mode

If the host running the bridge reboots and the bridge process comes up before the homeserver is reachable (or the homeserver's front proxy briefly returns 5xx), every bridge's startup m.login.application_service fails:

ERROR bridge{name=<bridge>}: appservice login failed — ... status=502 Bad Gateway
ERROR bridge{name=<bridge>}: supervisor: bridge failed error=appservice login failed: 502
INFO  nvb_discord::puppet: channel unregistered from puppet pool bridge="<bridge>"

Every configured bridge plus the management bot dies within ~1s of startup and never retries — the outage lasts until an operator manually restarts the service. (Observed live 2026-07-22 after a host reboot.)

The failure mode is nasty because it's masked: the Discord puppet bots reconnect fine, so systemctl is-active reports active and the process looks healthy while zero bridging is happening in either direction.

Root cause

The supervisor treats a failed startup appservice login as a terminal bridge error. That's the wrong call for a network-shaped error: a 502/timeout at boot means "homeserver not up yet", not "config is wrong". The same applies to the management bot, which exits with management-bot appservice login.

On appservice login failure with a retryable status (5xx, connect error, timeout), retry with capped exponential backoff (e.g. 5s → 10s → 20s → ... cap 60s, indefinitely or for a generous window), keeping the bridge in a "starting" state instead of failing it. Non-retryable statuses (401/403 = bad token, M_FORBIDDEN, 404 wrong URL) should still fail fast and loudly — those are config errors.

Same treatment for the management bot's login.

Why not `After=`/`Wants=` systemd ordering instead?

The homeserver generally doesn't run on the same host as the bridge, so there may be no local unit to order against.
A proxy-fronted homeserver can intermittently 502 even in steady state. A bridge that dies permanently on one 502 at startup is fragile regardless of boot ordering; the retry fixes the whole class.
A Restart=on-failure band-aid also doesn't fit: the process as a whole stays up (puppets alive), so systemd never sees a failure.

Acceptance

Start the bridge with the homeserver unreachable, restore reachability ~2 min later → all bridges + management bot come up on their own, no manual restart.
Startup with a wrong as_token still fails fast with a clear error (no infinite silent retry).

## Failure mode If the host running the bridge reboots and the bridge process comes up before the homeserver is reachable (or the homeserver's front proxy briefly returns 5xx), every bridge's startup `m.login.application_service` fails: ``` ERROR bridge{name=<bridge>}: appservice login failed — ... status=502 Bad Gateway ERROR bridge{name=<bridge>}: supervisor: bridge failed error=appservice login failed: 502 INFO nvb_discord::puppet: channel unregistered from puppet pool bridge="<bridge>" ``` Every configured bridge plus the management bot dies within ~1s of startup and **never retries** — the outage lasts until an operator manually restarts the service. (Observed live 2026-07-22 after a host reboot.) The failure mode is nasty because it's **masked**: the Discord puppet bots reconnect fine, so `systemctl is-active` reports `active` and the process looks healthy while zero bridging is happening in either direction. ## Root cause The supervisor treats a failed startup appservice login as a **terminal** bridge error. That's the wrong call for a *network-shaped* error: a 502/timeout at boot means "homeserver not up yet", not "config is wrong". The same applies to the management bot, which exits with `management-bot appservice login`. ## Proposed fix: login retry/backoff in the supervisor On appservice login failure with a **retryable** status (5xx, connect error, timeout), retry with capped exponential backoff (e.g. 5s → 10s → 20s → ... cap 60s, indefinitely or for a generous window), keeping the bridge in a "starting" state instead of failing it. Non-retryable statuses (401/403 = bad token, `M_FORBIDDEN`, 404 wrong URL) should still fail fast and loudly — those *are* config errors. Same treatment for the management bot's login. ### Why not `After=`/`Wants=` systemd ordering instead? - The homeserver generally doesn't run on the same host as the bridge, so there may be no local unit to order against. - A proxy-fronted homeserver can intermittently 502 even in steady state. A bridge that dies permanently on one 502 at startup is fragile regardless of boot ordering; the retry fixes the whole class. - A `Restart=on-failure` band-aid also doesn't fit: the process as a whole stays up (puppets alive), so systemd never sees a failure. ## Acceptance - Start the bridge with the homeserver unreachable, restore reachability ~2 min later → all bridges + management bot come up on their own, no manual restart. - Startup with a wrong `as_token` still fails fast with a clear error (no infinite silent retry).