infra: nether.autos incus container — federated-fixture setup runbook (new-session handoff for #106/#108) #109

Open
opened 2026-07-12 14:07:55 +00:00 by robocub · 6 comments
Collaborator

Handoff runbook for a NEW Claude Code session (the #100 pattern — this issue must suffice without any prior chat context). Goal: stand up the nether.autos incus container as the federated test fixture from #106, verify federation with nether.im both directions, and integrate it into the voice-bridge e2e harness. This container doubles as phase-0 prototype for #108.

Operator pre-work (done before the session starts)

  • Incus container on cubuntu (suggest name autos), Ubuntu LTS, modest (2 CPU / 2–4 GB — Tuwunel is light).
  • Tailscale on it; claude-code ssh access (same pattern as the mautrix/tuwunel containers).
  • DNS: nether.autos → Cloudflare; tunnel hostname route → the container (same cloudflared pattern as nether.im — cubuntu has no open inbound; federation hairpins via the CF edge, proven by nether.im's own inbound federation).
  • Cloudflare /.well-known/ cache exclusion for the new hostname (the nether.im OIDC-login lesson: CF interfering with well-known breaks discovery).

Session work

  1. Mirror, don't guess: inspect how prod's tuwunel container is deployed (ssh root@tuwunel — package/systemd/config layout) and replicate the style.
  2. Tuwunel: server_name = "nether.autos", registration CLOSED, federation on. Persist the server signing key separately from wipeable state — fixture wipes must never rotate the key (remote peers cache server keys; a rotated key after a wipe sours federation trust). Document the wipe procedure: drop rooms/users/media, keep key + config.
  3. Appservice registration for the harness: exclusive namespace @autotest_.*:nether.autos, fresh as_token. Never a user-DMable-looking MXID (#96 lesson).
  4. Verification ladder (in order): federation version/key endpoints from BOTH sides → scripted invite/join round trip nether.im↔nether.autos in both directions (the tinier.town failure was directional) → create a v12 (domainless-id) room on the fixture, cross-server join it.
  5. Harness integration (voice-bridge repo — everything stays here for now; own repo only at #108 phase 1): e2e.toml gains a [federation] section (fixture HS url, as_token, domain); scenarios must skip cleanly when the section is absent. First scenarios (per #106): federated DM invite lifecycle (the #97 race with REAL multi-second joins), federated v12 link (the #103 acceptance test, automated), slow-join greeting window, #104 cold-outreach UTD against a genuinely remote crypto user. Later tier: a chaos proxy in front of the fixture mangling make_join/invite responses (deterministic reproduction of the tinier.town class).
  6. Runbook lands as docs/e2e-homeserver.md.

Ground rules for the session

  • The container's INSIDES are a fixture — exempt from the shared-infra rule: restart/wipe/reconfigure freely. Everything OUTSIDE it (cubuntu, tunnel config, DNS, prod tuwunel, prod bridge) remains operator-only / explicit-go territory.
  • Voice-bridge conventions apply: never cargo-build in the workspace container (build on forgejo-runner), staging e2e runs under flock /opt/nvb-staging/e2e-live.lock, no merge/push to master without the operator's explicit word, coordinate cross-session via this issue.
  • Known CF gotchas that WILL bite otherwise: the /.well-known/ exclusion above, and CF user-agent filtering (set a UA on harness/curl probes).

Done means

Federation green both directions incl. a v12 cross-join; harness mints @autotest_* users; at least one federated scenario passing in the suite (and wired into nightly); docs/e2e-homeserver.md merged; evidence posted here.

**Handoff runbook for a NEW Claude Code session** (the #100 pattern — this issue must suffice without any prior chat context). Goal: stand up the `nether.autos` incus container as the federated test fixture from #106, verify federation with nether.im both directions, and integrate it into the voice-bridge e2e harness. This container doubles as phase-0 prototype for #108. ## Operator pre-work (done before the session starts) - Incus container on cubuntu (suggest name `autos`), Ubuntu LTS, modest (2 CPU / 2–4 GB — Tuwunel is light). - Tailscale on it; claude-code ssh access (same pattern as the mautrix/tuwunel containers). - DNS: `nether.autos` → Cloudflare; **tunnel hostname route** → the container (same cloudflared pattern as nether.im — cubuntu has no open inbound; federation hairpins via the CF edge, proven by nether.im's own inbound federation). - **Cloudflare `/.well-known/` cache exclusion for the new hostname** (the nether.im OIDC-login lesson: CF interfering with well-known breaks discovery). ## Session work 1. **Mirror, don't guess**: inspect how prod's tuwunel container is deployed (`ssh root@tuwunel` — package/systemd/config layout) and replicate the style. 2. Tuwunel: `server_name = "nether.autos"`, **registration CLOSED**, federation on. **Persist the server signing key separately from wipeable state** — fixture wipes must never rotate the key (remote peers cache server keys; a rotated key after a wipe sours federation trust). Document the wipe procedure: drop rooms/users/media, keep key + config. 3. Appservice registration for the harness: exclusive namespace `@autotest_.*:nether.autos`, fresh as_token. **Never a user-DMable-looking MXID** (#96 lesson). 4. **Verification ladder** (in order): federation `version`/`key` endpoints from BOTH sides → scripted invite/join round trip nether.im↔nether.autos **in both directions** (the tinier.town failure was directional) → create a **v12 (domainless-id) room** on the fixture, cross-server join it. 5. **Harness integration** (voice-bridge repo — everything stays here for now; own repo only at #108 phase 1): `e2e.toml` gains a `[federation]` section (fixture HS url, as_token, domain); scenarios must skip cleanly when the section is absent. First scenarios (per #106): federated DM invite lifecycle (the #97 race with REAL multi-second joins), federated v12 `link` (the #103 acceptance test, automated), slow-join greeting window, #104 cold-outreach UTD against a genuinely remote crypto user. Later tier: a chaos proxy in front of the fixture mangling `make_join`/`invite` responses (deterministic reproduction of the tinier.town class). 6. Runbook lands as `docs/e2e-homeserver.md`. ## Ground rules for the session - **The container's INSIDES are a fixture — exempt from the shared-infra rule**: restart/wipe/reconfigure freely. Everything OUTSIDE it (cubuntu, tunnel config, DNS, prod tuwunel, prod bridge) remains operator-only / explicit-go territory. - Voice-bridge conventions apply: never cargo-build in the workspace container (build on forgejo-runner), staging e2e runs under `flock /opt/nvb-staging/e2e-live.lock`, no merge/push to master without the operator's explicit word, coordinate cross-session via this issue. - Known CF gotchas that WILL bite otherwise: the `/.well-known/` exclusion above, and CF user-agent filtering (set a UA on harness/curl probes). ## Done means Federation green both directions incl. a v12 cross-join; harness mints `@autotest_*` users; at least one federated scenario passing in the suite (and wired into nightly); `docs/e2e-homeserver.md` merged; evidence posted here.
Author
Collaborator

Pre-work decisions finalized with the operator: container name autos, Ubuntu 24.04 LTS (deliberately matching prod's tuwunel container — verified: 24.04.4 + Tuwunel 1.8.1-1 as a native deb — so "mirror prod's deployment style" means: same OS, same deb packaging). Operator is creating the container + tailscale now; DNS/tunnel/well-known steps may land later — the session can do local homeserver setup first and run the federation verification ladder once the tunnel route exists.

Pre-work decisions finalized with the operator: container name **`autos`**, **Ubuntu 24.04 LTS** (deliberately matching prod's tuwunel container — verified: 24.04.4 + Tuwunel 1.8.1-1 as a native deb — so "mirror prod's deployment style" means: same OS, same deb packaging). Operator is creating the container + tailscale now; DNS/tunnel/well-known steps may land later — the session can do local homeserver setup first and run the federation verification ladder once the tunnel route exists.
Author
Collaborator

Container is LIVE: autos created by the operator, verified reachable via ssh root@autos (tailscale, accept-new host key on first connect). Ubuntu 24.04.4 LTS (prod-matched), 12 cores / 31 GB visible, ZFS root with ~538 GB free, tailscale 1.98.8. Operator pre-work remaining per this runbook: DNS nether.autos → CF, tunnel hostname route, /.well-known/ cache exclusion — the executing session can begin the local homeserver setup (steps 1–3) immediately and run the federation verification ladder once the tunnel lands. This issue is the complete brief; also read MEMORY.md (parallel-instance rules apply if other sessions are active).

✅ **Container is LIVE**: `autos` created by the operator, verified reachable via `ssh root@autos` (tailscale, accept-new host key on first connect). Ubuntu 24.04.4 LTS (prod-matched), 12 cores / 31 GB visible, ZFS root with ~538 GB free, tailscale 1.98.8. Operator pre-work remaining per this runbook: DNS `nether.autos` → CF, tunnel hostname route, `/.well-known/` cache exclusion — the executing session can begin the local homeserver setup (steps 1–3) immediately and run the federation verification ladder once the tunnel lands. This issue is the complete brief; also read MEMORY.md (parallel-instance rules apply if other sessions are active).
Author
Collaborator

🔒 Session claim (2026-07-12): a Claude Code session is now executing this runbook end-to-end. Plan: Part A (Tuwunel 1.8.1 prod-mirrored on autos + autotest appservice + golden-image key-preserving wipe) proceeds now; cloudflared goes INSIDE the container once the operator drops a tunnel token (prod pattern confirmed — cloudflared runs in the tuwunel container → localhost:8008). Harness side lands on branch feat/109-federation-fixture: first-class skipped outcome, flat fed_* config keys (env↔file bijection; staging e2e.toml is flat TOML — table avoided deliberately), AsHomeserver actor split, scenarios fed_dm_invite_lifecycle (Pass, done-gate) + fed_cold_outreach_utd (KnownFail #104) + fed_link_v12 (#103 acceptance); fed_slow_join_greeting deferred (needs latency injection). Other sessions: crates/e2e/** and this fixture are claimed by this session until a /handoff comment lands here.

🔒 **Session claim (2026-07-12):** a Claude Code session is now executing this runbook end-to-end. Plan: Part A (Tuwunel 1.8.1 prod-mirrored on `autos` + `autotest` appservice + golden-image key-preserving wipe) proceeds now; cloudflared goes INSIDE the container once the operator drops a tunnel token (prod pattern confirmed — cloudflared runs in the tuwunel container → localhost:8008). Harness side lands on branch `feat/109-federation-fixture`: first-class `skipped` outcome, flat `fed_*` config keys (env↔file bijection; staging e2e.toml is flat TOML — table avoided deliberately), `AsHomeserver` actor split, scenarios `fed_dm_invite_lifecycle` (Pass, done-gate) + `fed_cold_outreach_utd` (KnownFail #104) + `fed_link_v12` (#103 acceptance); `fed_slow_join_greeting` deferred (needs latency injection). Other sessions: crates/e2e/** and this fixture are claimed by this session until a ✅/handoff comment lands here.
Author
Collaborator

Progress — fixture up, harness code pushed

Part A (steps 1–4) DONE on autos:

  • Tuwunel 1.8.1-1 (upstream x86-64-v3 deb, prod-matched) installed + running; server_name nether.autos, registration closed, federation on, listener 0.0.0.0:8008, well-known served by Tuwunel (server = nether.autos:443, CF-edge hairpin like prod).
  • autotest appservice registered (passive url: null, exclusive ^@autotest_.*:nether\.autos$, fresh 64-hex tokens, sender_localpart autotest_bot) — smoke-minted @autotest_smoke:nether.autos ✓.
  • Signing-key persistence drilled: golden image /var/lib/tuwunel-golden.tar.gz taken pristine after first boot; full wipe (rm DB → restore) → key ed25519:WiQzDgvI identical, smoke user gone. Baseline at /root/signing-key-baseline.txt. Wipe procedure documented in docs/e2e-homeserver.md.

Part B (harness) on branch feat/109-federation-fixture (3 commits, pushed; hermetic CI running):

  • First-class skipped outcome (env-gated Skip beats the expectation; suite-neutral) + summary counter.
  • Flat fed_* config group → AsHomeserver; absent = skip, partial = loud load error; register_on() split in both Matrix actors (23 existing call sites untouched).
  • Scenarios (suite 23→26): fed_dm_invite_lifecycle (Pass — #97 over real federation, the done-gate), fed_cold_outreach_utd (KnownFail #104), fed_link_v12 (Pass — #103 acceptance, v12 createRoom helper). fed_slow_join_greeting deferred (needs the chaos-proxy tier).
  • e2e.yml wires NVB_E2E_FED_* (url dropped in-shell when the secret is absent → skip, not config error). Forgejo secret NVB_E2E_FED_AS_TOKEN still to be added (I will, before the live CI run).
  • docs/e2e-homeserver.md + testing-design federation section rewritten. cargo check/test/clippy -p nvb-e2e green on forgejo-runner (21/21).

Staging: branch syncing at /opt/nvb-staging/src109 (build109 running); next: skip-run without fed_* keys (expect skipped, exit 0).

BLOCKED on operator pre-work (per runbook): DNS nether.autos → CF, tunnel hostname route → http://localhost:8008 in the container, /.well-known/ cache exclusion, and a tunnel token — drop it at /root/tunnel-token on autos (or comment here) and I install cloudflared inside the container (prod pattern) and run the full verification ladder + live fed scenarios.

## Progress — fixture up, harness code pushed **Part A (steps 1–4) DONE on `autos`:** - Tuwunel **1.8.1-1** (upstream x86-64-v3 deb, prod-matched) installed + running; `server_name nether.autos`, registration closed, federation on, listener `0.0.0.0:8008`, well-known served by Tuwunel (`server = nether.autos:443`, CF-edge hairpin like prod). - `autotest` appservice registered (passive `url: null`, exclusive `^@autotest_.*:nether\.autos$`, fresh 64-hex tokens, `sender_localpart autotest_bot`) — smoke-minted `@autotest_smoke:nether.autos` ✓. - **Signing-key persistence drilled:** golden image `/var/lib/tuwunel-golden.tar.gz` taken pristine after first boot; full wipe (rm DB → restore) → key `ed25519:WiQzDgvI` **identical**, smoke user gone. Baseline at `/root/signing-key-baseline.txt`. Wipe procedure documented in `docs/e2e-homeserver.md`. **Part B (harness) on branch `feat/109-federation-fixture`** (3 commits, pushed; hermetic CI running): - First-class `skipped` outcome (env-gated Skip beats the expectation; suite-neutral) + summary counter. - Flat `fed_*` config group → `AsHomeserver`; absent = skip, partial = loud load error; `register_on()` split in both Matrix actors (23 existing call sites untouched). - Scenarios (suite 23→26): `fed_dm_invite_lifecycle` (Pass — #97 over real federation, the done-gate), `fed_cold_outreach_utd` (KnownFail #104), `fed_link_v12` (Pass — #103 acceptance, v12 createRoom helper). `fed_slow_join_greeting` deferred (needs the chaos-proxy tier). - `e2e.yml` wires `NVB_E2E_FED_*` (url dropped in-shell when the secret is absent → skip, not config error). Forgejo secret `NVB_E2E_FED_AS_TOKEN` still to be added (I will, before the live CI run). - `docs/e2e-homeserver.md` + testing-design federation section rewritten. `cargo check/test/clippy -p nvb-e2e` green on forgejo-runner (21/21). **Staging:** branch syncing at `/opt/nvb-staging/src109` (build109 running); next: skip-run without `fed_*` keys (expect `skipped`, exit 0). **⏳ BLOCKED on operator pre-work** (per runbook): DNS `nether.autos` → CF, tunnel hostname route → `http://localhost:8008` in the container, `/.well-known/` cache exclusion, and a **tunnel token** — drop it at `/root/tunnel-token` on `autos` (or comment here) and I install cloudflared inside the container (prod pattern) and run the full verification ladder + live fed scenarios.
Author
Collaborator

Deliverable-1 verified live (pre-tunnel), branch CI green. Staging (/opt/nvb-staging, bins *-109, under flock):

  • No fed_* keys → all three fed scenarios report skipped in 0.0s, exit 0 (suite-neutral; message points at the keys + docs/e2e-homeserver.md).
  • Partial group (url only) → loud config error, exit 1 ("must both be set when any fed_* key is present") — a typo can never demote a federated run to silently-skipped.
  • Hermetic CI (test) on feat/109-federation-fixture: success (incl. the new Skip/config unit tests, 21/21).

Everything actionable without the tunnel is now done. Waiting on the operator pre-work (DNS + tunnel route + /.well-known/ exclusion + token at /root/tunnel-token on autos) → then A5 cloudflared, the A6 ladder (both directions + v12 cross-join), live fed runs + full sweep, the NVB_E2E_FED_AS_TOKEN Forgejo secret, and a wipe→re-run drill.

**Deliverable-1 verified live (pre-tunnel), branch CI green.** Staging (`/opt/nvb-staging`, bins `*-109`, under flock): - No `fed_*` keys → all three fed scenarios report **`skipped` in 0.0s, exit 0** (suite-neutral; message points at the keys + `docs/e2e-homeserver.md`). - Partial group (url only) → **loud config error, exit 1** ("must both be set when any fed_* key is present") — a typo can never demote a federated run to silently-skipped. - Hermetic CI (`test`) on `feat/109-federation-fixture`: ✅ success (incl. the new Skip/config unit tests, 21/21). Everything actionable without the tunnel is now done. Waiting on the operator pre-work (DNS + tunnel route + `/.well-known/` exclusion + token at `/root/tunnel-token` on `autos`) → then A5 cloudflared, the A6 ladder (both directions + v12 cross-join), live fed runs + full sweep, the `NVB_E2E_FED_AS_TOKEN` Forgejo secret, and a wipe→re-run drill.
Author
Collaborator

Done — every "done means" item, with evidence

Merged to master 25138b1 (operator-approved after green gates). The fixture caught three real bridge bugs within its first hour live#110 (fixed), #111 (open, product decision), #112 (fixed) — the exact class 23 same-server scenarios were structurally blind to.

Done-means checklist

  • Federation green both directions incl. a v12 cross-join ✓ — well-known/versions via the CF edge (cache-status DYNAMIC, no redirects — no cache rule needed: nether.im's rule is a website-redirect exclusion, N/A here); federation/v1/version + key/v2/server probed FROM each side; scripted invite/join round trips both directions; v12 domainless room minted on the fixture and cross-joined from nether.im.
  • Harness mints @autotest_* users ✓ — passive autotest appservice (exclusive ^@autotest_.*:nether\.autos$), exercised by every fed run + CI.
  • At least one federated scenario passing, wired into nightly ✓✓ — fed_dm_invite_lifecycle pass (4× incl. post-wipe + CI dispatch) and fed_link_v12 pass; fed_cold_outreach_utd = KnownFail{#104} now pinning the REAL cross-server UTD. Secret NVB_E2E_FED_AS_TOKEN set (operator); nightly --all picks all three up; CI workflow_dispatch leg verified green (fixture journal shows the run's autotest_remote/autotest_linker activity).
  • docs/e2e-homeserver.md merged ✓ — deploy, autotest registration, golden-image wipe procedure (drilled twice: key ed25519:WiQzDgvI identical across full DB wipes; post-wipe fed scenario green in 28.4 s), verification ladder, fed_* config semantics (skip when absent — verified live: 3× skipped exit 0; partial group = loud error — verified live). Testing-design federation section rewritten as the coverage map.

Also from tonight

  • Full 26-scenario sweep: fed/mgmt/lifecycle green; audio-block failures forensically attributed to environment (CI-build contention + Discord voice weather after ~45 runs): same-binary pass↔fail flips + Jul-11 control failing identically on audio_d2m_enc. mgmt_dm_pending_reap unexpected-passed once (its #104 mode is flaky same-server; the fed twin pins it deterministically now).
  • Tuwunel federation report enriched with tonight's control experiments (Synapse control, fixture 1.8.1↔1.8.1 control, forged-signature probe) — hold-until-dau-answers gate in the internal notes.
  • Gotcha for posterity (now in the runbook + memory): git archive stamps sources with COMMIT time → copying a warm target/ alongside makes cargo rebuild NOTHING — verify binary identity (grep -ac a marker string) before trusting an A/B.

Fixture claim releasedautos is standing infra now; wipe freely per the runbook. #111 and #104 remain open; the #106 chaos-proxy tier is the natural next step (first candidate: replay the tinier.town 200 {} shape deterministically).

## ✅ Done — every "done means" item, with evidence **Merged to master `25138b1`** (operator-approved after green gates). The fixture caught **three real bridge bugs within its first hour live** — #110 (fixed), #111 (open, product decision), #112 (fixed) — the exact class 23 same-server scenarios were structurally blind to. ### Done-means checklist - **Federation green both directions incl. a v12 cross-join** ✓ — well-known/versions via the CF edge (cache-status DYNAMIC, no redirects — no cache rule needed: nether.im's rule is a website-redirect exclusion, N/A here); `federation/v1/version` + `key/v2/server` probed FROM each side; scripted invite/join round trips **both directions**; v12 domainless room minted on the fixture and cross-joined from nether.im. - **Harness mints `@autotest_*` users** ✓ — passive `autotest` appservice (exclusive `^@autotest_.*:nether\.autos$`), exercised by every fed run + CI. - **At least one federated scenario passing, wired into nightly** ✓✓ — `fed_dm_invite_lifecycle` **pass** (4× incl. post-wipe + CI dispatch) and `fed_link_v12` **pass**; `fed_cold_outreach_utd` = KnownFail{#104} now pinning the REAL cross-server UTD. Secret `NVB_E2E_FED_AS_TOKEN` set (operator); nightly `--all` picks all three up; CI `workflow_dispatch` leg verified green (fixture journal shows the run's `autotest_remote`/`autotest_linker` activity). - **`docs/e2e-homeserver.md` merged** ✓ — deploy, `autotest` registration, **golden-image wipe procedure** (drilled twice: key `ed25519:WiQzDgvI` identical across full DB wipes; post-wipe fed scenario green in 28.4 s), verification ladder, `fed_*` config semantics (skip when absent — verified live: 3× `skipped` exit 0; partial group = loud error — verified live). Testing-design federation section rewritten as the coverage map. ### Also from tonight - Full 26-scenario sweep: fed/mgmt/lifecycle green; audio-block failures forensically attributed to environment (CI-build contention + Discord voice weather after ~45 runs): same-binary pass↔fail flips + Jul-11 control failing identically on `audio_d2m_enc`. `mgmt_dm_pending_reap` unexpected-passed once (its #104 mode is flaky same-server; the fed twin pins it deterministically now). - Tuwunel federation report enriched with tonight's control experiments (Synapse control, fixture 1.8.1↔1.8.1 control, forged-signature probe) — hold-until-dau-answers gate in the internal notes. - Gotcha for posterity (now in the runbook + memory): `git archive` stamps sources with COMMIT time → copying a warm `target/` alongside makes cargo rebuild NOTHING — verify binary identity (`grep -ac` a marker string) before trusting an A/B. **Fixture claim released** — `autos` is standing infra now; wipe freely per the runbook. #111 and #104 remain open; the #106 chaos-proxy tier is the natural next step (first candidate: replay the tinier.town `200 {}` shape deterministically).
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
dark/nether-voicebridge#109
No description provided.