Automatic Testing #38

Closed
opened 2026-06-23 17:00:41 +00:00 by dark · 4 comments
Owner

We should have a dedicated discord server with discord bots that do automatic testing for regressions every time we make major changes to the bridge. Things like simulating a large amount of participants speaking (perhaps playing well-known audio files). Make sure overflow works, ensuring there are no dangling ghost users. We should have a set of dedicated encrypted and unencrypted matirx voice rooms that correspond to these discord voice channels.

This could help spot regressions and debug, and we'd want to make this testing framework flexible enough to handle eventual self-bots for livestream testing, and perhaps eventual support for other platforms like Signal or Telegram.

Note: This might also help us spot tricky audio corruption (#32) issues. We may want several sets of tests depending on how we want to simulate users talking to each other.

We should have a dedicated discord server with discord bots that do automatic testing for regressions every time we make major changes to the bridge. Things like simulating a large amount of participants speaking (perhaps playing well-known audio files). Make sure overflow works, ensuring there are no dangling ghost users. We should have a set of dedicated encrypted and unencrypted matirx voice rooms that correspond to these discord voice channels. This could help spot regressions and debug, and we'd want to make this testing framework flexible enough to handle eventual self-bots for livestream testing, and perhaps eventual support for other platforms like Signal or Telegram. Note: This might also help us spot tricky audio corruption (#32) issues. We may want several sets of tests depending on how we want to simulate users talking to each other.
Author
Owner

We should spend some time thinking about the 'standard algorithm' of matrix joins, discord joins, channel joins, moving between channels, exchanging audio, etc so that we cover all possible permutations / regressions for the bridge's interactions.

We should spend some time thinking about the 'standard algorithm' of matrix joins, discord joins, channel joins, moving between channels, exchanging audio, etc so that we cover all possible permutations / regressions for the bridge's interactions.
Author
Owner

Note: We should also brainstorm other scenarios, for example someone just moved the bot to another channel, how should the bridge respond?

Note: We should also brainstorm other scenarios, for example someone just moved the bot to another channel, how should the bridge respond?
dark added this to the Roadmap project 2026-06-27 16:29:34 +00:00
Collaborator

Design v2 committed (merged to master 2bf32d3, docs/nether-voicebridge-testing-design.md) — the 2026-06-23 feasibility assessment is now a committed, phased implementation plan. Key updates since v1: the external-disconnect gap is already fixed (#46) and becomes a must-pass regression guard; the supervisor control plane (#4) is the lifecycle seam for the live harness; Matrix fake users come from a dedicated @nvbtest_* appservice (masquerade) instead of password accounts; MFCC is dropped from the audio oracle (time-domain metrics + dropout/glitch detectors, zero new deps).

Implementation checklist:

  • Phase A — per-PR CI job: cargo test + clippy (.forgejo/workflows/ci.yml, merged 8687459)
  • Phase B — Tier 1: hermetic audio-integrity oracle (crates/audio oracle.rs/probe.rs, deterministic probe synthesis, negative self-tests)
  • Phase C1[timing] config threading (merged e775730; smoke-tested on staging: defaults byte-identical — 12.002s leave-grace, no WARN — and NVB_TIMING=compressed → loud WARN + 2.002s leave-grace)
  • Phase C2 — Tier 2: named allocation/lifecycle/reaper scenario tests over the existing test_* seams (merged 4fd83ae; +6 scenarios incl. paused-time anchor-idle walks, 247 tests total)
  • Phase D1nvb-e2e crate: bridge-under-test child process, LogScraper (JSON logs), RoomStatePoller, ConnCountProbe, baseline pre/postcondition, smoke scenario (merged 968da57; live smoke_open_close PASSED in 11.9s against the standing test env with prod running — dedicated harness bots, no token overlap)
  • Phase D2testkit feature in nvb-matrix-rtc: fake Matrix speakers/phantoms/listeners + audio-crossing scenarios (oracle-verified)
  • Phase D3 — mod-bot adversarial actor, full scenario catalogue v1 (16 scenarios), nightly e2e.yml workflow

Operator provisioning (needed from Phase D, runbook in the doc): 2-3 speaker bots + 1 mod bot + 2-3 harness puppet bots, nvbtest appservice registration on Tuwunel, test users invited to the Automated Testing space, unlink the 3 Auto Test tenants from prod before the first live run.

Known-fail companion issue: puppet not re-joining after a mod force-move (filed separately; catalogue scenario 10).

Update 2026-07-06: Phase B merged (e6de6ee) — oracle + deterministic probes in nvb-audio, 49 tests, CI run 73 green. Next: Phase C1 timing config.

**Design v2 committed** (merged to master `2bf32d3`, [`docs/nether-voicebridge-testing-design.md`](https://nether.codes/dark/nether-voicebridge/src/branch/master/docs/nether-voicebridge-testing-design.md)) — the 2026-06-23 feasibility assessment is now a committed, phased implementation plan. Key updates since v1: the external-disconnect gap is already fixed (#46) and becomes a must-pass regression guard; the supervisor control plane (#4) is the lifecycle seam for the live harness; Matrix fake users come from a dedicated `@nvbtest_*` appservice (masquerade) instead of password accounts; MFCC is dropped from the audio oracle (time-domain metrics + dropout/glitch detectors, zero new deps). Implementation checklist: - [x] **Phase A** — per-PR CI job: `cargo test` + clippy (`.forgejo/workflows/ci.yml`, merged `8687459`) - [x] **Phase B** — Tier 1: hermetic audio-integrity oracle (`crates/audio` `oracle.rs`/`probe.rs`, deterministic probe synthesis, negative self-tests) - [x] **Phase C1** — `[timing]` config threading (merged `e775730`; smoke-tested on staging: defaults byte-identical — 12.002s leave-grace, no WARN — and `NVB_TIMING=compressed` → loud WARN + 2.002s leave-grace) - [x] **Phase C2** — Tier 2: named allocation/lifecycle/reaper scenario tests over the existing `test_*` seams (merged `4fd83ae`; +6 scenarios incl. paused-time anchor-idle walks, 247 tests total) - [x] **Phase D1** — `nvb-e2e` crate: bridge-under-test child process, LogScraper (JSON logs), RoomStatePoller, ConnCountProbe, baseline pre/postcondition, smoke scenario (merged `968da57`; live `smoke_open_close` PASSED in 11.9s against the standing test env with prod running — dedicated harness bots, no token overlap) - [ ] **Phase D2** — `testkit` feature in `nvb-matrix-rtc`: fake Matrix speakers/phantoms/listeners + audio-crossing scenarios (oracle-verified) - [ ] **Phase D3** — mod-bot adversarial actor, full scenario catalogue v1 (16 scenarios), nightly `e2e.yml` workflow Operator provisioning (needed from Phase D, runbook in the doc): 2-3 speaker bots + 1 mod bot + 2-3 harness puppet bots, `nvbtest` appservice registration on Tuwunel, test users invited to the Automated Testing space, unlink the 3 Auto Test tenants from prod before the first live run. Known-fail companion issue: puppet not re-joining after a mod force-move (filed separately; catalogue scenario 10). *Update 2026-07-06: Phase B merged (`e6de6ee`) — oracle + deterministic probes in `nvb-audio`, 49 tests, CI run 73 green. Next: Phase C1 timing config.*
Collaborator

Closing — the framework is complete and battle-proven.

What shipped (Phases A–D, 2026-07-06 → 2026-07-11)

  • Tier 1/CI: per-PR hermetic gate (ci.yml — workspace tests + clippy -D warnings, cached).
  • Tier 2: audio oracle (nvb-audio probe/oracle, per-cycle NCC + robust aggregates), TimingConfig two-speed timers (NVB_TIMING=compressed), named hermetic scenarios in puppet.rs/matrix.rs/supervisor.rs.
  • Tier 3: the nvb-e2e live orchestrator — real binary under test, JSON log scraping, out-of-band Matrix/Discord observers, Matrix testkit actors (speaker/phantom/listener incl. the decrypting E2EE listener), Discord actors (speaker/listener/mod), management-client actor, secrets-safe artifacts, report.json.
  • Catalogue: 17 live scenarios (the full 16-row design catalogue + recovery_no_spare), all live-verified green; nightly e2e.yml runs the compressed suite + the real-timing subset (#8/#13).

Proven in anger — bugs it caught that reached or would have reached users

  • #73 decryptor wedge (deterministic repro + fix validation)
  • #81 same-bot rejoin JoinError::Dropped race (recovery silently degraded to failover; no-spare case = no recovery)
  • #74 shutdown voice-leave not flushed (~90 s zombie puppet)
  • #91 LiveKit ICE route lottery — prod audio silently broken ~2/3 per connection since June 19; the suite's pass-rate collapse was the detection signal
  • plus multiple harness/scenario bugs caught pre-merge by the framework's own discipline.

Standing guard

The 06:00 UTC nightly is the ongoing regression net (red = triage ticket, never a merge gate). Follow-ups tracked separately: #24 (possible cross-channel steal/re-promotion scenario — all building blocks now exist), #90 (build-host isolation), #49 (arm64 runner).

Closing — the framework is complete and battle-proven. ## What shipped (Phases A–D, 2026-07-06 → 2026-07-11) - **Tier 1/CI**: per-PR hermetic gate (`ci.yml` — workspace tests + clippy `-D warnings`, cached). - **Tier 2**: audio oracle (`nvb-audio` probe/oracle, per-cycle NCC + robust aggregates), `TimingConfig` two-speed timers (`NVB_TIMING=compressed`), named hermetic scenarios in `puppet.rs`/`matrix.rs`/`supervisor.rs`. - **Tier 3**: the `nvb-e2e` live orchestrator — real binary under test, JSON log scraping, out-of-band Matrix/Discord observers, Matrix testkit actors (speaker/phantom/listener incl. the decrypting E2EE listener), Discord actors (speaker/listener/mod), management-client actor, secrets-safe artifacts, `report.json`. - **Catalogue: 17 live scenarios** (the full 16-row design catalogue + `recovery_no_spare`), all live-verified green; nightly `e2e.yml` runs the compressed suite + the real-timing subset (#8/#13). ## Proven in anger — bugs it caught that reached or would have reached users - **#73** decryptor wedge (deterministic repro + fix validation) - **#81** same-bot rejoin `JoinError::Dropped` race (recovery silently degraded to failover; no-spare case = no recovery) - **#74** shutdown voice-leave not flushed (~90 s zombie puppet) - **#91** LiveKit ICE route lottery — **prod audio silently broken ~2/3 per connection since June 19**; the suite's pass-rate collapse was the detection signal - plus multiple harness/scenario bugs caught pre-merge by the framework's own discipline. ## Standing guard The 06:00 UTC nightly is the ongoing regression net (red = triage ticket, never a merge gate). Follow-ups tracked separately: #24 (possible cross-channel steal/re-promotion scenario — all building blocks now exist), #90 (build-host isolation), #49 (arm64 runner).
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
dark/nether-voicebridge#38
No description provided.