Automatic Testing #38
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
We should have a dedicated discord server with discord bots that do automatic testing for regressions every time we make major changes to the bridge. Things like simulating a large amount of participants speaking (perhaps playing well-known audio files). Make sure overflow works, ensuring there are no dangling ghost users. We should have a set of dedicated encrypted and unencrypted matirx voice rooms that correspond to these discord voice channels.
This could help spot regressions and debug, and we'd want to make this testing framework flexible enough to handle eventual self-bots for livestream testing, and perhaps eventual support for other platforms like Signal or Telegram.
Note: This might also help us spot tricky audio corruption (#32) issues. We may want several sets of tests depending on how we want to simulate users talking to each other.
We should spend some time thinking about the 'standard algorithm' of matrix joins, discord joins, channel joins, moving between channels, exchanging audio, etc so that we cover all possible permutations / regressions for the bridge's interactions.
Note: We should also brainstorm other scenarios, for example someone just moved the bot to another channel, how should the bridge respond?
robocub referenced this issue2026-06-23 22:06:16 +00:00
Design v2 committed (merged to master
2bf32d3,docs/nether-voicebridge-testing-design.md) — the 2026-06-23 feasibility assessment is now a committed, phased implementation plan. Key updates since v1: the external-disconnect gap is already fixed (#46) and becomes a must-pass regression guard; the supervisor control plane (#4) is the lifecycle seam for the live harness; Matrix fake users come from a dedicated@nvbtest_*appservice (masquerade) instead of password accounts; MFCC is dropped from the audio oracle (time-domain metrics + dropout/glitch detectors, zero new deps).Implementation checklist:
cargo test+ clippy (.forgejo/workflows/ci.yml, merged8687459)crates/audiooracle.rs/probe.rs, deterministic probe synthesis, negative self-tests)[timing]config threading (mergede775730; smoke-tested on staging: defaults byte-identical — 12.002s leave-grace, no WARN — andNVB_TIMING=compressed→ loud WARN + 2.002s leave-grace)test_*seams (merged4fd83ae; +6 scenarios incl. paused-time anchor-idle walks, 247 tests total)nvb-e2ecrate: bridge-under-test child process, LogScraper (JSON logs), RoomStatePoller, ConnCountProbe, baseline pre/postcondition, smoke scenario (merged968da57; livesmoke_open_closePASSED in 11.9s against the standing test env with prod running — dedicated harness bots, no token overlap)testkitfeature innvb-matrix-rtc: fake Matrix speakers/phantoms/listeners + audio-crossing scenarios (oracle-verified)e2e.ymlworkflowOperator provisioning (needed from Phase D, runbook in the doc): 2-3 speaker bots + 1 mod bot + 2-3 harness puppet bots,
nvbtestappservice registration on Tuwunel, test users invited to the Automated Testing space, unlink the 3 Auto Test tenants from prod before the first live run.Known-fail companion issue: puppet not re-joining after a mod force-move (filed separately; catalogue scenario 10).
Update 2026-07-06: Phase B merged (
e6de6ee) — oracle + deterministic probes innvb-audio, 49 tests, CI run 73 green. Next: Phase C1 timing config.Closing — the framework is complete and battle-proven.
What shipped (Phases A–D, 2026-07-06 → 2026-07-11)
ci.yml— workspace tests + clippy-D warnings, cached).nvb-audioprobe/oracle, per-cycle NCC + robust aggregates),TimingConfigtwo-speed timers (NVB_TIMING=compressed), named hermetic scenarios inpuppet.rs/matrix.rs/supervisor.rs.nvb-e2elive orchestrator — real binary under test, JSON log scraping, out-of-band Matrix/Discord observers, Matrix testkit actors (speaker/phantom/listener incl. the decrypting E2EE listener), Discord actors (speaker/listener/mod), management-client actor, secrets-safe artifacts,report.json.recovery_no_spare), all live-verified green; nightlye2e.ymlruns the compressed suite + the real-timing subset (#8/#13).Proven in anger — bugs it caught that reached or would have reached users
JoinError::Droppedrace (recovery silently degraded to failover; no-spare case = no recovery)Standing guard
The 06:00 UTC nightly is the ongoing regression net (red = triage ticket, never a merge gate). Follow-ups tracked separately: #24 (possible cross-channel steal/re-promotion scenario — all building blocks now exist), #90 (build-host isolation), #49 (arm64 runner).