Automated testing roadmap #46

New issue

Open

opened 2026-10-05 21:03:22 +00:00 by robocub · 0 comments

robocub commented

2026-10-05 21:03:22 +00:00

Owner

The goal is the voice bridge's level of automated testing for Vommet: every sync candidate and every topic gets exercised before it reaches testers. Status as of 2026-10-05:

Have

Static: format, codegen, analyze (build.yml).
Unit tests: flutter test unit_test (~74).
Fork invariant: no unlisted *.commet.chat reference (ci/fork-invariants). It found #42.
Integration harness (ci/integration-tests): the Linux app under Xvfb against a throwaway Tuwunel at https://localhost, with a screen recording and logs as artifacts. Nightly on testing. Upstream's two login tests pass. Getting there found three real app bugs (see VOMMET_CHANGES).
Upstream sync gate: promote requires green build (sync/main) and build + android (sync/testing).

Next

Rewrite upstream's rotted tests against today's UI: create space, multi-account, change space name. Key verification needs CommonFlows.createTestClient (an upstream stub): a second Dart SDK client in the test.
Second-actor scenarios: a curl/SDK "bob" that sends messages, images and stickers and creates rooms, so the app under test receives and renders them. Encrypted DM decrypts. Message history after re-login.
Fork feature scenarios, one per topic as it lands:
- Telegram pack import against a local stub of the proxy routes
- sticker rooms (#19)
- room banners (#6)
- media mosaic grouping (#11)
- remove room from space (#13)
- room-picker search (#10)
- copy selection (#9)
- voice-message send path (with an injected Ogg file; no mic in CI)
- SSO-first login (#44): needs an SSO-offering homeserver stub
- push gateway default (#42): assert the registered pusher URL
Network invariant at runtime: run a session through a logging proxy/DNS and assert no request goes to Commet's hosts (complements the static check).
Packaging smoke: Flatpak install + launch to the login screen under Xvfb (done by hand for testing-2026-10-05; automate on vomit-proxy). APK: an emulator needs KVM on the runner, so this needs operator infra; until then an aapt manifest/signature check.
Add integration to the promote gate (REQUIRED_CHECKS_TESTING) once the suite has been green for a week of nightlies.

The goal is the voice bridge's level of automated testing for Vommet: every sync candidate and every topic gets exercised before it reaches testers. Status as of 2026-10-05: **Have** - **Static:** format, codegen, analyze (`build.yml`). - **Unit tests:** `flutter test unit_test` (~74). - **Fork invariant:** no unlisted `*.commet.chat` reference (`ci/fork-invariants`). It found #42. - **Integration harness** (`ci/integration-tests`): the Linux app under Xvfb against a throwaway Tuwunel at https://localhost, with a screen recording and logs as artifacts. Nightly on testing. Upstream's two login tests pass. Getting there found three real app bugs (see VOMMET_CHANGES). - **Upstream sync gate:** promote requires green `build` (sync/main) and `build` + `android` (sync/testing). **Next** 1. Rewrite upstream's rotted tests against today's UI: create space, multi-account, change space name. Key verification needs `CommonFlows.createTestClient` (an upstream stub): a second Dart SDK client in the test. 2. **Second-actor scenarios:** a curl/SDK "bob" that sends messages, images and stickers and creates rooms, so the app under test receives and renders them. Encrypted DM decrypts. Message history after re-login. 3. **Fork feature scenarios**, one per topic as it lands: - Telegram pack import against a local stub of the proxy routes - sticker rooms (#19) - room banners (#6) - media mosaic grouping (#11) - remove room from space (#13) - room-picker search (#10) - copy selection (#9) - voice-message *send path* (with an injected Ogg file; no mic in CI) - SSO-first login (#44): needs an SSO-offering homeserver stub - push gateway default (#42): assert the registered pusher URL 4. **Network invariant at runtime:** run a session through a logging proxy/DNS and assert no request goes to Commet's hosts (complements the static check). 5. **Packaging smoke:** Flatpak install + launch to the login screen under Xvfb (done by hand for testing-2026-10-05; automate on vomit-proxy). APK: an emulator needs KVM on the runner, so this needs operator infra; until then an `aapt` manifest/signature check. 6. Add `integration` to the promote gate (`REQUIRED_CHECKS_TESTING`) once the suite has been green for a week of nightlies.

robocub added the

infra

effort: large

labels

2026-10-05 21:03:22 +00:00