dark/nether-voicebridge
1
0
Fork 1
Code Issues 37 Pull requests Projects 1 Releases 3 Packages 1 Activity Actions

fix(e2e): retry a Pass-expected scenario once on transient failure (#124) #125

Merged
robocub merged 1 commit from fix/124-e2e-flaky-retry into master 2026-07-21 01:38:15 +00:00
Conversation 1 Commits 1 Files changed 2 +92 -4
robocub commented 2026-07-19 23:23:25 +00:00
Collaborator
Copy link

What

Retry a Pass-expected e2e scenario once on a transient failure before recording it. KnownFail scenarios never retry; a Skip stays terminal.

Why (#124 root cause)

The nightly e2e.yml suite has been red since ~07-11. I root-caused it with clean-tree builds (git archive, per the #123 rule) on a quiet runner:

  • Not a bridge regression. The core discord/audio/matrix-rtc code is unchanged since the last green nightly (only two recovery fixes that predate the scenarios testing them, plus an additive guild_footprint() for #94). 48/48 hermetic puppet-allocation tests pass on master.
  • Every cluster scenario passes standalone on a clean quiet host: steal_release_repromotion, multi_bridge_isolation, external_disconnect_recovery, recovery_no_spare all green in isolation.
  • They only fail inside the full --all run while infra is flapping. In a full run I captured 23 pass / 3 fail / 2 known-fail, where the 3 fails were:
    • audio_d2m_enc and steal_release_repromotion — a phantom's Matrix membership-heartbeat dropped mid-run (bridge log shows Carol/Alice memberships vanishing ~3 s after join, collapsing the multi-actor choreography). nether.im was returning intermittent 502s during the run (they also caused both #104 known-fails).
    • multi_bridge_isolation — the bridge released its puppet and left voice on time (per its log), but the harness listener never observed the leave within 60 s → Discord voice-state propagation lag at the tail of a 30-scenario suite on 3 shared bots.
    • audio_d2m_enc re-ran with a different ncc (0.253 → 0.500) — non-deterministic, i.e. flaky, not a hard bug.

So the cluster is infra-induced flakiness in a live triage suite, not a code fault. A single retry (each scenario re-sweeps memberships on entry, so it starts clean) turns a transient blip green while a real regression still fails both attempts.

Change

  • run_scenarios: per-scenario attempt loop, max 2 attempts for Pass, 1 for KnownFail; 5 s backoff between attempts; last attempt's artifacts are kept.
  • crates/e2e/Cargo.toml: dev-dep tokio test-util so the unit tests virtualize the backoff under a paused clock.
  • New test retry_turns_a_transient_failure_green_and_spares_knownfail (flaky-then-pass → pass in 2 runs; KnownFail runs exactly once).

Related (separate)

  • Ops guardrail for the actual #124 problem — "red went unnoticed for 8 days" — deployed to nvb-watch (polls the Actions API, alerts the ops room on a new failing/stale scheduled run). Not in this PR (ops repo).
  • mgmt_dm_pending_reap: keep as KnownFail — it known-failed on a 502 this run, so it is not reliably unexpected-passing.

🤖 Generated with Claude Code

## What Retry a **Pass-expected** e2e scenario **once** on a transient failure before recording it. `KnownFail` scenarios never retry; a `Skip` stays terminal. ## Why (#124 root cause) The nightly `e2e.yml` suite has been red since ~07-11. I root-caused it with clean-tree builds (`git archive`, per the #123 rule) on a quiet runner: - **Not a bridge regression.** The core discord/audio/matrix-rtc code is unchanged since the last green nightly (only two recovery fixes that *predate* the scenarios testing them, plus an additive `guild_footprint()` for #94). 48/48 hermetic puppet-allocation tests pass on master. - **Every cluster scenario passes standalone on a clean quiet host**: `steal_release_repromotion`, `multi_bridge_isolation`, `external_disconnect_recovery`, `recovery_no_spare` all green in isolation. - **They only fail inside the full `--all` run while infra is flapping.** In a full run I captured 23 pass / 3 fail / 2 known-fail, where the 3 fails were: - `audio_d2m_enc` and `steal_release_repromotion` — a phantom's Matrix **membership-heartbeat dropped mid-run** (bridge log shows Carol/Alice memberships vanishing ~3 s after join, collapsing the multi-actor choreography). nether.im was returning intermittent **502s** during the run (they also caused both `#104` known-fails). - `multi_bridge_isolation` — the bridge released its puppet and left voice **on time** (per its log), but the harness listener never observed the leave within 60 s → Discord **voice-state propagation lag** at the tail of a 30-scenario suite on 3 shared bots. - `audio_d2m_enc` re-ran with a *different* ncc (0.253 → 0.500) — non-deterministic, i.e. flaky, not a hard bug. So the cluster is **infra-induced flakiness in a live triage suite**, not a code fault. A single retry (each scenario re-sweeps memberships on entry, so it starts clean) turns a transient blip green while a real regression still fails both attempts. ## Change - `run_scenarios`: per-scenario attempt loop, max 2 attempts for `Pass`, 1 for `KnownFail`; 5 s backoff between attempts; last attempt's artifacts are kept. - `crates/e2e/Cargo.toml`: dev-dep `tokio` `test-util` so the unit tests virtualize the backoff under a paused clock. - New test `retry_turns_a_transient_failure_green_and_spares_knownfail` (flaky-then-pass → `pass` in 2 runs; `KnownFail` runs exactly once). ## Related (separate) - Ops guardrail for the *actual* #124 problem — "red went unnoticed for 8 days" — deployed to `nvb-watch` (polls the Actions API, alerts the ops room on a new failing/stale scheduled run). Not in this PR (ops repo). - `mgmt_dm_pending_reap`: keep as `KnownFail` — it known-failed on a 502 this run, so it is **not** reliably unexpected-passing. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
fix(e2e): retry a Pass-expected scenario once on transient failure (#124)
All checks were successful
CI / test (push) Successful in 16m17s
Details
CI / test (pull_request) Successful in 11m19s
Details
6fc4eda91e 
The Tier-3 live suite runs against real Discord/Matrix/Cloudflare, so
Pass-expected scenarios fail intermittently on transient infra — homeserver
502s dropping a phantom's membership-heartbeat mid-run (collapses the
audio_d2m_enc / steal_release_repromotion choreography), Discord voice-state
propagation lag at the tail of a long suite (multi_bridge_isolation's teardown
assertion), etc. Root-caused in #124: with a clean-tree build on a quiet host,
every cluster scenario passes standalone; they only fail inside the full --all
run while the homeserver is flapping 502s.

Retry a Pass-expected hard failure ONCE before recording it. Each scenario
re-sweeps stale test memberships on entry, so the retry starts clean, and a
genuine regression fails BOTH attempts — this masks flakiness, not bugs. A
KnownFail never retries (its failure is the expected outcome) and a Skip is
terminal (a missing fixture can't be conjured by a retry). A 5s backoff sits
between attempts; the e2e crate gains tokio `test-util` so the unit tests
virtualize it under a paused clock.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
robocub commented 2026-07-19 23:40:30 +00:00
Author
Collaborator
Copy link

Validation

  • CI green: ci.yml #278 on this branch is success — the hermetic gate (cargo test --workspace) includes the new retry_turns_a_transient_failure_green_and_spares_knownfail unit test.
  • Retry demonstrably activates live: in a manual --all on the runner, audio_d2m_unenc failed its first attempt and the harness logged attempt 1 failed — retrying once (transient-infra tolerance, #124).

Caveat on live --all evidence

I could not get a clean green live --all because, mid-run, a concurrent Forgejo CI cargo test --workspace landed on the shared host and drove load to 17 — the audio scenarios then failed at ncc≈0.0 on both attempts (total non-crossing = CPU starvation of the audio pipeline), and the prod bridge was at 128% CPU. I SIGINT/killed my manual run to protect prod.

This is not a regression in the fix — it's a live reproduction of the shared-host contention that is one arm of #124. Note the real nightly is serialized against CI via the live-host concurrency group, so it does not see CI contention; a manual run isn't serialized. The retry helps single-blip flakes (membership drop, voice-state lag) but by design cannot beat sustained starvation — that needs #90 (offload builds to a dedicated runner). Recommend merging on CI-green + the next real serialized nightly, not on a manual contended run.

### Validation - **CI green**: `ci.yml` #278 on this branch is **success** — the hermetic gate (`cargo test --workspace`) includes the new `retry_turns_a_transient_failure_green_and_spares_knownfail` unit test. - **Retry demonstrably activates live**: in a manual `--all` on the runner, `audio_d2m_unenc` failed its first attempt and the harness logged `attempt 1 failed — retrying once (transient-infra tolerance, #124)`. ### Caveat on live `--all` evidence I could not get a clean green live `--all` because, mid-run, a concurrent Forgejo CI `cargo test --workspace` landed on the shared host and drove load to **17** — the audio scenarios then failed at ncc≈0.0 on **both** attempts (total non-crossing = CPU starvation of the audio pipeline), and the prod bridge was at 128% CPU. I SIGINT/killed my manual run to protect prod. This is not a regression in the fix — it's a live reproduction of the shared-host contention that is one arm of #124. Note the *real* nightly is serialized against CI via the `live-host` concurrency group, so it does **not** see CI contention; a manual run isn't serialized. The retry helps single-blip flakes (membership drop, voice-state lag) but by design cannot beat *sustained* starvation — that needs #90 (offload builds to a dedicated runner). Recommend merging on CI-green + the next real serialized nightly, not on a manual contended run.
robocub merged commit 522a5bd856 into master 2026-07-21 01:38:15 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
dark/nether-voicebridge!125
No description provided.