e2e/staging post-mortem: 4 red nightlies (apt-daily contention) + a stale-fingerprint frankenbuild that faked a d2m regression #123

New issue

Open

opened 2026-07-14 20:51:19 +00:00 by robocub · 1 comment

robocub commented

2026-07-14 20:51:19 +00:00

Collaborator

Post-mortem of two intertwined testing-infrastructure failures found while gating v0.3.0. No shipped code was ever wrong; both were environmental.

1. Every 06:00 nightly from 07-11 through 07-14 was red — apt-daily-upgrade contention

Manually-dispatched e2e runs passed all four days while the crons failed. The runner host's apt-daily-upgrade.timer fires ~06:00–07:00 UTC (verified: 06:16/06:56 on 07-14) and its dpkg/IO load garbles the audio-oracle timing — the same contention class the live-host concurrency group fixed for CI builds, from a source no concurrency group can see. Fix (shipped in the 0.3.0 release commit): nightly moved to 08:30 UTC, clear of that window and of mautrix's 04:15 backup. Watch the first few 08:30 runs to confirm green.

2. A staging "regression" that was a stale-fingerprint frankenbuild

While validating the 0.3.0 candidate, audio_d2m_unenc failed 3/3 on staging with a striking signature: SSRC mapped, speaking=1, frames decoding at full shape with zero errors — but pure-silence PCM (rms_db=-96.0). Bisect:

binary	tree	d2m
staging `bin-109`	accumulated scp-synced tree + 3-day-old shared `target/`	FAIL 3/3
prod beta.4 deb	release CI (pristine checkout)	pass
clean-tree master	fresh `git archive`, touched mtimes	pass
clean-tree master + #116	same	pass 2/2

Source diff between the trees: nothing but cosmetic version strings — the failing binary's own strings confirmed it contained the expected code. Conclusion: the long-lived staging target/ (shared across tree variants, with repeated touch-based rebuild forcing) produced a mixed-vintage link — stale objects + new objects → UB in the audio path → silent PCM. A false fail; the same mechanism could in principle produce subtler wrongness.

Remediations applied:

staging tree replaced wholesale with a clean git archive extract; old tree+target quarantined at /opt/nvb-staging/src109-quarantine (delete after a week);
fresh target/ for the new tree, fingerprints purged.

Standing rules going forward (also in operator memory):

release-gating verdicts ONLY from clean-tree builds (git archive + full touch) or CI artifacts — never from an accumulated scp-synced tree;
periodically (and after any binary-vs-binary discrepancy) rebuild staging from scratch;
a scenario failure that contradicts a fresh build of the same source = suspect the build, not the code.

Post-mortem of two intertwined testing-infrastructure failures found while gating v0.3.0. No shipped code was ever wrong; both were environmental. ## 1. Every 06:00 nightly from 07-11 through 07-14 was red — apt-daily-upgrade contention Manually-dispatched e2e runs passed all four days while the crons failed. The runner host's `apt-daily-upgrade.timer` fires ~06:00–07:00 UTC (verified: 06:16/06:56 on 07-14) and its dpkg/IO load garbles the audio-oracle timing — the same contention class the `live-host` concurrency group fixed for CI builds, from a source no concurrency group can see. **Fix (shipped in the 0.3.0 release commit): nightly moved to 08:30 UTC**, clear of that window and of mautrix's 04:15 backup. Watch the first few 08:30 runs to confirm green. ## 2. A staging "regression" that was a stale-fingerprint frankenbuild While validating the 0.3.0 candidate, `audio_d2m_unenc` failed 3/3 on staging with a striking signature: SSRC mapped, `speaking=1`, frames decoding at full shape with zero errors — but pure-silence PCM (`rms_db=-96.0`). Bisect: | binary | tree | d2m | |---|---|---| | staging `bin-109` | accumulated scp-synced tree + 3-day-old shared `target/` | **FAIL 3/3** | | prod beta.4 deb | release CI (pristine checkout) | pass | | clean-tree master | fresh `git archive`, touched mtimes | pass | | clean-tree master + #116 | same | pass 2/2 | Source diff between the trees: **nothing but cosmetic version strings** — the failing binary's own strings confirmed it contained the expected code. Conclusion: the long-lived staging `target/` (shared across tree variants, with repeated `touch`-based rebuild forcing) produced a mixed-vintage link — stale objects + new objects → UB in the audio path → silent PCM. A false *fail*; the same mechanism could in principle produce subtler wrongness. **Remediations applied:** - staging tree replaced wholesale with a clean `git archive` extract; old tree+target quarantined at `/opt/nvb-staging/src109-quarantine` (delete after a week); - fresh `target/` for the new tree, fingerprints purged. **Standing rules going forward** (also in operator memory): - release-gating verdicts ONLY from clean-tree builds (git archive + full touch) or CI artifacts — never from an accumulated scp-synced tree; - periodically (and after any binary-vs-binary discrepancy) rebuild staging from scratch; - a scenario failure that contradicts a fresh build of the same source = suspect the build, not the code.

robocub referenced this issue

2026-07-19 22:12:58 +00:00

e2e nightly RED 8 days — real multi-channel/encrypted/recovery audio failure cluster (NOT contention); env-vs-code TBD #124

robocub commented

2026-07-19 22:14:17 +00:00

Author

Collaborator

Correction (2026-07-19): the 08:30 cron move did NOT fix the nightly reds, and contention was never the cause. Nightlies stayed red 07-15…07-19 at 08:30. Full triage in #124: the failures reproduce deterministically on a QUIET host (load ~2) — a real multi-channel/encrypted/recovery audio failure cluster, not load. Keep the cron at 08:30 (harmless, still off the apt window) but the post-mortems apt-contention explanation was wrong. The red-nightly ops-room alert proposed in Step 4 is now clearly justified — this cluster went unnoticed for 8 days.

**Correction (2026-07-19): the 08:30 cron move did NOT fix the nightly reds, and contention was never the cause.** Nightlies stayed red 07-15…07-19 at 08:30. Full triage in #124: the failures reproduce deterministically on a QUIET host (load ~2) — a real multi-channel/encrypted/recovery audio failure cluster, not load. Keep the cron at 08:30 (harmless, still off the apt window) but the post-mortems apt-contention explanation was wrong. The red-nightly ops-room alert proposed in Step 4 is now clearly justified — this cluster went unnoticed for 8 days.

robocub referenced this issue

2026-07-19 23:23:25 +00:00

fix(e2e): retry a Pass-expected scenario once on transient failure (#124) #125

robocub referenced this issue

2026-07-19 23:23:53 +00:00

e2e nightly RED 8 days — real multi-channel/encrypted/recovery audio failure cluster (NOT contention); env-vs-code TBD #124