Isolate CI/release builds from the prod host (dedicated/community build runner) #90

New issue

Open

opened 2026-07-11 18:48:57 +00:00 by robocub · 0 comments

robocub commented

2026-07-11 18:48:57 +00:00

Collaborator

Problem

The prod bridge runs on the mautrix incus container and CI runs on the forgejo-runner container — both on the same cubuntu host, sharing cores. A heavy CI/release build (release image, ~16 min, cargo at full parallelism) pegs the host and starves the prod bridge's real-time audio pipeline → live users get audio glitches during builds.

Confirmed 2026-07-11: the alpha.18/19 releases coincided with staging e2e audio scenarios failing (silent/uncorrelated audio, mean_ncc~0) while LiveKit was healthy and its host (nether-red) idle — i.e. the media endpoints on cubuntu were CPU-starved. Not Discord rate-limits, not LiveKit.

Two contention sources:

CI/release/nightly builds on forgejo-runner (cross-container, shared cubuntu cores).
build-d1.sh staging builds run their docker run inside mautrix — the same container as prod.

Stopgap already applied (2026-07-11)

incus config set forgejo-runner limits.cpu.allowance=800ms/100ms — hard-cap CI to 8 of the 12 cores (prod always has ~4 free); limits.cpu.priority=0 (cpu.weight 90 — yields under contention). Verified via cpu.max = 800000 100000.
build-d1.sh now --cpus=4 + nice -n 19 + CARGO_BUILD_JOBS=4.

These protect prod, but builds still run on the prod host.

Proposed fix: offload builds to a dedicated / community build runner

Physical isolation beats cgroup capping — if builds run on a separate box, they never touch the prod host. Forgejo supports this natively via additional runners.

Register the contributed machine as an additional Forgejo act_runner with a distinct label; route heavy jobs to it via runs-on.
Offload the secret-free compile/test load (ci.yml: build + clippy + test) — the bulk of the CPU, needs no credentials → safe on third-party hardware.
Keep on the trusted (owned) runner: the release job (docker.yml — holds RELEASE_TOKEN, pushes the prod image) and the live e2e/nightly (e2e.yml — holds live Discord/Matrix bot tokens; also wants proximity to LiveKit/staging). ⚠️ Runners see whatever secrets a job uses, so prod/release/e2e secrets must not land on hardware we don't control.
Arch matters (x86_64 vs arm64) — worth knowing what's offered; dovetails with #49 (arm64 runner / multi-arch CI).

Notes

Jason L (shadowdrake) intends to file a related issue — combine when it lands. Filed separately so the analysis + stopgap aren't lost in the meantime.

## Problem The prod bridge runs on the `mautrix` incus container and CI runs on the `forgejo-runner` container — **both on the same `cubuntu` host, sharing cores**. A heavy CI/release build (release image, ~16 min, cargo at full parallelism) pegs the host and starves the prod bridge's real-time audio pipeline → live users get audio glitches *during builds*. Confirmed 2026-07-11: the alpha.18/19 releases coincided with staging e2e audio scenarios failing (silent/uncorrelated audio, mean_ncc~0) while **LiveKit was healthy and its host (nether-red) idle** — i.e. the media *endpoints* on cubuntu were CPU-starved. Not Discord rate-limits, not LiveKit. Two contention sources: 1. CI/release/nightly builds on `forgejo-runner` (cross-container, shared cubuntu cores). 2. `build-d1.sh` staging builds run their `docker run` **inside `mautrix`** — the same container as prod. ## Stopgap already applied (2026-07-11) - `incus config set forgejo-runner limits.cpu.allowance=800ms/100ms` — hard-cap CI to 8 of the 12 cores (prod always has ~4 free); `limits.cpu.priority=0` (cpu.weight 90 — yields under contention). Verified via `cpu.max = 800000 100000`. - `build-d1.sh` now `--cpus=4` + `nice -n 19` + `CARGO_BUILD_JOBS=4`. These protect prod, but builds still *run on the prod host*. ## Proposed fix: offload builds to a dedicated / community build runner Physical isolation beats cgroup capping — if builds run on a separate box, they never touch the prod host. Forgejo supports this natively via additional runners. - Register the contributed machine as an additional Forgejo `act_runner` with a distinct label; route heavy jobs to it via `runs-on`. - **Offload the secret-free compile/test load** (`ci.yml`: build + clippy + test) — the bulk of the CPU, needs no credentials → safe on third-party hardware. - **Keep on the trusted (owned) runner:** the release job (`docker.yml` — holds `RELEASE_TOKEN`, pushes the prod image) and the live e2e/nightly (`e2e.yml` — holds live Discord/Matrix bot tokens; also wants proximity to LiveKit/staging). ⚠️ Runners see whatever secrets a job uses, so prod/release/e2e secrets must not land on hardware we don't control. - Arch matters (x86_64 vs arm64) — worth knowing what's offered; dovetails with #49 (arm64 runner / multi-arch CI). ## Notes Jason L (shadowdrake) intends to file a related issue — **combine when it lands.** Filed separately so the analysis + stopgap aren't lost in the meantime.

robocub referenced this issue

2026-07-11 20:04:55 +00:00

LiveKit ICE candidate lottery: connections intermittently establish with ZERO media (affects prod audio) #91

robocub referenced this issue

2026-07-11 22:33:36 +00:00

Automatic Testing #38

robocub referenced this issue

2026-07-13 16:35:42 +00:00

🎯 NEXT BIG PROJECT (post-stability): whole-codebase refactor — performance/security/flexibility/reliability/maintainability + full documentation + operator walkthrough + human-contributor readiness #41

robocub referenced this issue

2026-07-19 23:40:30 +00:00

fix(e2e): retry a Pass-expected scenario once on transient failure (#124) #125