LiveKit ICE candidate lottery: connections intermittently establish with ZERO media (affects prod audio) #91

New issue

Closed

opened 2026-07-11 20:04:55 +00:00 by robocub · 1 comment

robocub commented

2026-07-11 20:04:55 +00:00

Collaborator

Symptom

LiveKit connections intermittently deliver zero media while signaling stays healthy: the publisher believes it is publishing (local track published, no errors, self-paced feed consumed), the subscriber stays subscribed with no errors — but no RTP arrives. Captured audio is pure digital silence (rms 0.0, 0% nonzero samples). Per-connection coin flip: on 2026-07-11 evening, ~1/3 of e2e connections carried media, ~2/3 delivered nothing — uniformly across unrelated scenarios (baseline audio_m2d_unenc, recovery scenarios), flipping pass/fail run-to-run within minutes at low host load, clean ICMP, LiveKit HTTP healthy.

Likely mechanism (evidence)

Speaker-side livekit=debug captured LiveKit's SDP answer advertising six ICE candidate families from the SFU (nether-red):

97.107.138.76 (public Linode IPv4) — udp 50146 + tcp 7881
100.64.0.2 (tailscale IP) — udp + tcp
10.20.1.1 (LAN/incus bridge IP) — udp + tcp
2600:3c03::… (public IPv6), fd7a:115c:a1e0::2 (tailscale IPv6) — udp + tcp

Each new connection from cubuntu (bridge, ghosts, harness actors) runs an ICE lottery across these. If a pair passes connectivity checks but doesn't actually carry RTP (asymmetric tailscale ACL — the ACL was modified 2026-07-11; unreachable LAN IP; IPv6 asymmetry), the connection establishes with zero media. Matches all observations: per-connection randomness, signaling up, silence.

Why it matters — PROD

Prod's subscriber + ghosts connect to LiveKit the same way from the same host. A broken-but-checks-passing path means real calls with dead air at the same per-connection rate. This may also explain historical intermittent "can't hear / everyone dropped" reports previously attributed to CPU contention alone.

Suggested fix

Constrain LiveKit's advertised candidates to paths that verifiably carry media from cubuntu:

rtc.node_ip / use_external_ip in livekit.yaml to advertise only the public IP, or
fix/verify the tailscale ACL passes UDP media both directions for the tailscale pair, and stop advertising the LAN bridge IP if unreachable from cubuntu.

Then re-verify with the e2e baseline in a loop (audio_m2d_unenc ×5 — should pass 5/5; today it passes ~1/3).

Context

Found while live-verifying the recovery_no_spare e2e scenario (#81 follow-up); blocked its confirmation. Evidence: /opt/nvb-staging/logs/run-{gate3,confirm2,diag2,ns2}.log + WAV RMS analysis in /opt/nvb-staging/e2e-artifacts/. Related: #90 (build/prod isolation).

## Symptom LiveKit connections intermittently deliver **zero media** while signaling stays healthy: the publisher believes it is publishing (local track published, no errors, self-paced feed consumed), the subscriber stays subscribed with no errors — but no RTP arrives. Captured audio is **pure digital silence** (rms 0.0, 0% nonzero samples). Per-connection coin flip: on 2026-07-11 evening, ~1/3 of e2e connections carried media, ~2/3 delivered nothing — uniformly across unrelated scenarios (baseline `audio_m2d_unenc`, recovery scenarios), flipping pass/fail run-to-run within minutes at low host load, clean ICMP, LiveKit HTTP healthy. ## Likely mechanism (evidence) Speaker-side `livekit=debug` captured LiveKit's SDP answer advertising **six ICE candidate families** from the SFU (nether-red): - `97.107.138.76` (public Linode IPv4) — udp 50146 + tcp 7881 - `100.64.0.2` (**tailscale IP**) — udp + tcp - `10.20.1.1` (**LAN/incus bridge IP**) — udp + tcp - `2600:3c03::…` (public IPv6), `fd7a:115c:a1e0::2` (tailscale IPv6) — udp + tcp Each new connection from cubuntu (bridge, ghosts, harness actors) runs an ICE lottery across these. If a pair passes connectivity checks but doesn't actually carry RTP (asymmetric tailscale ACL — **the ACL was modified 2026-07-11**; unreachable LAN IP; IPv6 asymmetry), the connection establishes with **zero media**. Matches all observations: per-connection randomness, signaling up, silence. ## Why it matters — PROD Prod's subscriber + ghosts connect to LiveKit the same way from the same host. A broken-but-checks-passing path means **real calls with dead air at the same per-connection rate**. This may also explain historical intermittent "can't hear / everyone dropped" reports previously attributed to CPU contention alone. ## Suggested fix Constrain LiveKit's advertised candidates to paths that verifiably carry media from cubuntu: - `rtc.node_ip` / `use_external_ip` in livekit.yaml to advertise **only the public IP**, or - fix/verify the tailscale ACL passes UDP media both directions for the tailscale pair, and stop advertising the LAN bridge IP if unreachable from cubuntu. Then re-verify with the e2e baseline in a loop (`audio_m2d_unenc` ×5 — should pass 5/5; today it passes ~1/3). ## Context Found while live-verifying the `recovery_no_spare` e2e scenario (#81 follow-up); blocked its confirmation. Evidence: `/opt/nvb-staging/logs/run-{gate3,confirm2,diag2,ns2}.log` + WAV RMS analysis in `/opt/nvb-staging/e2e-artifacts/`. Related: #90 (build/prod isolation).

robocub commented

2026-07-11 20:50:11 +00:00

Author

Collaborator

RESOLVED — root cause confirmed and fixed 2026-07-11 evening.

Root cause: livekit-server migrated from an incus container to the nether-red host on 2026-06-19 (systemd active since matches the old container's tailscale node going offline 22d ago, to the day). Running on the host, LiveKit enumerated ALL host interfaces and advertised them as ICE candidates — including tailscale0 (100.64.0.2 / fd7a::) and the incus bridge (10.20.1.1). The tailscale path is ACL-filtered: nether-red's own tailscaled journal shows it dropping LiveKit traffic (Drop: TCP{100.64.0.33 → 100.64.0.2:7881}, Drop: UDP{… → 100.64.0.2:501xx}) — including bursts at 06:08 (during the nightly e2e) and 18:53 (the exact second of a failing test run). Per-connection ICE selection then landed on a poisoned route ~2/3 of the time → connection established, zero media, pure-silence audio. The lottery existed since June 19; the tailscale ACL/tag changes made today were unrelated (drops predate them).

Fix (applied by dark on nether-red, /etc/livekit/livekit.yaml): restrict candidate gathering to the public interface —

rtc:
  interfaces:
    includes: [eth0]

(+ node_ip: 97.107.138.76 kept as belt-and-suspenders; note node_ip alone did NOT stop interface enumeration.) systemctl restart livekit-server.

Verified: post-restart SDP advertises ONLY eth0's public IPv4 + IPv6 (tailscale/LAN candidates gone; IPv6 media preserved). Acceptance: audio_m2d_unenc 5/5 PASS (was ~1/3 before the fix), then external_disconnect_recovery PASS 0.918 + recovery_no_spare PASS 0.923. Prod's media path is fixed by the same change. Cleanup note: the stale offline livekit tailscale node (100.64.0.25, the old container's) can be deleted from headscale.

**RESOLVED** — root cause confirmed and fixed 2026-07-11 evening. **Root cause:** livekit-server migrated from an incus container to the nether-red host on **2026-06-19** (systemd `active since` matches the old container's tailscale node going offline 22d ago, to the day). Running on the host, LiveKit enumerated ALL host interfaces and advertised them as ICE candidates — including `tailscale0` (100.64.0.2 / fd7a::) and the incus bridge (10.20.1.1). The tailscale path is ACL-filtered: nether-red's own tailscaled journal shows it **dropping LiveKit traffic** (`Drop: TCP{100.64.0.33 → 100.64.0.2:7881}`, `Drop: UDP{… → 100.64.0.2:501xx}`) — including bursts at 06:08 (during the nightly e2e) and 18:53 (the exact second of a failing test run). Per-connection ICE selection then landed on a poisoned route ~2/3 of the time → connection established, zero media, pure-silence audio. **The lottery existed since June 19**; the tailscale ACL/tag changes made today were unrelated (drops predate them). **Fix** (applied by dark on nether-red, `/etc/livekit/livekit.yaml`): restrict candidate gathering to the public interface — ```yaml rtc: interfaces: includes: [eth0] ``` (+ `node_ip: 97.107.138.76` kept as belt-and-suspenders; note `node_ip` alone did NOT stop interface enumeration.) `systemctl restart livekit-server`. **Verified:** post-restart SDP advertises ONLY eth0's public IPv4 + IPv6 (tailscale/LAN candidates gone; IPv6 media preserved). Acceptance: `audio_m2d_unenc` **5/5 PASS** (was ~1/3 before the fix), then `external_disconnect_recovery` PASS 0.918 + `recovery_no_spare` PASS 0.923. Prod's media path is fixed by the same change. Cleanup note: the stale offline `livekit` tailscale node (100.64.0.25, the old container's) can be deleted from headscale.

robocub closed this issue

2026-07-11 20:50:11 +00:00

robocub referenced this issue

2026-07-11 22:33:36 +00:00

Automatic Testing #38