nether/meta

tuwunel: intermittent hickory DNS UDP source-mismatch (benign) — consider local caching resolver #22

New issue

Open

opened 2026-07-12 23:28:30 +00:00 by robocub · 0 comments

robocub commented

2026-07-12 23:28:30 +00:00

Member

Priority: very low / cosmetic. Filed for tracking only — no action needed unless it grows.

Symptom

nether.im's tuwunel (incus container) logs ~13/day:

WARN hickory_net::udp::udp_client_stream: ignoring response from 1.1.1.1:53 (or 1.0.0.1)
because it does not match name_server: 127.0.0.53:53

/etc/resolv.conf points only at 127.0.0.53 (systemd-resolved), which forwards to Cloudflare; the upstream reply's source IP leaks through the container NAT, so hickory's anti-spoofing check drops that answer and the lookup fails.

Impact

Essentially none. It causes an occasional well-known fetch to fail/retry. It was a contributing trigger to a federation wedge with tinier.town on 2026-07-12 (a failed well-known fetch → fallback to their legacy :8448 route → their Caddy returned a silent empty 200 → cached 18–36 h), but the sticky part was their proxy misconfig, which they have since fixed. On its own this is harmless — federation with ~160 peers is unaffected.

⚠️ Do NOT "fix" this with `query_over_tcp_only = true`

Tried 2026-07-12, reverted within ~12 min. systemd-resolved cannot handle DNS-over-TCP at nether's federation query volume → DNS timeout storm (0 → 3,500+ timeouts in 10 min) and outbound federation collapsed (~150 → 15 connections). Reverted (restored config + restart) → recovered. The tuwunel config-doc hint that this option helps "container setups" is a trap for systemd-resolved stubs.

Suggested real fix (if/when worth it)

Run a local caching resolver (unbound or dnsmasq) on the container and point /etc/resolv.conf at it — handles UDP + TCP, caches, and removes the dependency on the flaky systemd-resolved stub. Validate over a 10–15 min window through a federation netburst with an easy rollback; ideally trial on staging first.

Diagnosed during the tinier.town federation debugging session, 2026-07-12.

**Priority: very low / cosmetic.** Filed for tracking only — no action needed unless it grows. ### Symptom nether.im's tuwunel (incus container) logs ~13/day: ``` WARN hickory_net::udp::udp_client_stream: ignoring response from 1.1.1.1:53 (or 1.0.0.1) because it does not match name_server: 127.0.0.53:53 ``` `/etc/resolv.conf` points only at `127.0.0.53` (systemd-resolved), which forwards to Cloudflare; the upstream reply's source IP leaks through the container NAT, so hickory's anti-spoofing check drops that answer and the lookup fails. ### Impact Essentially none. It causes an *occasional* well-known fetch to fail/retry. It was a contributing trigger to a federation wedge with **tinier.town** on 2026-07-12 (a failed well-known fetch → fallback to their legacy `:8448` route → their Caddy returned a silent empty `200` → cached 18–36 h), but the *sticky* part was their proxy misconfig, which they have since fixed. On its own this is harmless — federation with ~160 peers is unaffected. ### ⚠️ Do NOT "fix" this with `query_over_tcp_only = true` Tried 2026-07-12, reverted within ~12 min. systemd-resolved cannot handle DNS-over-TCP at nether's federation query volume → **DNS timeout storm** (0 → 3,500+ timeouts in 10 min) and outbound federation **collapsed (~150 → 15 connections)**. Reverted (restored config + restart) → recovered. The tuwunel config-doc hint that this option helps "container setups" is a trap for systemd-resolved stubs. ### Suggested real fix (if/when worth it) Run a **local caching resolver** (unbound or dnsmasq) on the container and point `/etc/resolv.conf` at it — handles UDP + TCP, caches, and removes the dependency on the flaky systemd-resolved stub. Validate over a **10–15 min window through a federation netburst** with an easy rollback; ideally trial on staging first. _Diagnosed during the tinier.town federation debugging session, 2026-07-12._

robocub added the

labels

2026-07-12 23:28:30 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

nether/meta#22

No description provided.

Rows
Columns

tuwunel: intermittent hickory DNS UDP source-mismatch (benign) — consider local caching resolver #22

Symptom

Impact

⚠️ Do NOT "fix" this with query_over_tcp_only = true

Suggested real fix (if/when worth it)

⚠️ Do NOT "fix" this with `query_over_tcp_only = true`