observability: opt-in Prometheus /metrics exporter (nether services dashboard) #118

New issue

Open

opened 2026-07-13 16:29:56 +00:00 by robocub · 0 comments

robocub commented

2026-07-13 16:29:56 +00:00

Collaborator

Decision: yes, eventually — the operator wants a single dashboard for the health of all nether services; a Prometheus /metrics endpoint on the bridge is the right foundation (Grafana on top; other services can join the same scrape config).

Scope (bridge side): an opt-in [metrics] listen = "127.0.0.1:9464" knob exposing:

per-bridge: up/uptime, call_active, participants (matrix/discord), puppet assignments
audio: frames_in/out, jitter_dropped, jitter_depth, capture_errs, shape_warns (PLC/loss proxy), voicetick rate
rtc: LiveKit connection quality gauge + packet-loss/jitter per participant (depends on the ConnectionQuality issue)
mgmt: tenants count, distinct owners, pending links, invite/link gate rejections, per-bot guild count (verification tripwire as an alert rule instead of a string match)
process: RSS, fd count

Notes: prometheus or metrics + metrics-exporter-prometheus crate; default OFF so prod config stays untouched until wanted; bind loopback + scrape over the existing tailnet. Not urgent — log-based nvb-watch stays the alerting path until this lands. Priority below feature/bug work; good "quiet week" project.

**Decision: yes, eventually** — the operator wants a single dashboard for the health of all nether services; a Prometheus `/metrics` endpoint on the bridge is the right foundation (Grafana on top; other services can join the same scrape config). **Scope (bridge side):** an opt-in `[metrics] listen = "127.0.0.1:9464"` knob exposing: - per-bridge: up/uptime, call_active, participants (matrix/discord), puppet assignments - audio: frames_in/out, jitter_dropped, jitter_depth, capture_errs, shape_warns (PLC/loss proxy), voicetick rate - rtc: LiveKit connection quality gauge + packet-loss/jitter per participant (depends on the ConnectionQuality issue) - mgmt: tenants count, distinct owners, pending links, invite/link gate rejections, per-bot guild count (verification tripwire as an alert rule instead of a string match) - process: RSS, fd count **Notes:** `prometheus` or `metrics` + `metrics-exporter-prometheus` crate; default OFF so prod config stays untouched until wanted; bind loopback + scrape over the existing tailnet. Not urgent — log-based nvb-watch stays the alerting path until this lands. Priority below feature/bug work; good "quiet week" project.