nether/meta

Nether Cloud: architecture direction — decisions & roadmap (2026-07-12) #21

New issue

Open

opened 2026-07-12 16:29:20 +00:00 by robocub · 0 comments

robocub commented

2026-07-12 16:29:20 +00:00

Member

Nether Cloud: architecture direction — decisions & roadmap (2026-07-12)

Summary of a long design conversation (Fox + Claude) covering admin tooling for the Tuwunel stack, how etke.cc actually works, k8s vs. simpler models, and a Guix-based architecture for a community-run, donated-hardware "Nether Cloud." Filed as a meta issue so decisions and open forks are captured in one place.

Research findings

Admin UI / Tuwunel. Ketesa (the renamed etke.cc synapse-admin) does not support Tuwunel — its "multi-backend" support means Synapse + MAS admin APIs. Tuwunel has no HTTP admin API at all; administration is admin-room text commands (conduwuit convention). Porting Ketesa is the worst option (its whole data layer is Synapse REST resources). Realistic paths: Tuwunel grows a native/Synapse-compatible admin API upstream (requested, uncommitted), or purpose-built UIs on the command channel. knadh/tuwunel-admin is the current stopgap (sub-1.0, author-warned); if deployed, bind it to the private overlay network only. Moderation is unaffected — Draupnir works over the client-server API.

etke.cc architecture. No k8s: one dedicated Hetzner VPS (or on-prem box) per customer, provisioned by matrix-docker-ansible-deploy (which they maintain), Docker containers under systemd, customer changes = playbook vars + an Ansible run. Validates the single-tenant-isolation, config-driven model; the hard part is the config-management layer, not orchestration tech.

K8s assessment. Wrong shape for this workload: stateful single-tenant monoliths that never scale horizontally; k8s tax (control plane, storage/network plumbing, upgrade treadmill) buys features that go unused, while its weaknesses (soft multi-tenancy, shared blast radius, non-portability of deployments) hit exactly where this model needs strength.

Guix orchestration gap (verified). guix system container is an instantiation mechanism, not a management plane. The missing container daemon was identified by the feature's own author in 2015 and never built; the 2026 cookbook still describes the script-emitting workflow; the heaviest production users (GeneNetwork) manage containers with hand-written systemd units and manual GC-root symlinks. The community's revealed preference when needing a management plane is to package one (first-class Ganeti service type; Incus packaged). Incus package exists in Guix; an incus-service-type does not appear in upstream manual/services (to be re-confirmed against current master).

Ganeti (context): Google-originated cluster VM manager with DRBD-replicated disks, live migration, failover — the paved road if state-preserving failover is ever wanted, at the replicated-storage complexity cost we're deliberately declining.

Vaultwarden. OIDC SSO is upstream now. SSO handles authentication only; the master password derives the encryption key client-side and never reaches the server (zero-knowledge), so "unlock vault with the SSO password" is impossible by design — and that's the selling point for us: infrastructure compromise/seizure cannot open vaults. Key Connector (master-password-less) won't come to Vaultwarden (Bitwarden license). Decide the emergency-access/recovery story before rollout; a forgotten master password = unrecoverable vault.

Web clients. Sable (Cinny fork) is a React SPA — trivially self-hostable as static files behind Caddy. Commet has a web build but it's the weakest platform (Flutter/CanvasKit: heavy, laggy reports, verification quirks); officially supported targets are Windows/Linux/Android. Confirm intended web-support status with Airyzz before offering it.

Architecture direction (agreed in conversation; ratify/amend here)

The stack sandwich. Guix System (+ nonguix for firmware) on bare metal → Incus as the tenancy/management layer → guests preferably Guix (guix system image) generated by a single tenant->operating-system function over per-tenant manifests; non-Guix guests allowed as a pressure valve, used sparingly. Inside guests: native Guix services for the boring layer; oci-container-service-type (digest-pinned images) for the fast-moving Matrix application layer (Tuwunel, bridges, Draupnir) until native packages/service types exist.

Trust ladder — isolation level is an instantiation choice, same declarations:

Trusted internal services → raw guix system container under Shepherd.
Standard tenants → hardened Incus containers (idmap, seccomp/profiles, quotas, snapshots).
High-sensitivity tenants → Incus KVM VMs (own kernel); optionally nested pattern: tenant = VM, tenant's services = containers inside it, so blast radius aligns with trust boundaries.

Git is the control plane. Desired state (node declarations + tenant manifests) lives in a Forgejo-hosted infrastructure channel. The future provisioning webui does not talk to servers: it writes signed commits/PRs; CI (Forgejo runner) validates and test-builds closures; merge triggers guix deploy from a dedicated control node. Technical users edit Scheme/manifests directly — both interfaces converge on the same commits, one audit log, one rollback story (git revert / guix system roll-back). The control node + infra repo are the crown jewels; key custody, signed commits, and merge rights get design effort up front.

Decline the scheduler ("cattle-shaped pets"). Static service placement recorded declaratively; fault tolerance defined as time-to-rebuild, not HA. Node-level monitoring alerts into a Matrix ops room; recovery = guix system init + state restore from aggressive off-node backups (≥2 nodes + off-fleet), runbooks kept as literate org files in the repo, executable by any trusted member. Later: a warden bot (Matrix bot-federation pattern) that diagnoses and executes runbook steps on human ack — recovery decisions visible in a public-to-community room. Automated cross-node failover deliberately out of scope (Raft/replicated-storage cliff; donated-hardware churn is the failure mode such clusters hate).

Networking. Per-tenant Incus bridges (+ ACLs; nftables on host as backstop); tenant-internal networks invisible outside their VM; overlay network (Headscale/Tailscale) as the only fabric; public ingress only via designated edge Caddy nodes reverse-proxying inward. Every boundary declared in text — auditable by reading three files.

Storage/secrets. btrfs over ZFS on Guix hosts (out-of-tree module treadmill avoided; keeps snapshots/CoW). Secrets never enter /gnu/store (world-readable): out-of-band provisioning (e.g. age-encrypted files deployed to runtime paths) — pick the boring mechanism early, before the webui needs it.

Build vs. buy ruling

Tier 0 — write incus-service-type (do now): config record + shepherd-service for incusd + account/activation extensions (~100–200 lines, patterned on libvirt's). Few sessions, unblocks the whole architecture, upstreamable to Guix. First concrete action item.
Tier 1 — native Guile container manager (long-horizon community project): registry of system containers as generated Shepherd services, veth/bridge wiring, cgroup v2 limits, btrfs state snapshots. Deployed only on the trusted bottom rung until it earns trust. Its compounding advantages over Incus: container topology inside the system generation (rollback covers which containers exist), read-only shared store (near-zero per-tenant disk, one substitute cache/GC), fleet-wide CVE patching as symlink flips, single readable Guile TCB (the "community can read the whole stack" property), and cross-boundary service extension (tenant declarations extending host Caddy, one dependency graph). Disadvantages don't shrink: security-critical code without LXC's 15 years of hardening, no VM story, no clustering.
Tier 2 — Incus parity: person-years; out of scope. Workloads migrate down the ladder only on demonstrated trustworthiness.

Roadmap sketch (10-year shape)

Phase 1 (Y1–2): migrate nether.im itself onto the pattern — tenant zero. Timebox ruthlessly; "the flagship homeserver runs on it" gates everything else. Tier-0 service type, backup discipline, first runbooks.
Phase 2 (Y2–4): friendly tenants; provisioning webui (thin client over git); first native service types (Tuwunel the obvious candidate) written and upstreamed; warden bot prototype.
Phase 3 (Y4–7): Nether Cloud as a forkable distribution — Guix channel + manifest conventions + literate runbooks any community can adopt for a sovereign cluster on donated hardware; the hosted service becomes one deployment of it.
Ongoing: the durable asset is the legible codebase and the people who understand it, not the fleet. Budget for tested restores, bridge breakage on upstream releases, and documentation freshness.

Open questions / next actions

Confirm incus-service-type absence on current Guix master (guix system search incus; grep gnu/services)
Tier-0 session: write + test incus-service-type on a Guix host; plan upstream submission
Ask Airyzz: is Commet's web target intended/supported, or exists-but-unendorsed?
Choose the boring secrets mechanism (age-encrypted out-of-band deploy or similar) — before webui work
Ratify storage backend ruling (btrfs proposed) for Guix hosts
Evaluate knadh/tuwunel-admin on the private overlay only; track upstream Tuwunel admin-API discussion
Vaultwarden: decide SSO rollout + emergency-access/recovery story; frame zero-knowledge as the feature in user docs
Sable web deployment as the recommended Nether web client; static hosting behind edge Caddy
Control-node design pass: key custody, signed commits, merge rights, who-can-deploy
Define tenant manifest schema + tenant->operating-system skeleton

Filed by robocub on Fox's request, summarizing the 2026-07-12 design conversation.

# Nether Cloud: architecture direction — decisions & roadmap (2026-07-12) Summary of a long design conversation (Fox + Claude) covering admin tooling for the Tuwunel stack, how etke.cc actually works, k8s vs. simpler models, and a Guix-based architecture for a community-run, donated-hardware "Nether Cloud." Filed as a meta issue so decisions and open forks are captured in one place. --- ## Research findings **Admin UI / Tuwunel.** Ketesa (the renamed etke.cc synapse-admin) does **not** support Tuwunel — its "multi-backend" support means Synapse + MAS admin APIs. Tuwunel has no HTTP admin API at all; administration is admin-room text commands (conduwuit convention). Porting Ketesa is the worst option (its whole data layer is Synapse REST resources). Realistic paths: Tuwunel grows a native/Synapse-compatible admin API upstream (requested, uncommitted), or purpose-built UIs on the command channel. `knadh/tuwunel-admin` is the current stopgap (sub-1.0, author-warned); if deployed, bind it to the private overlay network only. Moderation is unaffected — Draupnir works over the client-server API. **etke.cc architecture.** No k8s: one dedicated Hetzner VPS (or on-prem box) per customer, provisioned by `matrix-docker-ansible-deploy` (which they maintain), Docker containers under systemd, customer changes = playbook vars + an Ansible run. Validates the single-tenant-isolation, config-driven model; the hard part is the config-management layer, not orchestration tech. **K8s assessment.** Wrong shape for this workload: stateful single-tenant monoliths that never scale horizontally; k8s tax (control plane, storage/network plumbing, upgrade treadmill) buys features that go unused, while its weaknesses (soft multi-tenancy, shared blast radius, non-portability of deployments) hit exactly where this model needs strength. **Guix orchestration gap (verified).** `guix system container` is an instantiation mechanism, not a management plane. The missing container daemon was identified by the feature's own author in 2015 and never built; the 2026 cookbook still describes the script-emitting workflow; the heaviest production users (GeneNetwork) manage containers with hand-written systemd units and manual GC-root symlinks. The community's revealed preference when needing a management plane is to *package* one (first-class Ganeti service type; Incus packaged). Incus **package** exists in Guix; an `incus-service-type` does not appear in upstream manual/services (to be re-confirmed against current master). **Ganeti** (context): Google-originated cluster VM manager with DRBD-replicated disks, live migration, failover — the paved road *if* state-preserving failover is ever wanted, at the replicated-storage complexity cost we're deliberately declining. **Vaultwarden.** OIDC SSO is upstream now. SSO handles authentication only; the master password derives the encryption key client-side and never reaches the server (zero-knowledge), so "unlock vault with the SSO password" is impossible by design — and that's the selling point for us: infrastructure compromise/seizure cannot open vaults. Key Connector (master-password-less) won't come to Vaultwarden (Bitwarden license). Decide the emergency-access/recovery story before rollout; a forgotten master password = unrecoverable vault. **Web clients.** Sable (Cinny fork) is a React SPA — trivially self-hostable as static files behind Caddy. Commet has a web build but it's the weakest platform (Flutter/CanvasKit: heavy, laggy reports, verification quirks); officially supported targets are Windows/Linux/Android. Confirm intended web-support status with Airyzz before offering it. --- ## Architecture direction (agreed in conversation; ratify/amend here) **The stack sandwich.** Guix System (+ nonguix for firmware) on bare metal → **Incus** as the tenancy/management layer → guests preferably Guix (`guix system image`) generated by a single `tenant->operating-system` function over per-tenant manifests; non-Guix guests allowed as a pressure valve, used sparingly. Inside guests: native Guix services for the boring layer; `oci-container-service-type` (digest-pinned images) for the fast-moving Matrix application layer (Tuwunel, bridges, Draupnir) until native packages/service types exist. **Trust ladder — isolation level is an instantiation choice, same declarations:** 1. Trusted internal services → raw `guix system container` under Shepherd. 2. Standard tenants → hardened Incus containers (idmap, seccomp/profiles, quotas, snapshots). 3. High-sensitivity tenants → Incus KVM VMs (own kernel); optionally nested pattern: tenant = VM, tenant's services = containers inside it, so blast radius aligns with trust boundaries. **Git is the control plane.** Desired state (node declarations + tenant manifests) lives in a Forgejo-hosted infrastructure channel. The future provisioning webui does not talk to servers: it writes signed commits/PRs; CI (Forgejo runner) validates and test-builds closures; merge triggers `guix deploy` from a dedicated control node. Technical users edit Scheme/manifests directly — both interfaces converge on the same commits, one audit log, one rollback story (git revert / `guix system roll-back`). The control node + infra repo are the crown jewels; key custody, signed commits, and merge rights get design effort up front. **Decline the scheduler ("cattle-shaped pets").** Static service placement recorded declaratively; fault tolerance defined as *time-to-rebuild*, not HA. Node-level monitoring alerts into a Matrix ops room; recovery = `guix system init` + state restore from aggressive off-node backups (≥2 nodes + off-fleet), runbooks kept as literate org files in the repo, executable by any trusted member. Later: a warden bot (Matrix bot-federation pattern) that diagnoses and executes runbook steps on human ack — recovery decisions visible in a public-to-community room. Automated cross-node failover deliberately out of scope (Raft/replicated-storage cliff; donated-hardware churn is the failure mode such clusters hate). **Networking.** Per-tenant Incus bridges (+ ACLs; nftables on host as backstop); tenant-internal networks invisible outside their VM; overlay network (Headscale/Tailscale) as the only fabric; public ingress only via designated edge Caddy nodes reverse-proxying inward. Every boundary declared in text — auditable by reading three files. **Storage/secrets.** btrfs over ZFS on Guix hosts (out-of-tree module treadmill avoided; keeps snapshots/CoW). Secrets never enter `/gnu/store` (world-readable): out-of-band provisioning (e.g. age-encrypted files deployed to runtime paths) — pick the boring mechanism early, before the webui needs it. --- ## Build vs. buy ruling - **Tier 0 — write `incus-service-type` (do now):** config record + shepherd-service for incusd + account/activation extensions (~100–200 lines, patterned on libvirt's). Few sessions, unblocks the whole architecture, upstreamable to Guix. First concrete action item. - **Tier 1 — native Guile container manager (long-horizon community project):** registry of system containers as generated Shepherd services, veth/bridge wiring, cgroup v2 limits, btrfs state snapshots. Deployed **only** on the trusted bottom rung until it earns trust. Its compounding advantages over Incus: container *topology inside the system generation* (rollback covers which containers exist), read-only shared store (near-zero per-tenant disk, one substitute cache/GC), fleet-wide CVE patching as symlink flips, single readable Guile TCB (the "community can read the whole stack" property), and cross-boundary service extension (tenant declarations extending host Caddy, one dependency graph). Disadvantages don't shrink: security-critical code without LXC's 15 years of hardening, no VM story, no clustering. - **Tier 2 — Incus parity:** person-years; out of scope. Workloads migrate down the ladder only on demonstrated trustworthiness. --- ## Roadmap sketch (10-year shape) - **Phase 1 (Y1–2):** migrate nether.im itself onto the pattern — tenant zero. Timebox ruthlessly; "the flagship homeserver runs on it" gates everything else. Tier-0 service type, backup discipline, first runbooks. - **Phase 2 (Y2–4):** friendly tenants; provisioning webui (thin client over git); first native service types (Tuwunel the obvious candidate) written and upstreamed; warden bot prototype. - **Phase 3 (Y4–7):** Nether Cloud as a *forkable distribution* — Guix channel + manifest conventions + literate runbooks any community can adopt for a sovereign cluster on donated hardware; the hosted service becomes one deployment of it. - **Ongoing:** the durable asset is the legible codebase and the people who understand it, not the fleet. Budget for tested restores, bridge breakage on upstream releases, and documentation freshness. --- ## Open questions / next actions - [ ] Confirm `incus-service-type` absence on current Guix master (`guix system search incus`; grep gnu/services) - [ ] Tier-0 session: write + test `incus-service-type` on a Guix host; plan upstream submission - [ ] Ask Airyzz: is Commet's web target intended/supported, or exists-but-unendorsed? - [ ] Choose the boring secrets mechanism (age-encrypted out-of-band deploy or similar) — before webui work - [ ] Ratify storage backend ruling (btrfs proposed) for Guix hosts - [ ] Evaluate `knadh/tuwunel-admin` on the private overlay only; track upstream Tuwunel admin-API discussion - [ ] Vaultwarden: decide SSO rollout + emergency-access/recovery story; frame zero-knowledge as the feature in user docs - [ ] Sable web deployment as the recommended Nether web client; static hosting behind edge Caddy - [ ] Control-node design pass: key custody, signed commits, merge rights, who-can-deploy - [ ] Define tenant manifest schema + `tenant->operating-system` skeleton *Filed by robocub on Fox's request, summarizing the 2026-07-12 design conversation.*

robocub referenced this issue

2026-07-12 16:35:47 +00:00

Hosted Sable Instance #20

robocub referenced this issue

2026-07-12 16:35:47 +00:00

Vaultwarden Service #6

robocub referenced this issue

2026-07-12 16:35:48 +00:00

Matrix Moderation Tooling #8

robocub referenced this issue

2026-07-12 16:35:48 +00:00

Automatic Snapshots #19