The platform had the data. It didn’t yet cohere.
By the end of the previous runs Aros could observe the estate and act on it. But the operator kept hitting the same friction: the same physical server showed up as three separate rows across Hosts, BMCs, and Energy; an IP address was inert text you couldn’t click; a correlation that the backend had clearly computed only surfaced if you happened to open the right detail tab and chase it manually. Two menu items did the same thing; one was blank on every entity. The information was all there — it just wasn’t connected, and it didn’t read as one surface. This stretch was about making it cohere: surface the thread wherever you’re standing, make one machine read as one entity, tame the sprawl, and then look up from the features to the estate they live in.
From an inventory hero to an estate architecture
One continuous work stretch, each piece shipped, deployed to dev, and live-verified before the next began.
- Jun 12
The inventory hero + Energy & Cooling
The graph-integrity view had become an unusable wall of text, so it was replaced with a fleet-summary hero — 5 servers, 104 cores, 784 GiB RAM, 82 TB storage, live total power — and a single honest integrity signal that now reads ok:true (gated on the only real corruption signals, dangling parents and parent-chain cycles, so it stops crying wolf over a homelab’s many legitimate roots). Then a new Energy & Cooling tab: per-group and per-device power draw over time from iDRAC Redfish, peak and average, with inlet-temperature thermals. Live-verified at close: 284 W now / 317 W avg across 3 of 4 reporting hosts, inlet 28–30 °C, power and temperature sparklines resolving at 61 points/hour. - Jun 12–13
End-to-end network visibility for a VM
The headline ask: open a VM and see its whole network story. IP → the switch its host connects to → walk the uplink chain back to the router → the Traefik routes fronting it (with TLS and whether Authentik forward-auth gates them) → a read-only open-port probe (a TCP connect, no data sent — the one actively-probing capability, flagged as such). Verified live against the observability VM: client → US-24 → US-16-PoE → USG-Pro-4 gateway, behind monitoring.ryanit.com on Authentik, ports 22/3000/9090/9100 open. - Jun 13–14
Surface the thread from anywhere — and persist it
The operator’s sharpest note: "I don’t want to chase a thread; it should be surfaced regardless of where I’m viewing it from." The network-path resolver was generalised to anchor on a VM, a UniFi client, OR a switch/device, with a hypervisor fallback — and the VM↔client correlation is now stamped on both sides at discovery time, so the answer is stored and cheap to show on every pane, not recomputed by chasing links. - Jun 14
One Network section + the great menu de-sprawl
Network and UniFi merged into a single tabbed section (Overview · Devices · Clients · Routing · Security · Configuration), and the left sidebar was finally separated from the top nav. Then the whole menu, two weeks old and already sprawling, was consolidated: Health (a raw-JSON debug page) and Events (a strictly weaker copy of the Audit log) removed; Cluster folded into Platform as a Membership tab; Settings split into Environment / Preferences / Developer; the BMC registry demoted out of the top level. - Jun 14
Detail tabs consolidated · the Approvals pipeline
Entity-detail tabs were rebuilt with zero information loss: Provenance (blank on ~95% of entities by design) folded away unless it exists, lightweight entities collapsed to a single scrolling page, the same-machine peer de-duplicated. And a full Approvals pipeline: an Inbox pane of glass (what needs my decision vs my own in-flight requests) and a follow-the-chain timeline — request → approval → decision → action → result — reconstructed from the correlation id, shaped like the workflow engine to come. - Jun 15
Tech-debt paid + the estate turn
A cleanup pass fixed a genuine tenant-creation crash (an un-narrowed TenantOut | ApprovalOut union) and cleared six type-suppressions. Then the work lifted its head: a three-agent architecture review of how Aros is installed, how it is tenanted, and how the edge is wired produced ADR-0032 — and a dig into aros-edge-1 confirmed the operator’s instinct that the login path is a single point of failure.
What shipped
Eight feature PRs across two repos plus an architecture ADR — each merged CI-green and verified on the live dev estate.
backend #40–#41 · frontend #31–#36, every one CI-green + deployed + verified
Health, Events, Cluster, BMC registry — folded or deleted, none orphaned
plus one real tenant-creation crash fixed
green throughout; every nav route covered by the e2e smoke
one rule: MAC-via-EUI64 beats shared IP, ambiguity flagged not skipped
317 W avg, 3 of 4 hosts reporting via Redfish
install footprint · tenancy/adapters · auth topology — fed ADR-0032
GUI-driven estate onboarding (Proposed, awaiting ratification)
The bugs were the interesting part
Three of them changed the design. The recap convention is to keep the troubleshooting in, not just the happy-path receipts.
The same VM appeared twice on the topology — and it wasn't a render bug
A UniFi client’s topology pane listed a VM called private-client-services twice — two identical rows, same IP 192.168.1.94, same “via proxmox”. Meanwhile the new “Behind” column that should name the VM at that client’s IP was blank. Two surfaces computed from the same correlation disagreed: one over-showed, the other showed nothing.
The live data was the surprise: there are genuinely TWO distinct VMs both named private-client-services — vmid 102 on pve2 (the real one: guest agent live, Ubuntu, running Docker) and vmid 107 on pve1 (no guest agent, no OS, just claiming .94 — a stale clone with a cloned NIC config). Three different functions resolved IP-to-VM with three different ambiguity policies: the topology showed every claimant, the discovery linker silently skipped any IP owned by more than one VM, and the network path took the first match. They could never agree.
One rule, used everywhere. A MAC recovered from a VM’s EUI-64 IPv6 address (strip the ff:fe, flip the universal/local bit) is exact, so it beats a shared IPv4 — which correctly resolved one of the two clients despite the collision. When only the shared IP matches, the most-recently-seen VM wins, and the ambiguity is surfaced as a “1 of N” badge rather than dropped to blank. Same-name peers now carry their vmid and node so two real VMs read as two distinct rows, not a glitch. The clone was reported to the operator, who removed it. The fix made the data honest about a genuinely messy reality instead of papering over it.
The edge node was flagged degraded — but the alert was lying, and the truth was worse
A persistent P3 finding said the edge was degraded: a probe of the oauth2-proxy on aros-edge-1 timed out. Yet the operator confirmed login worked and every service behind the edge was reachable.
The health adapter probes oauth2-proxy on a host port that oauth2-proxy doesn’t publish — it only listens on the internal Docker network, which is exactly correct for forward-auth. So the check times out forever while the service is perfectly healthy: a false positive. But digging in surfaced the real concern: aros-edge-1 is a single VM running Traefik (ingress) AND oauth2-proxy (forward-auth) AND the Authentik server and worker (the identity provider) AND Authentik’s own local Postgres and Redis. If that one VM dies, there is no ingress, no auth, and no IdP — nobody can log into Aros at all.
The false positive is a one-line health-adapter fix (probe the auth path via Traefik, not the unpublished port). The single point of failure is the real story, and it became the cheap, high-value front of the next plan: move Authentik’s Postgres onto the existing 3-node Patroni cluster — Authentik speaks multi-host libpq natively, the cluster is ready, and ADR-0012 already documented the path — and formalise the break-glass route that already works (the backend trusts the forward-auth headers, so an operator on the LAN can reach the API directly with the edge down). Both captured in a security backlog for a dedicated sprint.
A two-week-old menu had already become a riot
The operator’s words: “Aros is two weeks old and already has menu sprawl.” Platform health sat next to a page literally called Health; there was a Cluster page and a Platform page; Tasks, Events, and Audit all looked like they answered the same question; the left sidebar and top nav were duplicates of each other.
Pages had been added by the subsystem that produced them rather than by the question they answer for the operator. “Health” was a raw-JSON debug surface; “Events” queried the identical endpoint as the Audit log with fewer features; Platform and Cluster answered different questions (“are the services up?” vs “are the right members present?”) over the same four services, so they read as duplicates.
Consolidated by the operator’s question, with a stated rule to stop it recurring: Browse the estate → Assess health → Track activity → Act → Configure. Health and Events deleted from the nav; Cluster folded into Platform as a second tab; Settings split into Environment (admin, audited) / Preferences (per-user) / Developer; BMC registry moved out of the top level. The sidebar now lists sections; each section owns its own sub-tabs. The principle is baked into the code comments so the next feature lands in exactly one bucket.
ADR-0032 — from features to onboarding a whole estate
The operator lifted the cap to a week and asked the bigger question: starting from bare-metal servers, how does someone deploy Aros — and how does one tenant run many sites?
A three-agent review mapped the honest state. The provisioning primitives are strong — real Terraform across ~30 VMs, real row-level multi-tenancy, a real Vault, and a working pattern in the BMC registry (GUI-registered, Vault-backed, with a test-connection). What is greenfield is the orchestration, multi-site, and self-service layer on top.
The keystone insight that organises the whole ADR: today every adapter reads one set of environment variables — one Proxmox URL, one UniFi controller — so the model literally cannot represent two estates. Generalising the proven BMC-registry table into a connector registry (any adapter, registered in the GUI, secrets in Vault, with a per-adapter test-connection) is the single change that unlocks both GUI adapter registration and a multi-site model at once. A connector becomes “this adapter, these credentials, at this site, for this tenant,” and discovery fans out over those rows instead of a hardcoded global.
ADR-0032 lays out that connector registry, a tenant↔site data model (with “new tenant” and “new site for an existing tenant” as distinct, correctly-gated flows), an install state-machine over the existing infrastructure-as-code, and a ranked plan to remove the edge single point of failure — cheapest first. It is Proposed, not Accepted: under the project’s own working agreements a Proposed ADR does not authorise code. A roughly one-week program to build the first, highest-value slice has been scoped and is waiting on ratification, not started.
One resolver to connect them all
The felt asks of the session turned out to share a single missing primitive.
“Make the same hardware stop appearing in three places.” “Let me click an IP and trace it.” “Open a Traefik route and show me what’s serving it.” “Surface the correlation everywhere, not just where I chase it.” Four different complaints — and the architecture work found they are one problem. Network identity (IPs, MACs) lives ad-hoc inside each entity’s metadata blob today, shaped differently per source and never joined, so correlation is value-matching stamped back into records rather than a thing you can query.
The answer scoped for next is a single normalized address table: it makes addresses consistent, resolves a Traefik backend host to the entity serving it, powers click-an-IP-to-trace, expresses hardware de-duplication as “same machine via a shared address,” and — crucially — is the same resolver that will turn inbound east-west flow records into VM-to-VM edges when the flow adapter lands. Build that one table first and everything else hangs off it. That is the shape of the next run; this session is the one that proved why it’s needed.