The Platform Speaks, Then Deploys

While distributing the new PROXMOX_WRITE_* variables to the three dev VMs, a double-hop ssh command (WSL → iaccn01 → VM) with nested quotes partially executed in the wrong shell context. The env files on .172 and .173 silently LOST four pre-existing lines — DOCKER_HOSTS plus the three read-credential PROXMOX_* variables.

Each ssh hop strips one layer of quoting. The inner command boundary dissolved, so part of the heredoc ran on the intermediate host and the remote append truncated mid-file. Classic, known, and still bit — because the command LOOKED safe in a single line.

Stop. Verify state on every host in the chain before continuing (iaccn01 was clean). Diff the surviving keys against the intact .171 file, copy the four exact lines back via single-hop scp + sudo tee -a, then verify all three files byte-identical on key count and value lengths (never values). Standing rule re-learned and written into the session receipts: script files + single hops, never nested inline quoting. The deploy went on to succeed.

M2 deployed the 3-VM cluster perfectly, discovery minted the deployed_group cluster entity — but the membership linker resolved 0/3 members and opened a P3 shortfall finding. The member chips the demo needed were empty.

link_cluster_memberships reads each declared member as member[host] — the canonical dict shape every health adapter emits. The new build_group_entities emitted members as bare name strings, which the linker classifies as malformed_member and skips, logging three unresolved lines that initially looked like a name-resolution failure rather than a shape mismatch.

Emit the canonical shape — a host/role dict per member — and pin it in the unit test with a comment naming the live-debugged lesson. Next cycle: 3/3 resolved, shortfall 0, the finding auto-resolved itself, and the member chips lit up. The fix PR went branch → CI → merge → deploy → re-verify inside 20 minutes.

During M2 verification the discovery worker log went silent — enqueues fired every 5 minutes, no cycle ran for 15. The deployed VMs were not landing in inventory and the e2e harness timed out waiting.

The action executor imports the discovery module (for the post-action discovery kick), which registers the discovery_run actor in the TASKS worker process — and a dramatiq CLI worker consumes every queue its registered actors declare. So discovery messages round-robin between two consumers, and the cycles were running fine in the OTHER container log. Same root cause produced an ActorNotFound: partition_run noise line for a seal-worker actor. Also explains a brief finding flap during a deploy window: the last old-image cycle and the first new-image cycle disagreed for one round.

No code change this run — behaviour is correct (the kick is exactly why deploys land in inventory within seconds), just confusing to observe. Tracked for Sprint 6: pin —queues per worker container so each consumes only its own. The debugging session also produced a monitor that sampled finding state across 8 cycles to prove the flap converged — it had, within one cycle of the deploy.

The M1 live run printed FAIL: task did not succeed — directly under a task result showing the clone had succeeded: vmid 113 allocated, cloned, configured, started.

The verification script polled for state == succeeded. The task state machine says completed. The pipeline was fine; the harness was wrong — a reminder that verification code is code and earns the same scrutiny.

Fix the predicate, resume verification from the discovery step with the already-known correlation id (no second VM minted), and note the mismatch inline in the captured evidence so the FAIL line in part 1 of the transcript reads correctly forever.

Clicking Inventory in the sidebar navigated to /infrastructure/inventory and crashed the pane with Invariant failed. The same page worked fine at its legacy /inventory address.

The /infrastructure/inventory route re-used the exported InventoryPage component — but that component called Route.useSearch() where Route is the /inventory route object. Mounted under a different route there is no matching context, and the typed hook throws. The infrastructure mount also never declared validateSearch, so even the search contract was absent.

Make the page route-agnostic: read params via useSearch with strict mode off, re-validate through one shared validator module that BOTH route declarations import, and navigate relative to whichever route mounted it. Both URLs joined the Playwright smoke as permanent regression pins, alongside the index-redirect fix for the empty /infrastructure pane.

The detail-page overhaul wanted pending apt updates per node. GET /nodes/{n}/apt/update returned 403 Sys.Modify with the read-only PVEAuditor token — there is no read-only apt privilege in the PVE permission model at all.

Proxmox treats the package index as a Sys.Modify surface even for reads. Granting it to the reader token means the token COULD post apt actions — a privilege expansion no autonomous run should make on its own authority.

First pass shipped without the data and said so in the UI — a one-line note naming the exact privilege gap instead of an empty field. The operator then approved a narrow grant: a dedicated ArosAptRead role carrying only Sys.Modify, ACL-scoped to /nodes, attached to the reader user and token, used exclusively for GETs. The collector reads the CACHED update list (the PVE daily timer refreshes it; the GET never runs apt). 542 pending updates across the three nodes surfaced as live P4 findings within one discovery cycle, and the UI note now only renders if the grant ever regresses.

A frontend PR went red in CI on the lint step after every local check passed.

The CI lint step runs eslint AND prettier format:check as separate gates; the local loop had only run eslint. The exact same trap exists on the backend (ruff check vs ruff format —check) and had already bitten there weeks earlier.

Format, re-push, green — then encode the rule where it cannot be forgotten: the cross-session memory note about local validation gates now covers both repos with the full gate list per side. The local gate scripts used for the rest of the run included format checks from then on.