Three primitives, the platform sprint
Three visualisations cover what Sprint 3 actually shipped. The adapter framework is the headline — one ABC, five live adapters, designed so Sprint 4 plugs new ones into the same socket. The policy decision tree shows the two-layer evaluator that now gates POST /api/v1/tenants — the animated token follows the most common path (request → MSP → tenant → 201). The worker topology is the runtime side: API enqueues, three dedicated worker containers consume their own Redis queue. APScheduler ticks in the seal container.
One ABC, five live adapters
Every external system Aros touches goes through the same Adapter ABC: manifest(), initialize(), discover(), health_check(), plus ingest() for MSP-mode call-home from day one. Sprint 3 ships five of them; Sprint 4 plugs in Proxmox, Cloudflare, PBS, Docker into the same socket.
POST /api/v1/tenants now passes through the gate
The MSP layer evaluates first. A DENY there is final — the tenant layer is never consulted. Otherwise the action descends; REQUIRE_APPROVAL at either layer creates an approval row and returns 202 + ApprovalOut. Eleven condition operators: EQUALS / NOT_EQUALS / IN / NOT_IN / CONTAINS / NOT_CONTAINS / GREATER_THAN / LESS_THAN / MATCHES / AND / OR / NOT.
Three worker containers, one image, three entrypoints
The API container enqueues; never consumes. Workers consume; never enqueue user actions. The seal worker holds the BYPASSRLS credentials and is physically isolated from user-triggered work. The seal entrypoint is bespoke — it colocates APScheduler with the Dramatiq Worker in one process so periodic enqueue survives the CLI's fork model.
From Sprint 2 audit substrate to platform self-awareness
Sprint 2 made every state change auditable. Sprint 3 makes the platform able to see itself, gate sensitive operations through a real policy engine, and split adapters out as the headline pattern that Sprint 4 will extend.
tasks · policy · approvals · platform · adapters
0006 tasks · 0007 policy + approvals · 0008 capabilities · 0009 roles
patroni · redis · minio · backend · edge
tasks · discovery · seal per env
Four phases shipped, close-out flushed five environment bugs
Phase A landed Dramatiq + core.tasks. Phase B layered the policy engine on top with the tenant CRUD retrofit. Phase C built adapter zero and the membership API. Phase D swapped the frontend pages. Then the first real /platform/health probe surfaced four adapter bugs plus the deploy-workers.sh chain of pain plus the Forgejo forward-auth collision plus the missing code splitting.
- Day 1 morning
Read the brief, absorb six docs
Platform vision, Sprint 3 plan, ADRs 0017 / 0018 / 0020 / 0021. Internalised the everything-adapter-pattern-first principle so the Sprint 3 health endpoint is built to the same ABC Sprint 4 inventory consumers will use. Confirmed the three-worker-container topology and the embedded-then-OPA migration path for the policy engine. - Day 1 morning
Phase A — Dramatiq + core.tasks
Migration 0006 ships tasks.task with FORCE RLS. core.tasks: state machine in service.py, generic run_task actor with handler registry, REST API. workers/broker.py shares a RedisBroker (db 1) or StubBroker. Three worker entrypoints. CI deploy step extended to start three worker containers on .171. Retry policy refactored: max_retries=0 on the actor, retries managed in the DB via send_with_options(delay=ms). - Day 1 midday
Phase B — policy + approvals
Migration 0007 lays down the policy + approval schemas with FORCE RLS where appropriate. core.policy: condition-tree evaluator with eleven operators, PolicyEvaluator ABC, EmbeddedEvaluator concrete. core.approvals: state machine (pending -> approved | rejected | expired | executed | failed) + router. Tenant CRUD retrofitted: POST /api/v1/tenants now routes through policy_service.evaluate. ~25 test cases over the evaluator written before the integration runs. - Day 1 afternoon
Phase C — adapter zero
aros_backend.adapters package per ADR-0017. Adapter ABC plus the value dataclasses. Five health adapters with parallel probe + 5s timeout. pyproject.toml registers the aros.adapters entry-point group. /api/v1/platform/health aggregates the probes. Membership API on /api/v1/tenants/{slug}/members with the four-role vocabulary. Contract tests parametrized across all five adapters. - Day 1 evening
Phase D — frontend swaps + docs + first recap
schema.d.ts extended to backend 0.7.0. /tasks, /approvals, /platform pages swapped from EmptyState / mocks to the real APIs. ai-context docs reconciled. First Sprint 3 recap MDX written. Operator pulled me back: do not publish the recap until sprint close is confirmed, and the MDX itself had an acorn parse error in one BentoGrid body string. Recap deleted, memory saved about the timing, the close-out work itself becomes recap material. - Close-out start
CI lint pass — ruff + mypy + prettier
Pipeline surfaced 17 ruff errors across the Sprint 3 surfaces (dead noqa BLE001, SIM105 contextlib.suppress, isort sorting, B008 on FastAPI Query() defaults, Yoda condition, unused imports) plus prettier on the four Sprint 3 route files. Set up a local Python venv with ruff + mypy + pytest so the autonomous loop could verify before pushing. Broker bootstrap refactored: actors modules import the broker as a hard dependency so isort cannot reorder workers/{tasks,seal}.py into a broker-not-bound state. - Close-out — workers deploy
BusyBox sh kept choking on inline ssh
CI deploy worker block hit /bin/sh: syntax error: unterminated quoted string twice — first as a multi-line ssh "..." body with backslash continuations, then again after rewriting to single-line semicolon-separated form. The first worker container actually started (docker run -d returned an ID); the error came from somewhere in the Alpine shell parsing that I could not pin down after byte-checking the YAML. Final fix: ship scripts/deploy-workers.sh via scp + execute via ssh ... bash /tmp/aros-deploy-workers.sh dev $CI_COMMIT_SHA $SHORT_SHA $IMAGE. No inline quoting surface. set -euo pipefail, ends with docker ps filtered by worker labels. - Close-out — migration ownership
Operator-side: tables created as postgres, app role could not ALTER
The dev migrations 0006-0009 were applied as the postgres superuser (alembic via docker exec runs under whatever role the env file points at; an earlier bootstrap step used postgres). The tables came up owned by postgres rather than the per-env aros_backend_{env} role, so subsequent app-role ALTERs (and the implicit FORCE RLS owner-must-also-be-bound rule) tripped. Operator-side fix: REASSIGN OWNED on the new schemas to the app role and re-run the migrations. Recorded as Sprint 4 hardening: the migration runner role must match the table-owner role we want. - Close-out — first /platform/health probe
Four adapter bugs in one go
Operator deployed dev and ran /api/v1/platform/health. Four adapter bugs surfaced that the contract tests cannot catch (no real network in CI). patroni: streaming flagged as not running. backend: self-loopback timeout (container probing itself through docker bridge + host LAN + docker port-map). edge: pointing at .92 instead of .241; needed Authentik on 9443 with skip-TLS. workers: all three Up (unhealthy) because the image-level HEALTHCHECK probes :8000 which only the API serves. All four fixed in one commit; --no-healthcheck on the worker docker runs. - Close-out — code splitting
Phase D leftover: autoCodeSplitting on the router-plugin
The Phase D plan called for route-level code splitting; the first cut deferred it as a TanStack .lazy.tsx refactor that would have rewritten every route file. Discovered that @tanstack/router-plugin/vite already supports autoCodeSplitting: true which transforms route files into lazy chunks at build time without touching the source. One-line flag flip. Initial bundle dropped to 132 KB gzipped (under the 150 KB target); every per-route chunk under 100 KB. - Close-out — Forgejo forward-auth
Operator-side: edge auth gate broke git clone + Woodpecker OAuth
Operator added oauth2-proxy forward-auth in front of all subdomains. That broke two paths that Aros depends on: git clone over HTTPS (Forgejo expects HTTP basic auth, not OIDC redirect) and Woodpeckers GitHub-OAuth-style callback (the callback URL got intercepted by the forward-auth gate). Operator-side fix: bypass forward-auth for the /api/internal/login/oauth callback path and for git-over-HTTPS. The work itself stays out of the recap as plumbing detail; the lesson is that adding a tenant-facing auth layer needs an audit of every internal HTTP surface that Aros depends on, recorded as Sprint 4 monitoring + Sprint 5 notifications work. - Close-out — docs reconciliation
ai-context refresh + this recap
02-current-sprint.md now has the full Sprint 3 close-out section with every post-Phase-D commit. 11-sprint-3-plan.md status banner refreshed. 01-architecture-summary.md picked up new Adapter framework / Task topology / Policy engine sections. 00-project-overview.md gained a current-state appendix listing the 8 shipped core modules. Then this recap, written only after the operator confirmed sprint close, with the troubleshooting trail (the deleted first-recap incident included) as load-bearing material rather than happy-path receipts.
Each of these moved us off the obvious path
The contract tests cover the interface; they cannot exercise real network or real database state. Every challenge below is something the platform learned only after running against the live homelab.
health.patroni treated streaming replicas as not running
First /api/v1/platform/health probe on dev returned patroni as degraded with the two replicas listed under not_running. Cluster was actually fine — the replicas were streaming, which is the normal state.
Adapter logic was state != "running" means degraded. Patroni distinguishes leader (state=running, accepting writes) from replicas (state=streaming, replicating from the leader and accepting reads). The first cut conflated leader-state with all-member-state.
Introduced _HEALTHY_MEMBER_STATES = frozenset(streaming). Cluster is healthy when the leader is running and every replica is streaming. stopped / crashed / start failed still degrade. Documented the Patroni state vocabulary at the top of the adapter so a future reader does not relearn this from the API docs.
health.backend timed out probing itself
The backend adapter probes :8000/health/live on each of the three app VMs for the current env. When the adapter ran on .171 (which is also one of the probe targets), the self-probe timed out at 3s, marking the local node as failed and degrading the rollup.
Inside the container, the request goes: docker bridge -> host LAN -> docker port-map -> back to the same container, which is currently processing the /api/v1/platform/health request that triggered the probe. Asyncio handles it (the event loop is not blocked) but the loopback path adds enough latency that 3s is tight, and the busy container makes it tighter still.
Added a hostname-to-IP map for the known backend VMs (aros-dev-1 -> .171, aros-dev-2 -> .172, and so on across dev/test/UAT). The adapter reads settings.deploy_host (already set by the deploy step as $(hostname) on the remote VM) and skips the matching IP. Self is recorded as healthy without an HTTP probe — if /api/v1/platform/health is responding, the local node is alive by definition.
health.edge probed the wrong host
Edge probes all failed; operator pointed out the adapter was checking 192.168.1.92 (a different homelab host) while aros-edge-1 actually lives at .241.
Hardcoded constant in the adapter (EDGE_HOST = “192.168.1.92”) taken from an older notes file. Sprint 1.5 ADR-0013 has the canonical address but I did not cross-check.
Added settings.edge_host with default .241 and EDGE_HOST env override. Probes now target Traefik on :80 (accept either 200 or 404 — without a Host header the router serves 404, which still proves the listener is alive), Authentik HTTPS on :9443/-/health/live/ with verify_tls=False (matches the mvm Traefik serversTransport insecureSkipVerify), and oauth2-proxy /ping on :4180. Extended http_probe with timeout, verify_tls, and accept_status kwargs.
Worker containers all Up (unhealthy)
After the worker containers deployed successfully, docker ps showed all three as Up (unhealthy). The processes were running fine; only the docker health status was wrong.
The image-level HEALTHCHECK in the Dockerfile probes :8000/health/live (correct for the API container). Worker containers run dramatiq or python -m aros_backend.workers.seal_main — no HTTP listener on 8000. Healthcheck always failed.
Added —no-healthcheck to the worker docker run flags in scripts/deploy-workers.sh. Worker process liveness is already covered by —restart unless-stopped (docker restarts a crashed worker automatically). A proper Dramatiq-broker-ping or process-alive sidecar is Sprint 4 monitoring work; for now the docker status reflects reality.
Deploy-workers via inline ssh kept tripping BusyBox sh
CI deploy step ended with /bin/sh: syntax error: unterminated quoted string after running the first worker container successfully. Tried first as a multi-line ssh hlvm-at-host with a heredoc-style body and backslash continuations inside double quotes; that bailed before the docker runs ever started. Rewrote as a single-line ssh per worker with semicolon-separated remote commands; that got past the first docker run (container ID returned) then bailed before the second.
Could not pin down exactly where the unterminated quote was — byte-checked the YAML for hidden chars, parsed the rendered shell script as best as visible, no smoking gun. Whatever it was, multi-statement remote commands inside an outer double-quoted string passed via ssh did not survive the BusyBox sh parser on the alpine deploy container. The exact same error message appeared in two structurally-different attempts.
Stopped trying to win the quoting battle. Wrote scripts/deploy-workers.sh as a real bash script in the repo (set -euo pipefail, takes positional args ENV / COMMIT_SHA / SHORT_SHA / IMAGE, runs the three docker stop / rm / run sequences, ends with docker ps filtered on worker labels). CI now scp-s the script to .171 then runs ssh hlvm-at-.171 bash /tmp/aros-deploy-workers.sh with the four args expanded inline. No multi-statement remote command, no inline-quoting surface. Worked on the first try.
Migration ownership: tables came up as postgres, app role could not ALTER
Operator applied 0006-0009 on dev via docker exec aros-backend-dev alembic upgrade head. Tables landed but subsequent app-role queries hit ownership-related errors when ALTERing, and the FORCE RLS on the new tables behaved oddly because the table owner was not the role doing the inserts.
An earlier bootstrap step had run alembic under the postgres superuser (because that role was what got connected at the time). The Sprint 3 migrations then created the new schemas / tables owned by postgres. The application role aros_backend_dev had USAGE / SELECT / INSERT / UPDATE on them but not the implicit ALTER that PG grants to the OWNER. FORCE RLS makes the table owner subject to the policy too, so an INSERT from the app role against a table owned by postgres can land in surprising places.
Operator-side REASSIGN OWNED BY postgres TO aros_backend_dev for the new schemas, then re-applied. Recorded as Sprint 4 hardening: alembic should always run as the per-env app role, never as postgres, so the migration-time ownership matches the run-time role. The CI migration guard runs against a fresh PG so it cannot detect this; only a stamped-past DB with a different historical owner trips it.
Forgejo forward-auth blew up git clone + Woodpecker OAuth
Operator extended the oauth2-proxy forward-auth gate to cover the Forgejo + Woodpecker subdomains. Result: git clone over HTTPS returned redirects instead of asking for HTTP basic auth, and Woodpeckers OAuth callback (used to enroll Forgejo users) got intercepted by the gate before it could complete.
Forgejo expects HTTP basic auth for git-over-HTTPS — a tool flow that does not understand OIDC redirects. Woodpecker uses an OAuth dance against Forgejo; that callback is itself an internal Aros HTTP surface that the gate then re-redirected, breaking the loop.
Operator-side: bypass forward-auth for git-over-HTTPS paths and for the Woodpecker OAuth callback URL. Documented as a Sprint 4 audit task — any tenant-facing auth layer needs to enumerate every internal HTTP surface Aros depends on before being enabled cluster-wide. Worth a dedicated runbook + a CI smoke test that hits these paths post-deploy.
Premature recap MDX with an acorn parse error
Wrote the first Sprint 3 recap right after Phase D pushed. Operator pulled me back: the recap should not be published until sprint close is confirmed, AND it should document the troubleshooting trail not just the happy-path receipts. The MDX itself broke the projects-recap build with an acorn parse error in one of the BentoGrid body strings (an apostrophe pattern the JS parser could not handle).
Two distinct problems on the same artefact: process (recap timing) and authoring (MDX body strings that contain apostrophes can fall over depending on adjacent characters because MDX is parsed as JS-with-JSX). Both rooted in the same haste — wanted to wrap the sprint visually before the actual close-out work was done.
Deleted the premature MDX from projects-recap to unblock the build. Saved a memory rule (recap-site-timing) covering the operator preference: do not generate the recap until they confirm sprint close, AND include the troubleshooting trail as load-bearing material. This recap (the one you are reading) was written only after the close-out adapter fixes, the deploy script rewrite, and the docs reconciliation all landed and the operator green-lit the rewrite.
Everything that landed in Sprint 3
On top of the Sprint 2 substrate. The adapter framework, the policy engine, the task system, and the platform health endpoint are all new.
Operator gate to Sprint 3 close + Sprint 4 setup
Implementation is done; promote + tag run through the operator queue. Sprint 4 picks up from the close-out tracked debt.
- Verify /api/v1/platform/health on dev shows all five adapters healthy — patroni leader running plus replicas streaming, redis reachable, minio reachable, backend 3/3 with self-IP recorded, edge probes green.
- Promote dev to test, then test to UAT via Woodpecker manual triggers. Run alembic upgrade head on each, redeploy workers via scripts/deploy-workers.sh, smoke-test.
- Tag sprint-3-uat-green at aros-backend HEAD once the chain runs clean through all three envs.
What we chose not to fix this sprint
Items called out in the ADRs or surfaced during Sprint 3 close-out that do not block the tag but want attention before Sprint 4 builds on top.
- policy·OPA-sidecar swap. ADR-0018 says validate the schema with the embedded evaluator first, then swap to OPA. Sprint 4 work — needs OPA in the container topology, Rego compilation from the JSON policy schema, and the bundle-push mechanism from the policy service.
- approvals·Approval execution + expiry actors. Sprint 3 ships approval lifecycle through approved / rejected. The Dramatiq actor that re-runs the approved request under the requester identity is Sprint 4. So is the periodic actor that scans for pending past expires_at.
- adapters·Mocked unit tier for adapter contract tests. Sprint 3 contract tests hit real (unreachable in CI) network targets and rely on the timeout classification path. Sprint 4 will mock external services per adapter to keep the unit tier fast, and reserve the real probes for the staging E2E suite per ADR-0020.
- workers·seal.env separation from env. ADR-0021 specifies the seal worker should mount /etc/aros-backend/seal.env in addition to the shared env file. Sprint 3 reuses the shared file. Sprint 4 Ansible work.
- migrations·Migration runner role must match app role. Sprint 3 close-out flushed an ownership bug where tables created via alembic-as-postgres tripped FORCE RLS on subsequent app-role inserts. Sprint 4 should add a CI check that the alembic-applied owner matches the per-env app role (aros_backend_dev / _test / _uat) before merging.
- vault·Vault Agent for adapter credentials. ADR-0017 specifies adapters should read credentials from Vault Agent templated files. Sprint 3 reads env vars. Sprint 4+ swap to per-adapter env files in /etc/aros-adapters/ named for each adapter.
- edge·Forward-auth audit of internal HTTP paths. The Forgejo + Woodpecker breakage during Sprint 3 close-out shows that adding a tenant-facing auth layer needs an enumeration of every internal HTTP surface Aros depends on. Sprint 4 should add a smoke-test runbook for the edge gate.
- monitoring·Worker liveness signal beyond restart-unless-stopped. Workers showed Up (unhealthy) until we set —no-healthcheck. A proper Dramatiq-broker-ping or process-alive sidecar so Prometheus can alert on stuck-without-crash is Sprint 4 monitoring stack work.