FastAPI modular monolith, SQLAlchemy 2.x async over psycopg3, libpq multi-host into the Patroni cluster. /health/db returns the live PG version and current leader IP. 32 files, 1650 lines.
570596d2d9 through all three environments via Woodpecker. Tagged sprint-1-uat-green at UAT close.
Stopped Patroni on the leader (.231) at 17:44:57 UTC; the next probe (2s later) already reported the new leader at .233. No "(no response)" lines in the captured 8-probe window.
Auth, Authentik, Frontend, and Identity deferred to Sprint 1.5. ADR-0004 introduces half-sprints as the catch-up mechanism and revises the v1.0 forecast from 11 → 6–8 months.
ci.yml + promote-test.yml + promote-uat.yml copied wholesale. Image at forgejo.ryanit.com/homelab/aros-backend/api.
All 9 hello containers stopped + removed; hello-* Traefik routes pruned; port 8000 released for the real backend.
One commit, three environments, zero dropped requests
Sprint 1 was scoped as 'first real Aros code through the existing pipeline.' What landed was the entire backend skeleton — app factory, middleware stack, async DB engine, first migration, three health endpoints — promoted under one commit to all three environments and stress-tested by killing the Patroni leader.
aros-backend initial scaffold
app + tests + CI + Dockerfile
dev + test + UAT, 3 each
no FAILED in the curl loop
The actual deploy → test → UAT arc
Three issues, three resolutions, then a leader kill. Times are relative to scaffold drafting.
- 00:00
aros-backend scaffold drafted
FastAPI app factory, async engine via psycopg3 multi-host, ULID request-id middleware, tenant-header middleware, Alembic 0001 (audit + tenant schemas), .woodpecker/ pipelines mirrored verbatim from aros-platform. - 00:05
Cross-repo bootstrap: Vault secret, Ansible playbook, Traefik routes
Bootstrap SQL ran on the Patroni leader via libpq multi-host. /etc/aros-backend/env distributed to all 9 app VMs. aros-{dev,test,uat}.ryanit.com routes added to /home/mvm/stacks/traefik/data/config/aros-bootstrap.yml. - 00:20
First push: Woodpecker hook did not fire
ci.yml gated on push|pull_request only. Added event: manual and pushed an empty commit to nudge the pipeline. Next push fired clean. - 00:30
Lint failed (ruff UP035 / UP007 / B008)
Stock Alembic ships old-style typing (Union, typing.Sequence); router used Depends() in argument defaults (FastAPI norm, flake8-bugbear flags it). Modernized typing in both the migration and the script.py.mako template; switched router to FastAPI Annotated dependencies (SessionDep / SettingsDep aliases). - 00:40
Build-and-push failed: 401 Unauthorized
docker login succeeded; the bearer-token negotiation for repository:homelab/aros-backend/api:push,pull then returned 401. Forgejo token in Vault was stale and lacked write:package on a new package path. - 00:55
Token regenerated, Vault path secret/aros/forgejo/admin-token
New admin token with write:package org-wide. Woodpecker org-level secret updated. Build-and-push went green on the next manual trigger. - 01:05
Deploy-dev failed: --env-file: permission denied
/etc/aros-backend/env was root:root 0600. The Woodpecker deploy step SSHs as hlvm; docker --env-file reads as the invoking user (hlvm), not root. - 01:15
Ansible playbook fixed: env file root:hlvm 0640
Idempotent re-run rewrote perms on all 9 VMs. Directory stayed root:root 0755. Deploy moved on — straight into the next failure. - 01:25
Port 8000 still occupied by hello-dev
The hello decommission script had been run and reported success — but ssh hlvm@host "sudo docker stop hello-dev" had silently no-op-d. hlvm has no non-interactive sudo for docker; the prompt timed out without a TTY, the SSH session returned 0, and the script moved on. - 01:35
Decommission re-run with aros-ci-deploy key
aros-ci-deploy has non-interactive sudo configured. Hello containers stopped + removed in one pass; hello-* Traefik routes pruned; port 8000 released. - 01:45
Deploy-dev green; /health/db responding
curl https://aros-dev.ryanit.com/health/db | jq returned status=ok, pg_version, leader=192.168.1.231, request_id (ULID), host (aros-dev-N hostname), commit=570596d2. - 01:55
Promote to test, then UAT
TARGET-gated manual triggers in Woodpecker UI. Both promotions green. /health/db responses confirmed on aros-test.ryanit.com and aros-uat.ryanit.com — same commit, different host stamps per request as Traefik round-robins. - 02:10
Patroni leader-kill demo
Stopped Patroni on aros-pg-dev-1 (.231); systemd logs the unit exiting at 17:44:57 UTC. Next /health/db probe at 17:44:59 already reports the new leader at 192.168.1.233 — graceful failover inside one 2s probe interval. No "(no response)" lines in the 8-probe window. Patroni restarted via Ansible at 17:45:46 UTC; rejoined as replica. - 02:25
Tagged sprint-1-uat-green at 570596d
Annotated tag pushed to Forgejo. Sprint 1 closed. ADR-0004 written to capture the scope recalibration; 02-current-sprint.md updated to open Sprint 1.5.
Three silent failures, in order
None of them were loud. All three returned exit 0 somewhere in the chain. Written up so the next pass — or the next operator — doesn't lose the same hour.
Forgejo token in Vault was stale and lacked write:package
Lint, typecheck, test, and image build all green. docker push hung on layer HEAD requests, then died with 401 Unauthorized on https://forgejo.ryanit.com/v2/homelab/aros-backend/api/blobs/sha256:.... The earlier docker login in the same step had reported Login Succeeded.
OCI distribution’s auth flow is two-stage: a successful docker login proves identity, but every subsequent operation negotiates a fresh bearer token scoped to repository:<name>:push,pull. The token in Woodpecker’s forgejo_token secret — sourced from a Vault entry that had been valid for writes to homelab/aros-platform/hello — was bound to a scope that didn’t include write:package for a new package path. Pull worked. Login worked. Push to a new package didn’t.
Regenerated the Forgejo personal access token with org-wide write:package. Stored at secret/aros/forgejo/admin-token. Updated the Woodpecker org-level secret to read from the new Vault path. Next pipeline run cleared build-and-push. Token rotation procedure added to the tracked-debt list.
/etc/aros-backend/env was root-only; docker reads it as the SSH user
Build-and-push green. deploy-dev died on the first VM with docker: --env-file: open /etc/aros-backend/env: permission denied.
The Ansible playbook wrote the env file as root:root 0600 — the security-instinct default that doesn’t survive contact with the deploy reality. The Woodpecker deploy step SSHs as hlvm and runs docker run --env-file .... Docker reads the env file from the invoking user’s perspective (hlvm), not from inside the container as root. 0600 root:root means hlvm can’t read it. The container never starts.
Switched the playbook to root:hlvm 0640. Root still owns and rewrites via Ansible; hlvm gets read-only group access; everyone else is locked out. Directory stays root:root 0755. Idempotent re-run rewrote perms on all 9 VMs. Header comment in the playbook now spells out the constraint so the next reader knows why the group bit is load-bearing.
Hello decommission silently no-op'd — port 8000 stayed occupied
After fixing env-file perms, deploy-dev failed again with docker: Error response from daemon: ... Bind for 0.0.0.0:8000 failed: port is already allocated. The hello decommission script had been run earlier and reported success across all 9 VMs.
The decommission script SSH’d in as hlvm and ran sudo docker stop hello-<env>. hlvm doesn’t have non-interactive sudo for docker on the app VMs — that privilege is on aros-ci-deploy, the CI’s account, not the operator’s. The remote sudo call prompted for a password; the SSH session had no TTY allocated; the prompt timed out; sudo exited non-zero — but the outer SSH session still returned 0 because the script wasn’t checking. Reported success. Docker never received the stop command.
Re-ran the decommission with the aros-ci-deploy key. All 9 hello containers stopped + removed in one pass. hello-* router/service entries pruned from aros-bootstrap.yml on mvm. Port 8000 released. Rule recorded: any script that SSHs to an Aros app VM and needs sudo should use aros-ci-deploy, not hlvm — and should check the exit code of the inner command, not just the SSH session.
What's running in the container
Application code, runtime, edge — all open-source, all reproducible from the four Forgejo repos.
Killed the leader, watched the loop
The whole point of libpq multi-host with target_session_attrs=read-write is that the application doesn't need to know who the leader is — the driver finds it. Demonstrated by killing the leader and watching aros-backend keep serving.
# Probe /health/db every 2s. host = which app VM answered (Traefik round-robin);
# leader = which PG IP the response reports as the current Patroni leader.
$ for i in {1..40}; do
printf "%s " "$(date +%H:%M:%S)"
curl -s --max-time 3 https://aros-dev.ryanit.com/health/db 2>/dev/null \
| jq -r '"host=\(.host) leader=\(.leader)"' 2>/dev/null \
|| echo "(no response — failover in progress)"
sleep 2
done
17:44:51 host=aros-dev-2 leader=192.168.1.231
17:44:53 host=aros-dev-3 leader=192.168.1.231
17:44:55 host=aros-dev-2 leader=192.168.1.231
17:44:57 host=aros-dev-3 leader=192.168.1.231
17:44:59 host=aros-dev-1 leader=192.168.1.233
17:45:01 host=aros-dev-3 leader=192.168.1.233
17:45:03 host=aros-dev-1 leader=192.168.1.233
17:45:05 host=aros-dev-2 leader=192.168.1.233
# Every probe returned 200 with a JSON body. No "(no response)" lines.
# Traefik spread the load across aros-dev-{1,2,3} throughout.
# Leader flipped .231 → .233 inside one probe interval (2s).# Second SSH session: bring Patroni back up on the killed node.
$ ansible -i inventories/data-dev/hosts.yml aros_pg_dev \
-m service -a "name=patroni state=started" \
--become --limit aros-pg-dev-1
aros-pg-dev-1 | CHANGED => {
"changed": true,
"name": "patroni",
"state": "started",
"status": {
"Description": "Patroni - PostgreSQL HA cluster member",
"ActiveExitTimestamp": "Sun 2026-05-24 17:44:57 UTC", # ← leader killed
"InactiveEnterTimestamp": "Sun 2026-05-24 17:44:58 UTC",
"ActiveEnterTimestamp": "Sun 2026-05-24 17:45:46 UTC", # ← restart
"ActiveState": "active",
"SubState": "running",
"Result": "success",
"NRestarts": "0"
}
}
# Re-running the same Ansible task afterwards is idempotent (changed=false).
# Aligning with the curl loop:
# 17:44:57 Patroni exited on .231 (systemd ActiveExit)
# 17:44:59 first probe sees new leader .233 — 2s, one probe interval
# 17:45:46 Patroni restarted on .231; rejoins the cluster as a replica
# Failover was graceful: systemctl stop signals Patroni to release the
# leader lock in etcd cleanly, so a replica is promoted without waiting
# out the TTL. libpq's target_session_attrs=read-write probe found .233
# in read-write mode on the next reconnect.Sprint 1.5 — catch up Auth, Authentik, Frontend, Identity
Sprint 1 delivered the platform skeleton. Sprint 1.5 closes the loop from a browser, behind oauth2-proxy / Authentik, through the new identity endpoints, into HA Postgres. ADR-0004 spells out the half-sprint mechanism.
- aros-backend live on dev / test / UAT under commit 570596d2d9, tagged sprint-1-uat-green
- HA Postgres proven via application-level failover, not just patronictl switchover
- CI/CD pattern validated for a second service — aros-platform pattern transferred cleanly
- Vault secret layout established:
secret/aros/<service>-<env>per-service, jq-into-tempfile-then-rm extraction in playbook headers - Sprint 1.5 kickoff: oauth2-proxy at the edge, Authentik for SSO, React + TypeScript + Vite frontend scaffold, first identity domain module
What we chose not to fix tonight
Three items surfaced during Sprint 1 that weren't worth pulling into the close-out, but should be visible at Sprint 1.5 kickoff rather than rediscovered.
- forgejo·Token rotation procedure — Sprint 1 was bitten by a stale token in Vault that lacked
write:packageon a new package path. Document scope hygiene and rotation cadence. - ansible·Sudo-via-SSH exit codes — the hello decommission silently succeeded because the outer SSH session returned 0 even though the inner sudo timed out. Add explicit exit-code checks in any ad-hoc remote-sudo loops.
- infra·Network HA hardware (LACP/MLAG, UDM shadow, cellular WAN) — operator-only hardware work, scheduled independently of software sprints per ADR-0004. Not blocking Sprint 1.5.