Projects / recap
Aros/Session recap/
Sprint 0

Sprint 0 — bootstrap, end-to-end CI/CD, and an HA Postgres bonus

Provisioned 13 VMs, shipped a working dev→test→UAT pipeline, and bolted on a 3-node Patroni cluster before sunrise.

One overnight session
909ea5b
terraform
ansible
woodpecker
patroni
etcd
postgresql
vault
proxmox
Infrastructure
13 VMs

9 app-tier (dev/test/UAT × 3), 3 PG (data-dev HA), 1 meta. All Terraform-provisioned, Ansible-hardened, PBS-backed.

Distribution
3 PVE hosts

pve1, pve2, pve3 — every HA tier spans all three.

Pipeline
dev → test → UAT

One commit, three promotable environments. SSH deploys, registry-pushed images, manual gating.

Data tier
etcd + Patroni 4.x

3-node PostgreSQL 16 cluster. libpq multi-host. ~35s failover, no data loss.

Secrets
5 vault bundles

Proxmox, Terraform AppRole, Forgejo, Woodpecker, PG.

Status
Sprint 0 ✓

Entry conditions met. Sprint 1 begins with HA database ready.

What we built

A reproducible platform foundation, end to end

Sprint 0 was scoped as 'establish the build pipeline.' What landed by morning was the entire base of the Aros platform: 12 production VMs split across three Proxmox hosts, a CI/CD pipeline that has already shipped a real service through three environments, and a highly-available PostgreSQL cluster that's been stress-tested and ready for Sprint 1 code.

VMs provisioned
13

all in IaC, all backed up

Repos shipped
3

docs / infra / platform

PVE hosts utilised
3

pve1, pve2, pve3

Failover test
35s

no data loss

Environments live
3

dev / test / uat

HA tiers
2

app + database

Pipeline triggers
manual

TARGET-gated for promotion

Auth model
OAuth

Forgejo SSO for Woodpecker

The narrative

How the session actually unfolded

Not the polished retro — the actual chronology, including the dead ends and the bonus rounds.

  1. 21:00

    Kicked off with TF + Vault working, repos initialised

    Inherited state: Proxmox cluster healthy, Forgejo + Woodpecker stack running on the legacy VM, Vault on iaccn01 holding Proxmox + AppRole credentials. Three empty repos waiting for code.
  2. 21:30

    First TF apply: 3 dev VMs (pve2 only)

    Deliberate deviation — dev cluster doesn't need HA. Documented in current-sprint.md as Sprint 1 debt.
  3. 22:00

    Cross-host clone broke on test/UAT environments

    Terraform tried to clone the template VM from pve2 to pve1/pve3 — failed because the source datastore name doesn't exist on the destination node.
  4. 22:20

    Replicated template to all three PVE hosts (9000/9001/9002)

    Per-node template lookup via var.node_templates. TF now picks the local template on each target node. Test + UAT applied cleanly.
  5. 22:45

    Ansible base playbook across all 9 VMs

    Hardened: unattended-upgrades, ssh config, Docker engine, ufw, NTP. Distributed CI deploy key alongside the operator workstation key.
  6. 23:10

    Hello-world FastAPI + Dockerfile + .woodpecker pipelines

    Three pipelines: build-push, deploy-dev (auto on push), promote-test (manual), promote-uat (manual). All wired to push images to forgejo.ryanit.com/homelab/aros-platform/hello.
  7. 23:40

    Webhook from Forgejo to Woodpecker silently dropped

    Push events triggered nothing. ALLOWED_HOST_LIST=private in compose env was overriding app.ini — and self-hosted *.ryanit.com is "external" from Forgejo's perspective.
  8. 00:05

    Forgejo config patched, webhook flowing

    Set ALLOWED_HOST_LIST=*.ryanit.com,external. First green build landed seconds later. Hello-world deployed to dev.
  9. 00:25

    Promote to test + UAT, verified live at hello-{test,uat}.ryanit.com

    One commit, two manual triggers, six VMs updated. Pipeline pattern locked in.
  10. 00:50

    Started HA Postgres prep (bonus scope)

    TAD mandated HA. Provisioned 3 dedicated VMs on pve1/pve2/pve3 for the data tier. etcd installed via apt, Patroni via Python venv at /opt/patroni.
  11. 01:30

    Patroni bootstrap silently skipped replication user creation

    Replicas couldn't attach. Turns out Patroni 4.x doesn't auto-create users defined under postgresql.authentication.* — they have to be in bootstrap.users. Fixed and templated.
  12. 02:00

    Cluster healthy, failover tested twice

    Stopped leader → ~35s election → write traffic resumed via libpq multi-host. Old leader rejoined as replica on restart. Zero data loss across three committed rows.
  13. 02:20

    Sprint 0 closeout doc + ADR-0002 + all repos pushed

    aros-docs 909ea5b, aros-infrastructure 57933a0, aros-platform 4ddd9b8. Sprint 1 entry conditions met.
  14. 02:35

    Provisioned hub-1 (meta tier) for this recap site

    New TF environment, same module pattern. Ansible base in 3 minutes. Docker ready. This page is being written into it now.
Challenges

The four things that didn't work the first time

Every one of these became a one-line fix in retrospect. Documenting them so the next pass — or the next operator — doesn't lose the same hour.

Cross-host VM cloning failed on the second-environment apply

terraformproxmox Moderate
Symptom

First terraform apply for the test environment hung, then errored with datastore 'Z1-1' does not exist on node pve1. The provider was trying to clone a template from pve2 over to pve1 by name, but Proxmox storage names are per-node.

Root cause

The bpg/proxmox provider clones by VMID + source-node and assumes the datastore exists at the destination. Our template lived only on pve2 with a node-local datastore name.

Resolution

Cloned the template to all three PVE hosts (9000 on pve2, 9001 on pve1, 9002 on pve3) via the Proxmox API. Added var.node_templates map keyed by target node so Terraform picks the local template. Sprint 1 debt: automate template replication.

Forgejo webhooks silently dropped events to Woodpecker

forgejowoodpeckernetworking Major
Symptom

Push events showed delivered in Forgejo’s webhook log, but Woodpecker never received them. No errors, no entries in the pipeline.

Root cause

Forgejo’s ALLOWED_HOST_LIST was set to private via the compose FORGEJO__webhook__ALLOWED_HOST_LIST env var, which overrode the app.ini. Anything routed through our public *.ryanit.com URLs counted as “external,” so Forgejo refused to dispatch.

Resolution

Switched the env var to *.ryanit.com,external. Restarted the Forgejo container. First commit afterwards triggered a green build inside 5 seconds. Bonus discovery: append-to-app.ini left a duplicate [webhook] section, which required extracting / editing / putting back with chown 1000:1000.

Manual Woodpecker triggers fanned out to every workflow

woodpeckerci Minor
Symptom

Hitting “manual trigger” on the Woodpecker UI fired all three promote workflows simultaneously — not just the one you wanted.

Root cause

Default behavior of event: manual is to fire every workflow that subscribes to it. There’s no built-in “pick one” UI.

Resolution

Gated every promote workflow with when: evaluate: 'TARGET == "test"' (or uat). User passes a TARGET variable on the manual trigger; only the matching workflow runs. Clean, declarative, no plugin needed.

Patroni 4.x didn't create the replication user from postgresql.authentication

patronipostgresql Major
Symptom

Leader bootstrapped fine. Replicas crashed at start with FATAL: role "replicator" does not exist.

Root cause

In Patroni 4.x, the postgresql.authentication.* section only configures which credentials Patroni uses — it does NOT auto-create those roles. Roles must be declared in bootstrap.users (which only runs on first init).

Resolution

Created replicator and rewind_user manually on the running leader, then patched the patroni.yml.j2 template so future clusters get them at bootstrap. Failover tests passed immediately afterwards.

Stack

What's in the box

Everything is open-source, self-hosted, and reproducible from the three Forgejo repos.

Hypervisor
Proxmox VE 8PBSUbuntu 24.04 (noble)
IaC
Terraformbpg/proxmox providerhashicorp/vault provider
Configuration
Ansiblecommunity.docker
Secrets
HashiCorp VaultAppRole authKVv2 backend
Source + Registry
Forgejo 9Woodpecker v3OCI registry
Data tier
PostgreSQL 16Patroni 4.xetcd 3.4libpq multi-host
Platform
FastAPIDocker 29
Edge
Traefik v3Cloudflare DNS
Observability
ntfy
Validation

Proof it works

Two artifacts captured during the session, kept here as receipts.

hello-world deploy receipt
bash 
# Push from workstation
$ git push origin main
# ... commit 9f49926d ...

# Woodpecker fires automatically:
#   build-push → forgejo.ryanit.com/homelab/aros-platform/hello:dev
#   deploy-dev → ssh to 192.168.1.171/172/173, pull + recreate

$ curl https://hello-dev.ryanit.com/
{"message":"hello from aros","env":"dev","commit":"9f49926d","host":"aros-dev-2"}

# Promote with TARGET=test:
$ curl https://hello-test.ryanit.com/
{"message":"hello from aros","env":"test","commit":"9f49926d","host":"aros-test-3"}
Patroni failover receipt
bash 
$ patronictl -c /etc/patroni/patroni.yml list
+ Cluster: aros-pg-dev ------+----+-----------+
| Member        | Host        | Role    | State |
+---------------+-------------+---------+-------+
| aros-pg-dev-1 | 192.168.1.231 | Leader  | running |
| aros-pg-dev-2 | 192.168.1.232 | Replica | streaming |
| aros-pg-dev-3 | 192.168.1.233 | Replica | streaming |

$ ssh aros-pg-dev-1 sudo systemctl stop patroni
# 35 seconds later...

| aros-pg-dev-1 | 192.168.1.231 | -       | stopped   |
| aros-pg-dev-2 | 192.168.1.232 | Leader  | running   |
| aros-pg-dev-3 | 192.168.1.233 | Replica | streaming |

# Three rows in sprint0_ha_test committed across the failover. No data loss.
What's next

Sprint 1 enters with the runway built

The point of the bootstrap sprint is that the next sprint can be all about product code, not infrastructure. Sprint 1 starts from a database that's already HA, a pipeline that's already validated, and three environments that already exist.

Sprint 1 entry conditions
  • Sprint 0 closeout complete, all three repos pushed to current HEADs
  • 13 VMs operational, hardened, backed up via PBS daily at 03:15
  • CI/CD validated end-to-end across dev → test → UAT
  • IaC patterns established (per-env Terraform, role-based Ansible)
  • HA PostgreSQL operational at 192.168.1.231,232,233
  • Sprint 1 kickoff: aros-backend modular monolith — FastAPI + SQLAlchemy 2.x async + Alembic, first migration creates tenant-aware schema
Tracked debt

What we chose not to fix tonight

Every shortcut explicitly noted so it's visible at Sprint 1 kickoff rather than rediscovered in production.

  • infra·Dev cluster runs single-PVE — re-evaluate with shared storage in Sprint 1.
  • infra·Terraform state is local — migrate to MinIO S3 backend once MinIO exists.
  • infra·Per-node template replication is manual — automate via Packer or scripted clone.
  • infra·Shared terraform AppRole — split into per-env roles for blast-radius containment.
  • data·No HAProxy / pgcat — fine for libpq clients, will need it for the first non-libpq consumer.
  • data·Patroni watchdog disabled — switch to required for fencing once we trust the cluster.
  • data·pg_hba is host all all 192.168.1.0/24 — tighten per-app / per-db in Sprint 1.
  • ci·Backup job uses VMID list — convert to tag-based selection.
  • ci·aros-ci ntfy user not yet created — pipeline notifications currently best-effort.
  • ci·Pipelines mount docker.sock on the Woodpecker agent — replace with dedicated build runner.
  • secrets·Vault Agent integration deferred — credentials still piped through /tmp during playbook runs.
  • recap·This site has no auth and no pipeline — add oauth2-proxy + Woodpecker pipeline in Sprint 1.