The chain itself
Every audit entry’s hash_self includes the previous entry’s hash_prev. The first entry chains off a fixed genesis hash — sha256(“aros-audit-genesis-v1”). Tampering with any block breaks every block after it.
Click a block to see the eleven canonical fields that go into the SHA-256 input; the dashed connector to the previous block lights up. The seal manifest below shows what gets sealed into MinIO every minute under aros-audit-seals/seals/dev/….
every 60 seconds the seal manifest that goes to MinIO ▾
{
"version": 1,
"env": "dev",
"sealed_at": "2026-05-26T14:06:30.481234+00:00",
"range": {
"first_id": "01HZK7…BA01",
"last_id": "01HZK7…BA06",
"count": 6
},
"chain_state": {
"head_hash": "f8d41362c290710ec1d3e6f7a8b9c0d1...",
"genesis_hash": "c6f8d79e8adbafc09addf281e30b0bf7..."
},
"integrity": "sha256(canonical(manifest \\ integrity))"
} From logged-in browser to discovery-grade audit
Sprint 1.5 closed the loop browser -> backend -> database. Sprint 2 makes the trip auditable, multi-tenanted, and event-driven. Five backend modules, two migrations, eight ADRs, one CI safety net.
tenants · audit · events · 3 tasks · seal
0003 tenants · 0004 audit · 0005 seal-admin
0005, 0006, 0007, 0008, 0010, 0011, 0016 + 0015 reserved
three real migration bugs caught before operator deploy
One day, autonomous loop, five iterations to clean migrations
Most of Sprint 2's runtime was spent on getting the migration chain right. The implementation itself was straightforward; the safety net to validate it ended up being the most valuable artefact of the sprint.
- Morning
core.tenants ships
Migration 0003 creates tenant.tenant + tenant.membership with check constraints on status and retention_policy. Default tenant seeded with well-known UUID. Middleware activated to resolve identity + tenant per request. Session dependency emits set_config(app.tenant_id) so subsequent queries see RLS via the policy that lands in 0004. - Midday
First migration bug — :id::VARCHAR coercion
Operator reported migration 0003 failing on dev. The default-tenant INSERT bound :id as a Python str; psycopg types that VARCHAR, so VALUES (uuid-string ::VARCHAR, ...) failed to coerce into the UUID column. Initial fix added :id::UUID. CI guard later caught that SQLAlchemy text-parameter regex skips :name when followed by :: (the PG cast operator), so the bind never substituted. Settled on inlining the UUID as a literal — matches the no-bind-params-in-DDL pattern used elsewhere. - Midday
core.audit ships
Migration 0004 drops the 0001 stub audit.event, recreates it partitioned by RANGE(occurred_at) MONTHLY with six composite indexes per ADR-0007. audit.chain_head single-row pointer table seeded with the genesis hash c6f8d79e... — sha256("aros-audit-genesis-v1"). ENABLE + FORCE RLS with tenant_isolation policy. Pointer-table refinement chosen over the ADR text SKIP LOCKED: SKIP LOCKED on the entry table returns 0 rows when the head is concurrently locked, which any reasonable interpretation reads as "chain empty -> use genesis hash" — fork risk. Documented as ADR-0007 amendment. - Afternoon
Second migration bug — partition bound parameter typing
CREATE TABLE ... PARTITION OF ... FOR VALUES FROM (:lo) TO (:hi) failed with "could not determine data type of parameter $1". DDL has no column context, so psycopg cannot infer the bind type. Fix: f-string interpolation of pre-formatted date literals — the inputs come from int year/month through f-strings already, no injection vector. - Afternoon
CI migration guard lands
New Woodpecker step runs three passes against postgres:16-alpine sidecar: alembic upgrade head -> downgrade base -> upgrade head. Catches type-cast bugs (forward), broken rollbacks (reverse), and missing IF NOT EXISTS on schema creates (re-forward). Gates build-and-push so a migration bug never reaches an operator. ADR-0011 written to pin the pattern. - Afternoon
Third migration bug — 0001 downgrade after 0004 CASCADE
Three-pass guard fail: 0001 downgrade tried to DROP INDEX audit.ix_audit_event_tenant_occurred which 0004 downgrade had already CASCADE-dropped with audit.event. Plain DROP fails when the object is gone. Batch fix: IF EXISTS on every DROP INDEX / DROP TABLE / DROP SCHEMA across all four downgrades. Each migration now stands alone — its downgrade only references objects it created and tolerates whichever order the chain runs in. - Evening
core.events ships
Redis Streams publisher via redis.asyncio. Stream naming aros:module:event_type, consumer-group convention cg:consumer_module. MAXLEN approx 10000 trim. Tenants router retrofits tenant.created + tenant.updated events; audit module emits audit.entry.created. Publish happens after the request transaction commits via FastAPI BackgroundTasks. Empty REDIS_HOST = publishers log and skip — graceful degradation pattern matches the watchdog and seal tasks. - Evening
Background tasks: watchdog + partition pre-create
Two asyncio loops in the FastAPI lifespan per ADR-0010. Watchdog reads chain_head (global, no RLS issue) every 5 min and POSTs to WATCHDOG_URL (empty = compute-but-skip, since Sprint 4 ships the consumer). Partition pre-creator wakes daily, ensures the today+30d monthly partition exists, idempotent via information_schema. pg_try_advisory_xact_lock for leader election across the three dev nodes — only one runs per cycle. - Late
ADR-0016 proposal -> revision -> acceptance
Seal task needs cross-tenant reads on audit.event under FORCE RLS. Original ADR-0016 draft proposed an app.operator_mode GUC override on the existing policy. Operator rejected in favour of a dedicated aros_seal_admin role with BYPASSRLS — role-level bypass is the standard PG mechanism, gives credential isolation, scopes the blast radius. ADR-0016 rewritten Proposed -> Accepted. Migration 0005 creates the role and grants SELECT + UPDATE (no INSERT, no DELETE — the seal task only annotates). - Late
Seal task ships
Third asyncio loop in the lifespan. Connects to PG via a separate AsyncEngine with seal-admin credentials — the request-side engine is never touched. Every 60 s reads unsealed entries across all tenants, builds the manifest per ADR-0007 (version, env, sealed_at, range with first_id+last_id+count, chain_state with head_hash+genesis_hash, integrity field = SHA-256 of the manifest excluding integrity), uploads to aros-audit-seals bucket with object-lock governance and a 7-year retention, stamps sealed_at + seal_ref in the same transaction. Bucket created on first cycle if missing. - Late
Frontend swaps + ai-context reconciliation
aros-frontend /tenants and /audit now use the typed openapi-fetch client against the real /api/v1/* endpoints. SimulatedBadge and amber pending banners removed. TopBar tenant switcher pulls live tenants. schema.d.ts hand-stubbed for the Sprint 2 surface. 02-current-sprint.md flipped from "kickoff" to "implementation done"; 09-sprint-2-plan.md gets a status banner. ADR README index already up to date.
The CI guard paid for itself before the operator saw a single one
Each of these failed locally on dev OR would have failed on test/UAT when the operator ran alembic upgrade head. The three-pass cycle catches them on the first push — before any image is built, let alone deployed.
Migration 0003 default-tenant seed: bound parameter never substituted
Operator’s first alembic upgrade head on dev failed.
Then a second iteration (with :id::UUID added for an explicit cast) passed on dev BUT failed the CI three-pass guard on a fresh DB — :id was apparently not substituting.
Two distinct issues stacked. First, the original INSERT INTO tenant.tenant (id, ...) VALUES (:id, ...) bound :id as a Python str; psycopg types parameters from Python types, so the value arrived at PG as VARCHAR and the implicit cast into a UUID column failed. Adding :id::UUID looked like the fix — and worked on dev because dev had already stamped past 0003 by the time the corrected code ran. On a fresh DB, the CI guard surfaced the actual bug: SQLAlchemy’s text() parameter regex deliberately skips :name when followed by :: (the PG cast operator), so :id::UUID is parsed as a literal :id followed by ::UUID. The bindparam value never substituted; the INSERT ran with the literal text :id in it.
Inline DEFAULT_TENANT_ID as a literal in the f-string; drop the .bindparams() call entirely. Matches the no-bind-params-in-DDL pattern used elsewhere. DEFAULT_TENANT_ID is a fixed constant from module code (no injection vector), and PG coerces string literals to UUID at INSERT time without needing an explicit cast.
Migration 0004 partition CREATE: could not determine data type of parameter
alembic upgrade head errored on dev: could not determine data type of parameter $1 on each CREATE TABLE audit.event_2026_05 PARTITION OF audit.event FOR VALUES FROM (:lo) TO (:hi).
DDL statements don’t carry column context, so psycopg cannot infer the type for a bound parameter inside a partition-bound clause. The driver sends the parameter as TEXT by default, and the partition bound parser refuses without an explicit cast.
Interpolate the bounds directly into the SQL via f-string. The inputs come from int year/month already, formatted through f"{year:04d}-{month:02d}-01 00:00:00+00" — no injection surface. Applied the same treatment to the chain_head seed INSERT for consistency: DDL and seeds inline; app-time DML uses binds.
Migration 0001 downgrade after 0004 CASCADE: object already gone
CI migration guard’s three-pass cycle failed on pass 2 (reverse): 0001.downgrade() tried DROP INDEX audit.ix_audit_event_tenant_occurred but the index no longer existed — 0004.downgrade() had already cascade-dropped it with audit.event.
Each migration’s downgrade() was written in isolation, referencing only the objects that migration created. That’s correct in spirit, but 0004.downgrade() uses DROP TABLE audit.event CASCADE, which removes the index 0001 created. By the time 0001.downgrade() runs, the index is gone — plain DROP fails.
Sweep every downgrade() and add IF EXISTS to every DROP INDEX, DROP TABLE, and DROP SCHEMA. Each migration still only references objects it created, but the drops are now idempotent under whatever the upstream cleanup happened to do. Three-pass cycle now passes end-to-end.
Tried to auto-migrate on deploy — contradicts ADR-0011
After the migration guard turned green, the obvious-looking next move was to make deploy-dev run alembic upgrade head automatically so dev doesn’t lag behind the image. Pushed the change, watched it fail mid-cycle with a quoting error in the ssh-into-vm bit.
ADR-0011 (Sprint 2 CI/CD strategy) had landed earlier that day with this exact alternative considered AND rejected: “Auto-migration on deploy — simpler operator workflow but removes the safety gate. A bad migration would automatically corrupt the live database before the operator can intervene. Rejected: safety over convenience.” Missed the ADR on first read; the quoting error was incidental.
Reverted the auto-migrate. Restored the original deploy step. The CI migration guard already proves the chain against a fresh PG on every push, so the migration shape is validated automatically — it’s only the live application that stays operator-gated. The retreat to the ADR position took one commit.
Everything that landed in Sprint 2
On top of Sprint 1.5's Authentik + oauth2-proxy + FastAPI + frontend shell. The audit substrate, the tenant model, and the events bus are all new.
Operator gate to Sprint 2 close
Implementation is done; the path to tag sprint-2-uat-green runs through the operator queue.
- alembic upgrade head on dev — applies migration 0005 (creates aros_seal_admin BYPASSRLS role + grants). Operator-initiated per ADR-0011.
- Env file rollout — Ansible distributes SEAL_PG_PASSWORD (from secret/aros/seal-admin) plus the four MINIO_* values (from secret/aros/minio) into /etc/aros-backend/env on dev/test/UAT VMs. Until then the seal task no-ops at startup with a “credentials missing” log line.
- Promote dev to test — Woodpecker manual trigger with TARGET=test, run alembic upgrade head on a test VM, smoke-test the audit + tenants endpoints.
- Promote test to UAT — same procedure on UAT.
- Tag sprint-2-uat-green at aros-backend HEAD once the chain runs clean through all three envs.
What we chose not to fix tonight
Three operational items called out in the ADRs that don't block Sprint 2 close but need attention before Sprint 3 builds on top.
- data·Per-tenant retention policies not yet wired through to seal manifests. Every seal currently uses the 7-year governance default. ADR-0007 supports standard / extended / custom per tenant; core.tenants exposes the field on writes. The seal task needs to consume retention_policy on a per-tenant-slice basis. Sprint 3 candidate.
- audit·Integrity verifier endpoint returns valid unconditionally. The chain_head pointer + entry_count are real, but the per-entry re-verification (re-compute hash_self for every entry and check links) hasn’t shipped. Operator-only runbook work or a Sprint 3+ endpoint.
- events·No event consumers yet. Audit + tenants publish, no module subscribes. Sprint 3’s core.tasks + notifications are the first consumers. Consumer group naming and lag monitoring drop in then.
- watchdog·Watchdog digest consumer is Sprint 4 (core.cluster). The publisher computes the digest every 5 min; WATCHDOG_URL empty = no POST. Ships properly when the adjacent service lands on independent infrastructure.
- fe·schema.d.ts hand-stubbed for the Sprint 2 surface — regenerate from live /api/openapi.json once the operator’s dev session is reachable from the workstation (CORS or VPN, operator preference).