The SSO handshake, end to end
Five swim lanes, nine messages. The browser hits Traefik, gets bounced through oauth2-proxy and Authentik, lands at Entra for the federated identity, comes back with an id_token, exchanges it for a session cookie, and the original request finally proceeds. Phase A was getting this loop to close without the browser ever seeing a raw 401 — four of the sprint’s silent failures live somewhere along this path.
From headless backend to logged-in browser app
Sprint 1 left the platform with a working /health/db but no concept of identity and no UI. Sprint 1.5 closed both gaps in two phases — auth in Phase A, the application shell in Phase B — over two days of focused work.
1 edge + 3 cache + 3 frontend; estate now 20
0009 frontend · 0012 Authentik · 0013 edge · 0014 Redis
four in Phase A, one in Phase B
Microsoft Entra → akadmin (bonus)
Day 1 evening through Day 2 close
Phase A took most of two days because every failure mode in the auth stack arrives as a successful HTTP response doing the wrong thing — empty bodies, missing claims, mis-quoted templates. Phase B took one focused session, four CI iterations, and a post-deploy backend fix.
- Day 1 · evening
Sprint 1 closed; Sprint 1.5 plan drafted
Tag sprint-1-uat-green at 570596d. ai-context/08-sprint-1.5-plan.md splits Phase A (auth + infra) from Phase B (frontend). - Day 1 · night
7 new VMs provisioned + base-hardened
Four commits across aros-infrastructure (meta/dev/test/uat stacks), one per env for clean plan reviews. Ansible base playbook applied verbatim — SSH keys, ufw, Docker, qemu-guest-agent. - Day 2 · morning · A
aros-edge.yml — Authentik install
New playbook spins up PostgreSQL 16 + Redis 7 (loopback only) + Authentik server/worker via Docker Compose with network_mode: host. - Day 2 · morning · A
First Authentik visit returns empty body
http://192.168.1.241:9000/ returns 200 with no content. Curl confirms empty body. Authentik's HTTP listener is unusable behind a reverse proxy. - Day 2 · morning · A
Pivoted to HTTPS 9443 + insecureSkipVerify
mvm Traefik route points at https://192.168.1.241:9443 with insecureSkipVerify=true (bundled self-signed cert). authentik.ryanit.com loads. Admin UI usable. - Day 2 · midday · A
Web UI: flow, provider, MFA configured
Default flow: identification → password → authenticator validation (TOTP + WebAuthn + Static). OAuth2 Confidential provider with redirect URIs for all three envs. Application slug "aros-backend". - Day 2 · midday · A
Microsoft Entra federation (bonus side-quest)
OAuth/OIDC federation source: user_matching_mode=email_link, user_path_template=users. Entra credentials vaulted at secret/aros/authentik/entra. Federated identity links to akadmin on first login. - Day 2 · afternoon · A
aros-edge.yml extended — oauth2-proxy + Traefik
/etc/aros-edge/oauth2-proxy.env (root:hlvm 0640 per Sprint 1 lesson). /opt/aros-edge/proxy/ compose stack with both containers on a shared bridge. Traefik file-provider config wires per-env routers + forward-auth middleware. - Day 2 · afternoon · A
Cookie secret bug — 44 chars vs 32 bytes
oauth2-proxy refuses to start. openssl rand -base64 32 → 44 ASCII chars; oauth2-proxy reads it as RAW bytes, not base64. Regenerated with openssl rand -hex 16 → 32 bytes valid. - Day 2 · afternoon · A
email_verified missing — SSO loop never closes
Browser auth completes, oauth2-proxy 403s with "email not verified". Authentik omits the claim entirely. Added OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL=true. - Day 2 · afternoon · A
Traefik errors template — the "Found." page
Browser hits /api/v1/me, sees blank page with just "Found." Traefik errors middleware only substitutes status and url placeholders; scheme/host pass through as literals and break the redirect. Shortened to /oauth2/start?rd=<url>. - Day 2 · late · A
aros-backend Step 6 — auth middleware + identity
Commit 65bf1ca. middleware/auth.py reads X-Auth-Request-* (plan said X-Forwarded-* but that's only oauth2-proxy proxy-mode). Identity module + migration 0002 (identity.user, gen_random_uuid() PK). - Day 2 · late · A
Per-env DB split discovered + fixed
aros-backend in dev pointed at aros_dev; test/UAT had been doing the same. Created aros_backend_test/uat roles + aros_test/uat DBs; rewrote /etc/aros-backend/env root:hlvm 0640 on all 9 app VMs; force-recreated containers (docker restart does NOT re-read env_file). - Day 2 · late · A
mvm Traefik patched + hello-* routes pruned
aros-dev/test/uat.ryanit.com forward to aros-edge-1:80 with X-Forwarded-Proto=https. Hello-world Traefik routes pruned. Hub-1 redeployed with the Sprint 1 recap as a bonus. - Day 2 · close · A
End-to-end browser auth + tag sprint-1.5a-uat-green
Hit aros-dev.ryanit.com → Authentik → Entra or password+MFA → land on /api/v1/me with the upserted user. Three ADRs landed: 0012 Authentik, 0013 edge topology, 0014 per-env Redis. Tag at c2b1721. - Day 2 · later · B
Phase B kickoff: aros-frontend repo + scaffold
Operator created Forgejo repo + cloned. CC scaffolded 46 files, 1689 lines in initial commit 343216f: Vite 5 + React 18 + TS strict + TanStack Router (scoped @tanstack/*) + TanStack Query + shadcn (Radix) + Tailwind + openapi-fetch + vitest. - Day 2 · later · B
CI iteration cycle — 4 passes to fully-green
First pass: prettier reformat on 8 files. Second: react-refresh warnings on button.tsx + theme.tsx, plus typecheck cascade after routeTree.gen.ts regeneration (Sidebar narrowing, missing vite-env.d.ts). Third: file splits (button → button-variants, theme → use-theme). Fourth: clean. Installed Node 22 standalone in WSL to run prettier locally. - Day 2 · later · B
Deploy to dev (.175), then test + UAT
Operator stopped native nginx on each FE VM (Phase A playbook installed it; container needs port 80; hlvm can't sudo non-interactively). Container deploys clean across all three. - Day 2 · late · B
/api/docs returned 404 — backend fix
API Explorer page links work, but every link returns FastAPI's default 404 body. Cause: FastAPI defaults docs to /docs, but Traefik only routes /api/* to the backend. One-line backend fix (commit 2db1de2) — pass docs_url=/api/docs etc. to the FastAPI constructor. - Day 2 · close · B
ADR-0009 + tag sprint-1.5b-uat-green
Frontend stack ADR landed. Tag at aros-frontend f8e4a19. Sprint 1.5 closed across both phases. Sprint 2 plan at ai-context/09-sprint-2-plan.md — audit, events, tenant foundation per Build Plan §4.
Five silent failures, in the order they bit
Each of these was easy to miss because the failure mode is a successful HTTP response doing the wrong thing. Captured so the next operator doesn't repeat them.
Authentik's HTTP :9000 listener returns empty bodies behind a reverse proxy
First browser visit to http://192.168.1.241:9000/: page loads, returns 200, body is empty. curl -i confirms Content-Length: 0. Same behaviour through mvm Traefik with HTTP upstream.
Authentik’s HTTP listener checks Host / X-Forwarded-Host headers strictly and silently 200-empty-bodies any request that doesn’t match its trusted-hosts allow-list. The plain HTTP path is configured for direct LAN only and falls apart the moment a real proxy is in front. The HTTPS listener on :9443 doesn’t do this.
Switched the mvm Traefik route for authentik.ryanit.com to forward to https://192.168.1.241:9443 with serverTransport.insecureSkipVerify=true. Browser loads. Tracked debt: replace self-signed with a cert from the same Cloudflare resolver as mvm.
oauth2-proxy cookie secret needs raw 16/24/32 bytes — not base64 of 32 bytes
oauth2-proxy refused to start: cookie_secret must be 16, 24, or 32 bytes to create an AES cipher. The generated secret was openssl rand -base64 32 — 44 ASCII characters.
oauth2-proxy reads OAUTH2_PROXY_COOKIE_SECRET as raw bytes from the env string — not as a base64 / hex / anything-decoded value. The byte length is the character length of the env var. A 44-character string is 44 bytes, which is none of the AES key sizes.
Regenerated with openssl rand -hex 16 → 32 hex chars = 32 bytes (AES-256). Patched into Vault at secret/aros/authentik/oauth2-proxy:cookie-secret. Playbook header now spells out the encoding requirement.
Authentik OIDC tokens omit email_verified — oauth2-proxy rejects the callback
Browser authenticates with Entra (or password + MFA), redirects to /oauth2/callback?code=...&state=..., oauth2-proxy logs unverified email, returns 403, browser shows a generic “Forbidden” page. SSO loop never closes.
Authentik does not emit the email_verified claim in OIDC tokens by default — the claim is absent, not false. oauth2-proxy’s default policy treats an absent email_verified as unverified, refuses the callback. Authentik considers email verification a property of the Authentication Flow (a separate Email Verification stage you opt into), not a baseline OIDC claim.
Added OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL=true to the env file. SSO loop closes. The flag name is correctly scary — proper fix is to add an Email Verification stage to Authentik’s authentication flow. Tracked debt; flagged in ADR-0012.
Traefik errors middleware: only status and url substitute — surfaces as a "Found." page
After the cookie + email fixes, hitting /api/v1/me cookie-less should redirect to Authentik. Instead: blank page with the literal text “Found.” View-source confirms the body is just Found.
The Traefik errors middleware’s query field uses {status} and {url} as its only template variables. The plan said /oauth2/start?rd={scheme}://{host}{url} — but {scheme} and {host} pass through as literals. oauth2-proxy received rd={scheme}://{host}/api/v1/me, treated the literal-brace string as a redirect target, issued a 302 with a malformed Location header. Browsers show the body of a 302 they can’t follow — most servers default that body to Found. (the HTTP 302 reason phrase).
Shortened to /oauth2/start?rd={url} — {url} is path + query, which is everything oauth2-proxy needs. oauth2-proxy reconstructs the absolute URL from X-Forwarded-*. SSO loop closed on the next attempt.
/api/docs returned 404 after the first successful frontend deploy
Application shell loaded fine in the browser. Sidebar nav worked. Dashboard fetched /api/v1/me and got a 200 with the OIDC user. But every link from the API Explorer page — “Open Swagger UI”, “Raw OpenAPI JSON”, “ReDoc” — opened a tab returning {"detail": "Not Found"} (FastAPI’s default 404 body).
FastAPI’s defaults for those three surfaces are /docs, /openapi.json, /redoc — no /api prefix. aros-edge-1’s Traefik routes /api/* to the backend cluster and /* to the frontend. So the request for /api/docs reached the backend (good), but the backend had no route at /api/docs (bad) — FastAPI 404’d. The un-prefixed /docs would have hit the frontend’s catch-all router and returned the SPA’s index.html, also wrong.
One-line fix in aros-backend’s main.py: pass docs_url="/api/docs", redoc_url="/api/redoc", openapi_url="/api/openapi.json" to the FastAPI constructor (commit 2db1de2). Same auth gate applies — opening Swagger UI inherits the existing OIDC session via the oauth2-proxy cookie.
Everything that landed in Sprint 1.5
On top of Sprint 1's FastAPI + SQLAlchemy + Patroni + Woodpecker + mvm Traefik. The auth, edge, and frontend tiers are entirely new.
SSO loop, then the shell, then the docs
One curl chain through the SSO redirect → /api/v1/me. Then the shell + /api/docs after the post-deploy backend fix.
# Step 1: no cookie → forward-auth says 401 → errors-mw rewrites
$ curl -isS -H "Host: aros-dev.ryanit.com" \
http://192.168.1.241/api/v1/me | head -3
HTTP/1.1 302 Found
Location: /oauth2/start?rd=/api/v1/me
# Step 2: /oauth2/start → 302 to Authentik authorize
$ curl -isS -H "Host: aros-dev.ryanit.com" \
"http://192.168.1.241/oauth2/start?rd=/api/v1/me" | head -3
HTTP/1.1 302 Found
Location: https://authentik.ryanit.com/application/o/authorize/?...
# (Browser does the human dance: Entra OR password+MFA,
# Authentik redirects to /oauth2/callback, oauth2-proxy
# validates the code and sets the _oauth2_proxy cookie.)
# Step 3: with the post-login cookie, the protected path returns 200
$ curl -isS --cookie "_oauth2_proxy=<session>" \
-H "Host: aros-dev.ryanit.com" \
http://192.168.1.241/api/v1/me | head -8
HTTP/1.1 200 OK
Content-Type: application/json
X-Request-ID: 01HZK7...
X-Auth-Request-User: akadmin
X-Auth-Request-Email: akadmin@ryanit.com
X-Auth-Request-Groups: authentik Admins
X-Auth-Request-Preferred-Username: Aros Admin
{"id":"<uuid>","username":"akadmin","email":"akadmin@ryanit.com",
"display_name":"Aros Admin","groups":["authentik Admins"],
"created_at":"2026-05-25T...","last_seen_at":"2026-05-25T..."}# Shell: SPA + assets served via /
$ curl -isS https://aros-dev.ryanit.com/ | head -3
HTTP/2 200
content-type: text/html
cache-control: no-store, must-revalidate
# /api/docs after the FastAPI docs_url fix
$ curl -isS --cookie "_oauth2_proxy=<session>" \
https://aros-dev.ryanit.com/api/docs | head -2
HTTP/2 200
content-type: text/html; charset=utf-8
# /api/openapi.json reachable, schema visible
$ curl -sS --cookie "_oauth2_proxy=<session>" \
https://aros-dev.ryanit.com/api/openapi.json \
| jq -r '.info.title, .info.version'
Aros Backend
0.2.0
# /health/* bypasses the auth gate (priority 60 routers on aros-edge-1)
$ curl -sS https://aros-dev.ryanit.com/health/db | jq -r '.leader, .host'
192.168.1.231
aros-dev-2Sprint 2 — Audit, Events, Tenant Foundation
With identity + UI in place, Sprint 2 builds the compliance + isolation substrate every later sprint leans on: hash-chained audit log, per-env Redis event bus, multi-tenant tenant_id + RLS.
- Identity available for the actor field — Sprint 1.5a /api/v1/me
- Per-env Redis VMs provisioned + base-hardened — Sprint 1.5a §1
- Frontend in place for the tenant context provider — Sprint 1.5b
- Per-env Redis CONFIGURED — aros-cache.yml playbook still TODO (Sprint 2 pre-flight, blocks core.events)
- MinIO deployed for audit external seal — operator placement decision (Sprint 2 pre-flight)
- Sprint 2 plan written at ai-context/09-sprint-2-plan.md — four phases + pre-flight
What we chose not to fix tonight
Four of these are deliberate trade-offs called out in the ADRs (0009 / 0012 / 0013 / 0014). The rest are operational discoveries that need follow-up before Sprint 2 leans on them.
- auth·insecureSkipVerify in two places — mvm Traefik → authentik.ryanit.com:9443, and oauth2-proxy → Authentik OIDC issuer. Replace with a cert from the same Cloudflare resolver, or an internal CA. Per ADR-0012 / ADR-0013.
- auth·email_verified workaround — INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL=true on oauth2-proxy because Authentik doesn’t emit the claim. Proper fix is an Email Verification stage in the Authentik flow. Per ADR-0012.
- auth·Single Authentik instance, no HA. Lose aros-edge-1 and the estate loses auth. Break-glass: recovery admin at secret/aros/authentik/bootstrap. Per ADR-0012.
- auth·Authentik configured by hand — Terraform-Authentik-provider migration tracked. Per ADR-0012.
- edge·aros-edge-1 single VM + plain-HTTP hop from mvm. Trust-the-LAN compromises waiting on VLAN segmentation + internal TLS. Per ADR-0013.
- data·aros-cache-dev/test/uat-1 unconfigured — aros-cache.yml playbook + UFW per-env lockdown land in Sprint 2 pre-flight. Sprint 2’s core.events module is the first consumer.
- data·1s AOF loss window per Redis. core.events must be designed around at-most-1s-loss; not exactly-once. Per ADR-0014.
- ops·hlvm sudo policy. hlvm can run docker (group member) but lacks non-interactive sudo for everything else. Bit Sprint 1’s hello decommission AND Sprint 1.5b’s native-nginx removal. Candidate for a narrow NOPASSWD sudoers drop-in via Ansible base.
- ops·
X-Auth-Request-*vsX-Forwarded-*. The Sprint 1.5 plan said backend readsX-Forwarded-User; that’s only oauth2-proxy proxy-mode. In forward-auth mode (what we ship), the headers areX-Auth-Request-*. Plan text now stale; backend middleware reads the right family. - fe·aros-frontend operational items (per ADR-0009): no package-lock.json (CI on npm install); schema.d.ts hand-stubbed pending live regen; routeTree.gen.ts committed but auto-generated; no Playwright e2e; no bundle-size budget; native nginx on FE VMs stopped manually for the first deploy.