First IaC-provisioned VM (home-services, VM 104) — declared in Terraform, cloned from template, cloud-init applied, network up. End of an era of clicking through wizards.
v1.14.8 + 2.20.4 + v1.21.4 on iaccn01 (Ubuntu 24.04).
Bare Ubuntu — TF/Ansible run on host, Vault runs in Docker. No Portainer here (different VM pattern: workstation, not service host).
HA + Portainer + Mosquitto, host networking for LAN device discovery.
Why this session was the inflection
Every new service was costing the same amount of toil. The cost wasn't tooling; it was the absence of any declarative layer underneath.
The trigger was the realisation that the n-th Ubuntu + Docker + Portainer VM build was about to start. Every Nextcloud, every Immich, every per-user instance, every dev sandbox — the same hour-plus of wizard-clicking, package-installing, SSH-key-pasting, hardening-applying. The pattern was identical; the execution was manual every time.
The fix wasn’t “use Ansible to install packages faster.” The fix was to make the entire VM lifecycle declarative: the existence of the VM (Terraform), its configuration (Ansible), and the credentials they both need (Vault) all live in code, get committed, and run from one control node.
This session built that control node and used it to deploy the first real workload — proving the loop end-to-end before scaling the pattern.
The architectural calls that locked in early
- control·Control node is bare Ubuntu, not a Docker host. Terraform / Ansible / Vault CLI live on the host directly. Workstation pattern, not service-host pattern. No Portainer here — it would add a web UI that gets used zero times.
- provider·bpg/proxmox provider, not telmate/proxmox. Telmate had bugs with API token permissions that ate hours; bpg is actively maintained and accepts the same patterns more cleanly.
- secrets·Vault from day one, not “we’ll add it later”. File-backed storage, Shamir seal (5 shares, 3 threshold), KVv2 for the Proxmox API token. Terraform pulls credentials at plan-time; nothing gets committed in clear.
- template·Ubuntu 24.04 cloud-init template (VM 9000) on pve2. Cloud image + qm template — clones in seconds, cloud-init injects hostname / IP / SSH keys on first boot.
- storage·NVMe is for workloads that earn it, not the default. Operator correction: default to Z1-1 (HDD ZFS), reach for vmfast (NVMe) only when something justifies it.
- scope·Ship the first real VM, not infinite preparation. The point wasn’t to perfect the pipeline; it was to prove the loop end-to-end and find what was actually broken before generalising.
The four hours that weren't in the plan
Terraform user couldn't get a password set
Created terraform@pam user in Proxmox; setting a password returned change password failed: user 'terraform' does not exist (500).
pam realm users in Proxmox need to exist as actual Linux users on the host — they don’t unless you create them on the OS too.
Skipped the password entirely. API tokens authenticate independently of the user’s OS-level password. Created the API token (terraform@pam!iac), assigned Administrator at /, stored in Vault at secret/proxmox. Permissions tightening is tracked debt.
Cross-datastore disk migration: 15 minutes per clone
Template lives on vmfast (NVMe ZFS pool on pve2), new VMs targeting Z1-1 (HDD). Clone time: ~15 minutes per VM, the bulk of which is the cross-datastore disk copy.
Proxmox can’t live-migrate disks between ZFS and LVM-thin storage types in a single step. Even within ZFS-to-ZFS, full clones across pools copy block by block at storage speed.
Tracked as known cost; not yet a blocker for the volumes involved. Sprint-level fix would be to keep multiple templates (one per target datastore) or use linked clones where appropriate.
bpg provider rejected disk size strings
disk { size = "32G" } failed to plan — provider expected an integer.
bpg/proxmox provider takes disk size as integer GB, not a human-readable string.
disk { size = 32 }. Cosmetic but undocumented for newcomers from the telmate provider.
Network interface name confusion
Initial config used eth0 for the network device label; bpg silently produced a VM without expected networking config.
bpg references network devices by Proxmox’s internal name (net0), not the guest-OS name (eth0).
Switched to net0. Operator correction caught it quickly.
The 32-second moment
$ cd ~/iac/terraform
$ terraform apply -var-file=home-services.tfvars
proxmox_virtual_environment_vm.vm: Creating...
proxmox_virtual_environment_vm.vm: Creation complete after 32s [id=104]
$ ssh hlvm@192.168.1.103
Welcome to Ubuntu 24.04.4 LTS
$ docker --version
Docker version 29.4.0, build ...
$ docker ps
CONTAINER ID IMAGE NAMES
abc123def456 ghcr.io/home-assistant/... homeassistant
def456abc789 portainer/portainer-ce:... portainer
789abc123def eclipse-mosquitto:2 mosquittoEverything downstream traces back to here
- →Bin Collection automation (Personal) — first real workload running on the new IaC platform.
- →iac.ryanit.com self-service portal — React frontend in front of the same Terraform / Ansible pipeline. One-click VM provisioning.
- →Domain Tracker on infraservices01 (VM 111) — first service deployed through the portal, monitoring RDAP domain expiry and Cloudflare DNS drift.
- →Aros Sprint 0 — 13 VMs in one overnight session, dev / test / UAT clusters distributed across pve1/2/3, HA Postgres on dedicated VMs. Direct descendant of the patterns proven here.
- →This site itself — projects-recap, on hub-1 (VM 240), provisioned via the same Terraform module pattern.