deploy-to-hetzner

Use when putting a containerized service on Hetzner Cloud with OpenTofu — a VM per environment, Cloud Firewall, private network, hardened cloud-init, Docker Compose with Traefik TLS, deploys by image

SKILL.md

GitHub

Deploy to Hetzner

One VM per environment, hardened by cloud-init, serving a Compose stack behind Traefik. The image, the compose files and the deploy script all ship through the same reviewed pipeline. Contract: environments.md, deploy-script.md.

1. Audit current state (change nothing)

cat .claude/stack-profile.md 2>/dev/null        # hosting, iac, database.host, ci
ls infra/modules/hetzner-host infra/modules/host-bootstrap deploy/ compose*.y*ml 2>/dev/null
ls deploy/stack deploy/host deploy/envs 2>/dev/null
hcloud context list 2>/dev/null                 # context names only, never token values
[[ -n "${HCLOUD_TOKEN:-}" ]] && echo "HCLOUD_TOKEN set" || echo "HCLOUD_TOKEN unset"
tofu version 2>&1 | head -1

Read from the profile: hosting (must list hetzner), iac (opentofu), database.host, ci. If there is no profile, detect from the infra/ and deploy/ trees above. Never ask what the profile already answers.

2. Decide

  • Profile says hosting is vercel or ionos (and not hetzner): stop. Point to deploy-to-vercel or deploy-to-ionos.
  • No image yet (containerize-service has not produced a digest): run containerize-service first; this skill deploys a digest, not a build.
  • No infra/modules/hetzner-host and no deploy/: full (steps 3 to 7).
  • Modules exist but a piece is missing (no deploy/stack/bin/backup.sh, no deploy/envs): delta, add only the missing piece.
  • Modules, deploy layout, GitHub environments, DNS and backups all present and verified: "already in place", run step 7, stop.

3. Detect the track

  • Single VM Compose (default): one VM per environment, one Compose project on it. Use this unless a trigger below is met.
  • Graduate to k3s: do not build k3s here. The triggers (more than one VM per environment behind a load balancer, rolling zero-downtime needs, several teams, node-level failure tolerance) and the k3s target (v1.37.1+k3s1) are in hetzner-patterns.md. This skill stops at the Compose host.

4. Install only what is missing

brew install hcloud        # 1.70.1; or mise use hcloud@1.70.1

OpenTofu 1.13.1 through set-up-opentofu — run it first, or in the same change. It writes the infra/ roots, encrypted remote state, TF_ENCRYPTION, the CI gate and drift detection. This skill adds the Hetzner modules on top.

5. Generate the seams

  1. Modules infra/modules/hetzner-host and infra/modules/host-bootstrap, plus one environment root (infra/envs/<env>/) that instantiates the module. Every file, verbatim, in hetzner-patterns.md. The module holds the environment validation, the firewall, the private network, the primary IP and the volume.
  2. deploy/ layout exactly as deploy-script.md shows: deploy/deploy.sh, deploy/host/run.sh, deploy/stack/bin/backup.sh, deploy/stack/bin/restore-drill.sh, deploy/stack/compose.yaml, deploy/stack/traefik/, deploy/envs/<env>/config.env. Link deploy.sh and run.sh to deploy-script.md; do not restate their bodies. The stack files are in compose-patterns.md.
  3. deploy/stack/compose.yaml and the two Traefik files, verbatim from compose-patterns.md.
  4. Backups: deploy/stack/bin/backup.sh and deploy/stack/bin/restore-drill.sh, verbatim from compose-patterns.md. The systemd timer comes from cloud-init.
  5. Host config deploy.conf and the deploy key's forced command: written by cloud-init, so nothing to hand-edit on the host.

6. Wire

  • GitHub environments (dev, stg, prd). Secrets: DEPLOY_SSH_KEY, HCLOUD_TOKEN, state credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) and TF_ENCRYPTION — the last three per set-up-opentofu. Variables: DEPLOY_HOST, DEPLOY_HOST_KEY, APP_URL, HEALTH_URL. DEPLOY_HOST_KEY is the pinned host key; no trust on first use.
  • No OIDC for Hetzner (none found 2026-10-09): DEPLOY_SSH_KEY and HCLOUD_TOKEN are long-lived. Scope the key to one environment's forced command and the token to one project; rotate both on a schedule. Say this in the README.
  • DNS: an A record and an AAAA record to the primary IP (tofu output ipv4 / ipv6). The IP outlives the server, so DNS survives a rebuild.
  • Host secrets (Postgres password, any app secret): the integration point is the optional hook HOOK_PRE_UP in /etc/deploy/deploy.conf (add HOOK_PRE_UP=/opt/deploy/hooks/pre-up.sh), which run.sh calls before compose up. The hook is written per manage-secrets (sops+age, one age key per host) and must create /opt/app/secrets/postgres_password (mode 0400, on disk: the file is bind-mounted, so it has to survive the unattended 04:30 reboot). Plaintext is never in the repo and never in the bundle. The hook script is in manage-secrets (run locally against sops + age; not yet on a live host).
  • Logs and traces: deploy-otel-collector adds the collector service to the same Compose project.
  • Approval: prd deploy and apply behind the environment's required reviewers (environments.md); dev on merge to main.

7. Verify

cd infra/envs/dev
tofu fmt -check -recursive ../.. ; tofu init -lockfile=readonly ; tofu validate   # valid; init succeeded
tofu plan -input=false                                                            # exit 0/2, the planned diff
tofu apply -input=false                                                           # creates server, firewall, network, IP, volume
ssh admin@<IP> 'cloud-init status --wait'                                          # "status: done"
ssh -o BatchMode=yes deploy@<IP> 'id'                                             # refused: forced command only accepts "<env> <image>"
deploy/deploy.sh dev ghcr.io/acme/shop@sha256:<digest>                             # "deployed dev ghcr.io/acme/shop@sha256:…"
curl -fsS https://<host>/healthz                                                    # "ok"
ssh admin@<IP> 'docker compose --project-directory /opt/app -f /opt/app/compose.yaml ps'   # every service: healthy
ssh admin@<IP> 'sudo bash /opt/app/bin/backup.sh'                                  # exit 0; restic snapshots lists one
ssh admin@<IP> 'sudo bash /opt/app/bin/restore-drill.sh'                           # "restore drill ok" (needs PG_IMAGE; see compose-patterns.md)

What was exercised during research, and what was not, is split in hetzner-patterns.md: the compose stack, the deploy script, the backup and restore drill, tofu validate/test of the modules and the cloud-init YAML parse were run locally; no real Hetzner project existed, so server creation, firewall effect, volume-mount timing, cloud-init on a live host and ACME issuance were not.

References