Stop configuring servers by hand. Provision with Terraform, configure with Ansible, keep secrets out of state, and rebuild any environment from an empty account in under an hour.
Every hand-configured server is a document nobody wrote. It works, and no one is quite sure why. Infrastructure as code replaces that document with something a machine can execute — and the real payoff is not automation, it is that rebuilding becomes routine instead of terrifying.
The confusion that wastes the most time is trying to do everything with one tool. The split is clean:
Terraform makes the box. Ansible fills it. Use Terraform to install packages and you get a fragile mess of remote-exec blocks; use Ansible to create servers and you lose the dependency graph that makes planning possible.
terraform {
required_version = ">= 1.10" # use_lockfile needs 1.10+
backend "s3" {
bucket = "tfstate-example"
key = "django-prod/terraform.tfstate"
region = "eu-central-1"
use_lockfile = true
}
}
variable "environment" { type = string }
variable "server_type" {
type = string
default = "cx32"
}
resource "provider_server" "app" {
name = "django-${var.environment}"
server_type = var.server_type
image = "ubuntu-24.04"
ssh_keys = [provider_ssh_key.deploy.id]
labels = { role = "app", env = var.environment }
}
resource "provider_firewall" "app" {
name = "django-${var.environment}"
rule {
direction = "in"
protocol = "tcp"
port = "80"
source_ips = ["0.0.0.0/0", "::/0"]
}
rule {
direction = "in"
protocol = "tcp"
port = "443"
source_ips = ["0.0.0.0/0", "::/0"]
}
rule {
direction = "in"
protocol = "tcp"
port = "22"
source_ips = var.admin_cidrs
}
}
Three habits that separate infrastructure code that lasts from a script that rots:
destroy, or replace on anything holding data. terraform plan is not a formality; it is the whole safety mechanism.resource "provider_volume" "media" {
name = "django-${var.environment}-media"
size = 100
lifecycle {
prevent_destroy = true
}
}
resource "provider_server" "app" {
# ...
lifecycle {
ignore_changes = [image, user_data]
}
}
# Renamed or moved into a module? Tell Terraform, or it plans destroy + create.
moved {
from = provider_server.app
to = module.production.provider_server.app
}
Each of these has a sharp edge worth knowing:
prevent_destroy only protects while the block exists. Delete the resource from the code and the protection goes with it. Where the provider offers its own delete protection flag on volumes or databases, enable that too — it also guards against a click in the web console.ignore_changes is a deliberate blind spot. It stops a new base image or edited cloud-init from silently proposing a rebuild, but it also hides those attributes from drift detection. When you do want the new image, say so explicitly with terraform apply -replace=provider_server.app.moved blocks make refactoring safe. Wrapping existing resources in a module changes their addresses. Without a moved block the plan shows a destroy and a create; with it, the plan shows a move and zero changes. Leave the block in place until every environment has applied it.module "staging" {
source = "./modules/django-stack"
environment = "staging"
server_type = "cx22"
domain = "staging.example.com"
}
module "production" {
source = "./modules/django-stack"
environment = "production"
server_type = "cx42"
domain = "example.com"
}
One module, two calls. The moment you copy the directory and edit it, staging stops resembling production, and staging that does not resemble production tests nothing.
This trips up almost everyone once: Terraform state contains every value it manages, in plain text. A database password passed as a variable is in the state file forever, including in its history.
Two workable approaches:
And regardless: encrypt the state bucket, restrict who can read it, and never commit terraform.tfstate — put it in .gitignore on day one, because removing it later means rotating everything it contained.
Marking a variable or output sensitive = true redacts it from plan and apply output. That is all it does. The value is still written to state and to saved plan files in full. Recent Terraform releases narrow the gap: 1.10 introduced ephemeral variables, outputs and resources, which are never persisted to plan or state, and 1.11 added write-only arguments that a provider accepts during apply without storing. Write-only arguments conventionally end in _wo and are paired with a version argument that you bump to trigger a rotation:
variable "db_password" {
type = string
ephemeral = true
}
resource "provider_database" "main" {
# ...
password_wo = var.db_password
password_wo_version = 1 # increment to push a new password
}
Argument names and support vary by provider, so check its documentation before relying on this. Where a resource has no write-only variant, fall back to the two approaches above.
Ansible's value is idempotence: running the playbook on a fresh server and on a five-month-old server both end with the same result.
- name: Configure Django application server
hosts: app
become: true
roles:
- common # users, ssh hardening, unattended-upgrades, timezone
- docker
- django_app
- nginx
- monitoring
# roles/django_app/tasks/main.yml
- name: Application directory
ansible.builtin.file:
path: /opt/{{ app_name }}
state: directory
owner: "{{ app_user }}"
mode: "0750"
- name: Environment file
ansible.builtin.template:
src: env.j2
dest: /opt/{{ app_name }}/.env
owner: "{{ app_user }}"
mode: "0600"
no_log: true
notify: restart app
- name: Compose stack
community.docker.docker_compose_v2:
project_src: /opt/{{ app_name }}
state: present
pull: always
no_log: true on anything touching secrets is not optional — without it your CI log prints the rendered environment file, and CI logs are widely readable.
# roles/common/tasks/ssh.yml
- name: Harden sshd
ansible.builtin.template:
src: sshd_config.j2
dest: /etc/ssh/sshd_config
owner: root
mode: "0600"
validate: /usr/sbin/sshd -t -f %s
notify: reload sshd
# roles/common/handlers/main.yml
- name: reload sshd
ansible.builtin.service:
name: ssh
state: reloaded
# roles/django_app/handlers/main.yml
- name: restart app
community.docker.docker_compose_v2:
project_src: /opt/{{ app_name }}
state: restarted
validate renders the file to a temporary path, runs the command against it, and only moves it into place if the command succeeds. For sshd_config that is the difference between a rejected change and a server nobody can log into. Handlers run once, at the end of the play, however many tasks notified them — ten template changes produce one restart.
The catch: if a later task fails, pending handlers do not run, and the next run sees no change and never fires them either. The new config sits on disk, unapplied. Run with --force-handlers in CI, or place - ansible.builtin.meta: flush_handlers at points where a restart must happen before continuing.
ansible-vault encrypt_string --name DJANGO_SECRET_KEY 'the-actual-value'
The encrypted blob goes in group_vars/production/vault.yml and is committed safely. The vault password lives in your password manager and in CI as a secret. You now have one secret to protect instead of thirty.
jobs:
plan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
- run: terraform init && terraform plan -out=tfplan
- uses: actions/upload-artifact@v4
with: { name: tfplan, path: tfplan }
apply:
needs: plan
if: github.ref == 'refs/heads/main'
environment: production # requires a human approval
steps:
- run: terraform apply tfplan # the exact plan that was reviewed
Two properties matter. Apply the saved plan, not a fresh one — otherwise you approve one thing and execute another. And gate production behind a human. Infrastructure automation should remove typing, not remove judgment.
The sketch above omits the parts that make it actually run. The apply job starts on a fresh runner with nothing checked out and no providers installed:
name: infrastructure
on:
pull_request:
paths: ["terraform/**"]
push:
branches: [main]
paths: ["terraform/**"]
permissions:
contents: read
id-token: write # short-lived cloud credentials via OIDC
concurrency:
group: terraform-production
cancel-in-progress: false
defaults:
run:
working-directory: terraform
jobs:
plan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: "1.x.y" # pin the exact version your team uses
# authenticate to your cloud here (OIDC, read-only plan role)
- run: terraform init -input=false
- run: terraform plan -input=false -out=tfplan
- uses: actions/upload-artifact@v4
with:
name: tfplan
path: terraform/tfplan
retention-days: 5
apply:
needs: plan
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
with:
terraform_version: "1.x.y"
# authenticate to your cloud here (OIDC, apply role)
- run: terraform init -input=false
- uses: actions/download-artifact@v4
with:
name: tfplan
path: terraform
- run: terraform apply -input=false tfplan
Details that each prevent a specific incident:
concurrency serialises runs. Two merges in quick succession would otherwise race for the lock, and the loser fails halfway through a pipeline.id-token: write permission lets the job exchange a signed token for short-lived cloud credentials, so there is no long-lived access key sitting in CI secrets waiting to leak.Ansible fits into the same pipeline as a later job, after apply: write the vault password from a CI secret to a file with mode 0600, run the playbook with --vault-id, and delete the file. Be careful with --diff in CI — it prints file contents, which is exactly what no_log exists to prevent.
Someone SSHes in at 2am, edits nginx, and fixes an outage. Nobody writes it down. Three months later a rebuild produces a server that is subtly different, and the bug comes back.
Catch it: run terraform plan -detailed-exitcode and ansible-playbook --check --diff on a schedule. Exit code 2 means reality no longer matches the code. Alert on it, then either bring the change into code or revert it — but never leave it undecided.
The cultural rule that makes this work: emergency SSH fixes are allowed, and must be followed by a pull request the same week. Forbidding them entirely just means they stop being mentioned.
ansible-playbook --check --diff is a strong signal, not a proof. In check mode, command and shell tasks are skipped unless marked check_mode: false, tasks that depend on their registered output may fail or report nonsense, and handlers do not fire. More fundamentally, Ansible only inspects what it manages:
| Drift | Caught by |
|---|---|
| Firewall rule edited in the provider console | terraform plan |
| Hand-edited nginx site managed by a template | ansible-playbook --check --diff |
Extra file dropped into /etc/nginx/sites-enabled/ | Neither, unless the role removes unmanaged files |
Package installed by hand, new file in /etc/cron.d/ | Neither by default |
| Server created in the console and never imported | Neither — audit provider resources against labels |
The fix for the middle rows is to manage directories, not just files: have the nginx role list sites-enabled and remove anything it did not put there. That also makes an unexpected file — a common sign of tampering — impossible to miss in the next check-mode diff. For the last row, a periodic script that lists every resource in the account and flags anything without your env label catches the console-created server before it becomes a forgotten, unpatched liability.
Infrastructure code that has never been run from scratch is a hypothesis. Test it:
prevent_destroy on it and mean it.# deploy.yml
- name: Migrate once, before any host runs new code
hosts: app[0]
become: true
tasks:
- name: Pull the new image
community.docker.docker_compose_v2_pull:
project_src: /opt/{{ app_name }}
- name: Apply migrations
ansible.builtin.command:
cmd: docker compose run --rm web python manage.py migrate --noinput
chdir: /opt/{{ app_name }}
register: migrate
changed_when: "'No migrations to apply' not in migrate.stdout"
- name: Roll the new release across hosts
hosts: app
become: true
serial: 1
max_fail_percentage: 0
tasks:
- name: Recreate containers with the new image
community.docker.docker_compose_v2:
project_src: /opt/{{ app_name }}
state: present
pull: always
- name: Wait until this host is healthy
ansible.builtin.uri:
url: "http://127.0.0.1:8000/healthz/"
status_code: 200
register: health
until: health.status == 200
retries: 20
delay: 3
Two plays, not one, and the reason is a classic trap: run_once combined with serial runs once per batch, so with serial: 1 it would migrate on every host after all. Putting migrations in their own play targeting a single host makes "exactly once" structural rather than a flag you hope behaves.
The rolling play stops at the first failed health check (max_fail_percentage: 0), leaving the rest of the fleet on the previous release. If you run a load balancer, add tasks to drain the host before the restart and re-enable it after the health check.
Rolling deploys impose one constraint on Django code: for a few minutes, old code runs against the new schema. Migrations must therefore be backwards-compatible — add a nullable column in one release and start using it; make it required or drop the old column in a later release. A migration that renames a column in one step breaks every host still running the old code.
| Symptom | Likely cause | What to do |
|---|---|---|
Error acquiring the state lock | A cancelled CI job or crashed laptop left the lock behind | Confirm no apply is actually running, then terraform force-unlock LOCK_ID with the ID from the error |
| Plan replaces the server after a provider upgrade | An attribute's handling changed; the plan marks it "forces replacement" | Revert the lock file change, read the provider changelog, then decide: ignore_changes, a code change, or a planned rebuild |
| Plan shows destroy and create after a refactor | Resource address changed | Add a moved block; never apply a plan like this on data-bearing resources |
| Saved plan is stale | State changed between plan and apply | Re-run plan and review again |
| SSH reports the remote host identification has changed | A rebuilt server reused the old IP address | Verify the new host key fingerprint via the provider console, then ssh-keygen -R 203.0.113.10; never disable host key checking globally |
| Vault decryption fails | Wrong vault ID or password file for this environment | Check the --vault-id label matches the one used to encrypt |
| Every Ansible run reports changes | Non-idempotent command or shell tasks | Add changed_when, creates, or switch to a proper module |
Do not rewrite your production estate this month. Do this instead:
The end state is not "we automated the servers". It is that a failed machine is an inconvenience rather than an incident, because the thing that made it is checked into git.