Immutable Infrastructure: Golden Images with Packer, Rolling Replacement, and Eliminating Configuration Drift

A server that has been patched by hand for two years is a unique object: nobody can rebuild it, and nobody can say exactly what is on it. Immutable infrastructure removes that problem by never changing a running machine. You bake a versioned image, boot replacements from it, shift traffic with health checks, and delete the old instances. Every change, from a kernel patch to a config tweak, follows the same path.

Mutable vs immutable

In a mutable fleet, configuration management or an operator changes servers in place. In an immutable fleet, the only write to a running root disk is what the application legitimately produces, and that data lives somewhere else anyway.

The cost is real. Build times lengthen the feedback loop, every instance must tolerate being deleted, and a single stateful host with a local database does not fit without separate treatment. The model pays off once you have more than a handful of servers or an audit requirement to prove what runs in production.

A Packer template

This example uses the QEMU builder so the template stays cloud-neutral; it produces a qcow2 disk that can be imported anywhere. On a cloud, replace the source block with that cloud's builder. The build block and the scripts stay the same. The project layout is:

image/
  web.pkr.hcl
  seed/user-data
  seed/meta-data
  scripts/10-packages.sh
  scripts/20-harden.sh
  scripts/30-app.sh
  scripts/90-cleanup.sh
  files/app.tar.gz
  tests/goss.yaml

packer {
  required_version = ">= 1.11.0"
  required_plugins {
    qemu = {
      version = ">= 1.1.0, < 2.0.0"
      source  = "github.com/hashicorp/qemu"
    }
  }
}

variable "base_image_url"      { type = string }
variable "base_image_checksum" { type = string }
variable "app_version"         { type = string }
variable "git_sha"             { type = string }
variable "ssh_key_file"        { type = string }

locals {
  stamp         = formatdate("YYYYMMDD-hhmm", timestamp())
  image_version = "${local.stamp}-${substr(var.git_sha, 0, 7)}"
  image_name    = "myproject-web-${local.image_version}"
}

source "qemu" "web" {
  iso_url              = var.base_image_url
  iso_checksum         = var.base_image_checksum
  disk_image           = true
  format               = "qcow2"
  disk_size            = "12G"
  accelerator          = "kvm"
  headless             = true
  memory               = 2048
  cpus                 = 2
  output_directory     = "output/${local.image_name}"
  vm_name              = "${local.image_name}.qcow2"
  cd_files             = ["seed/user-data", "seed/meta-data"]
  cd_label             = "cidata"
  ssh_username         = "packer"
  ssh_private_key_file = var.ssh_key_file
  ssh_timeout          = "10m"
  shutdown_command     = "sudo sh -c 'rm -f /etc/sudoers.d/90-cloud-init-users; userdel -rf packer; shutdown -P now'"
}

build {
  sources = ["source.qemu.web"]

  provisioner "shell" {
    inline           = ["cloud-init status --wait"]
    valid_exit_codes = [0, 2]
  }

  provisioner "file" {
    source      = "files/app.tar.gz"
    destination = "/tmp/app.tar.gz"
  }

  provisioner "shell" {
    execute_command  = "{{ .Vars }} sudo -E bash -eu '{{ .Path }}'"
    environment_vars = [
      "APP_VERSION=${var.app_version}",
      "IMAGE_VERSION=${local.image_version}",
    ]
    scripts = [
      "scripts/10-packages.sh",
      "scripts/20-harden.sh",
      "scripts/30-app.sh",
      "scripts/90-cleanup.sh",
    ]
  }

  post-processor "manifest" {
    output     = "manifest.json"
    strip_path = true
    custom_data = {
      image_version = local.image_version
      app_version   = var.app_version
      git_sha       = var.git_sha
    }
  }
}

Run packer init. The seed/user-data file creates a temporary packer user with a public key generated for this build. The shutdown command removes that user, so no build account or authorized key ships in the image. The cd_files option needs xorriso or mkisofs on the build host. Exit code 2 from cloud-init status means a recoverable error during the seed stage, which is why the first provisioner accepts it.

Point base_image_url at an internal mirror and pin its checksum. Building from a moving upstream URL means two builds of the same commit can differ, and an upstream outage stops your security patching.

Provisioning and hardening

Keep scripts small and ordered. The hardening script should change only what you can test afterwards.

#!/usr/bin/env bash
# scripts/20-harden.sh
set -euo pipefail

cat > /etc/ssh/sshd_config.d/10-hardening.conf <<'EOF'
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
X11Forwarding no
AllowTcpForwarding no
MaxAuthTries 3
EOF
sshd -t

cat > /etc/sysctl.d/60-hardening.conf <<'EOF'
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.tcp_syncookies = 1
kernel.kptr_restrict = 2
kernel.dmesg_restrict = 1
EOF

cat > /etc/nftables.conf <<'EOF'
#!/usr/sbin/nft -f
flush ruleset
table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;
    iif lo accept
    ct state established,related accept
    ip saddr 10.0.0.0/24 tcp dport 22 accept
    ip saddr 10.0.1.0/24 tcp dport 8000 accept
  }
}
EOF
systemctl enable nftables

# Patches arrive through new images, not on running hosts.
apt-get purge -y unattended-upgrades
systemctl disable apt-daily.timer apt-daily-upgrade.timer

The purge is a deliberate trade-off. Self-patching hosts drift from their image and from each other, and a package update can restart a service at an unplanned moment. In exchange, you accept that a host is only as patched as its image, so the rebuild cadence becomes a security control. If you cannot rebuild reliably on a schedule, keep automatic updates until you can.

The 10-packages.sh script, not shown, installs the application's OS dependencies plus nftables, aide, and the Vault binary with its unit file. The install script unpacks the release, writes a version marker, records a file integrity baseline, and deliberately leaves the application unit disabled:

#!/usr/bin/env bash
# scripts/30-app.sh
set -euo pipefail

useradd --system --home /srv/app --shell /usr/sbin/nologin myproject
mkdir -p /srv/app /etc/myproject
tar -xzf /tmp/app.tar.gz -C /srv/app
chown -R root:root /srv/app
install -m 0644 /srv/app/deploy/myproject.service /etc/systemd/system/
echo "d /run/myproject 0750 root myproject -" > /etc/tmpfiles.d/myproject.conf

cat > /etc/myproject/image-release <<EOF
IMAGE_VERSION=${IMAGE_VERSION}
APP_VERSION=${APP_VERSION}
EOF

aideinit --yes --force
mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

Cleanup makes the image generic. Without it, every clone shares a machine ID and SSH host keys, which breaks log correlation and hides impersonation:

#!/usr/bin/env bash
# scripts/90-cleanup.sh
set -euo pipefail

rm -f /tmp/app.tar.gz
apt-get clean
rm -rf /var/lib/apt/lists/*
rm -f /etc/ssh/ssh_host_*
journalctl --rotate
journalctl --vacuum-time=1s
cloud-init clean --logs --machine-id
fstrim -av || true

Image versioning and promotion

An image version is immutable and unique. Combine a timestamp with the commit, as the template does, and record the base image checksum, app version and commit in the manifest. Never reuse a name.

Promotion means moving a pointer, not rebuilding. A rebuild for production produces a different artifact from the one you tested, because package mirrors move. Keep one small record per channel:

{
  "channel": "prod",
  "image_id": "myproject-web-20261105-1412-a1b2c3d",
  "promoted_from": "staging",
  "promoted_by": "ci",
  "previous_image_id": "myproject-web-20261029-0900-f6e5d4c"
}

Store it in a versioned object store or parameter service. The pipeline is then simple: dev receives every build, staging receives builds that passed image tests, and prod receives builds that survived staging plus an approval. Deployment reads the channel and never accepts a free-form image ID. Retain every image referenced by a channel history entry for your rollback window, and garbage-collect only after checking that no launch configuration still references it.

Cloud-init for per-instance data

The image should be identical in every environment. What differs goes in user-data, which should be short and declarative:

#cloud-config
users: []
ssh_pwauth: false
disable_root: true
write_files:
  - path: /etc/myproject/instance.env
    permissions: "0644"
    content: |
      ENVIRONMENT=prod
      DATABASE_HOST=db.internal.example.com
      CACHE_URL=redis://cache.internal.example.com:6379/0
      LOG_TARGET=logs.internal.example.com:6514
runcmd:
  - [systemctl, enable, --now, --no-block, vault-agent.service]
  - [systemctl, enable, --now, --no-block, myproject.service]

If user-data grows into a long script that installs packages, you have rebuilt mutable configuration at boot time, with extra latency and a new dependency on package mirrors at scale-out time. Treat any package installation in user-data as a bug.

The units are enabled from runcmd for a specific reason. On common distributions, cloud-final.service orders itself after multi-user.target, so a service wanted by that target and ordered after cloud-final creates a dependency cycle that systemd breaks by dropping a job. Enabling from runcmd avoids that, ensures the application cannot start before its configuration exists, and still gives correct behaviour on reboot because the unit is then enabled.

The --no-block flag matters too. Without it, runcmd waits for the service, and a slow start stalls cloud-init completion.

Secrets at boot

Nothing secret belongs in the image: images are copied between accounts, stored for months and readable by anyone with launch rights. User-data is also unsuitable, because any process on the host can usually read it from the metadata service and it is often visible through the management API.

The instance should prove its identity and fetch secrets into memory-backed storage. This Vault Agent configuration uses a signed JWT, which a small early unit obtains from the platform's identity service and writes to /run/identity/token. Clouds with a native Vault auth method can use that method instead.

vault {
  address = "https://vault.example.com:8200"
}

auto_auth {
  method "jwt" {
    mount_path = "auth/jwt"
    config = {
      path = "/run/identity/token"
      role = "myproject-web-prod"
    }
  }
}

template {
  destination = "/run/myproject/secrets.env"
  user        = "root"
  group       = "myproject"
  perms       = "0640"
  command     = "systemctl try-restart myproject.service"
  contents    = <<EOT
{{ with secret "secret/data/myproject/prod" }}
DATABASE_PASSWORD={{ .Data.data.database_password }}
SESSION_KEY={{ .Data.data.session_key }}
{{ end }}
EOT
}

The agent must run as root for that systemctl command to work. Starting it does not mean the file exists yet, so the application unit waits for it:

[Unit]
Description=myproject web
Wants=network-online.target
After=network-online.target vault-agent.service
Requires=vault-agent.service

[Service]
User=myproject
EnvironmentFile=/etc/myproject/instance.env
EnvironmentFile=-/run/myproject/secrets.env
ExecStartPre=/usr/bin/timeout 60 sh -c 'until [ -s /run/myproject/secrets.env ]; do sleep 1; done'
ExecStart=/srv/app/.venv/bin/gunicorn -c /srv/app/deploy/gunicorn.py myproject.wsgi
Restart=on-failure
RestartSec=5
ProtectSystem=strict
PrivateTmp=true
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

The - prefix is needed because systemd also loads environment files for ExecStartPre, when the file may be absent. Failure mode to plan for: if Vault is down, new instances cannot start. Running instances keep working, so a rolling deploy stops at its first health failure, which is the correct behaviour. Autoscaling, however, stalls during the outage, so treat the secret store as a dependency with the same availability target as the service itself.

Replacement behind a load balancer

Health checks

Replacement is only as safe as the check that gates it. Separate two questions. Liveness asks whether the process should be restarted. Readiness asks whether this instance should receive traffic. A readiness check that queries the database means a brief database failure removes every instance at once, turning a partial outage into a total one. Keep the load balancer check focused on local readiness, and run a deeper smoke test once, during rollout, that exercises the database and cache.

global
  stats socket /run/haproxy/admin.sock mode 660 level admin

backend web_backend
  option httpchk
  http-check send meth GET uri /healthz ver HTTP/1.1 hdr Host app.example.com
  http-check expect status 200
  default-server inter 3s fall 2 rise 3 slowstart 30s
  server web1 10.0.1.21:8000 check
  server web2 10.0.1.22:8000 check
  server web3 10.0.1.23:8000 check

Rolling replacement

The script below shows the logic that any managed tool implements. The two create_instance and destroy_instance functions wrap your platform's CLI.

#!/usr/bin/env bash
# roll.sh IMAGE_ID
set -euo pipefail

IMAGE_ID="$1"
BACKEND="web_backend"
SOCK="/run/haproxy/admin.sock"
SERVERS=(web1 web2 web3)

hap() { echo "$*" | socat stdio "$SOCK"; }

sessions() {
  hap "show stat" | awk -F, -v b="$BACKEND" -v s="$1" \
    '$1==b && $2==s {print $5}'
}

field() {
  hap "show servers state $BACKEND" | awk -v s="$1" -v f="$2" \
    '$4==s {print $f}'
}

healthy() {
  curl -fsS --max-time 2 "http://$1:8000/healthz" >/dev/null
}

for name in "${SERVERS[@]}"; do
  new_ip=$(create_instance "$IMAGE_ID")

  for _ in $(seq 1 60); do
    healthy "$new_ip" && break
    sleep 5
  done
  if ! healthy "$new_ip"; then
    echo "instance for $name never became healthy, stopping" >&2
    destroy_instance "$new_ip"
    exit 1
  fi

  old_ip=$(field "$name" 5)
  hap "set server $BACKEND/$name state drain"
  for _ in $(seq 1 60); do
    [ "$(sessions "$name")" = "0" ] && break
    sleep 5
  done

  hap "set server $BACKEND/$name addr $new_ip port 8000"
  hap "set server $BACKEND/$name state ready"
  sleep 30

  # Operational state 2 means the server is up.
  if [ "$(field "$name" 6)" != "2" ]; then
    echo "$name is not up behind the balancer, stopping" >&2
    hap "set server $BACKEND/$name addr $old_ip port 8000"
    destroy_instance "$new_ip"
    exit 1
  fi
  destroy_instance "$old_ip"
done

Note the surge pattern: the new instance is healthy before the old one is drained, so capacity never drops below the original count. Runtime changes through the admin socket are lost on reload unless you persist them with server-state-file or update the configuration afterwards. In a real deployment, also write the new addresses back to your source of truth.

Managed groups provide the same flow. On AWS, for example, an Auto Scaling group can do it declaratively:

resource "aws_autoscaling_group" "web" {
  name                      = "myproject-web"
  min_size                  = 3
  max_size                  = 6
  vpc_zone_identifier       = var.subnet_ids
  target_group_arns         = [aws_lb_target_group.web.arn]
  health_check_type         = "ELB"
  health_check_grace_period = 120

  launch_template {
    id      = aws_launch_template.web.id
    version = aws_launch_template.web.latest_version
  }

  instance_refresh {
    strategy = "Rolling"
    preferences {
      min_healthy_percentage = 90
      instance_warmup        = 120
      auto_rollback          = true
    }
  }
}

Blue-green

Blue-green stands up a complete parallel fleet and moves traffic in one step. With HAProxy, a map file selects the backend, and the runtime API changes it without a reload:

# haproxy.cfg, frontend section
use_backend %[str(active),map(/etc/haproxy/active.map,web_blue)]

# /etc/haproxy/active.map
active web_blue

# Switch traffic, then update the file so a reload keeps it
echo "set map /etc/haproxy/active.map active web_green" | socat stdio /run/haproxy/admin.sock

All three need schema changes that tolerate two application versions. Rolling runs both at once, and any rollback runs old code against the new schema. Use expand and contract migrations: add columns first, deploy, backfill, and remove old columns in a later release.

Where state lives

Deleting an instance must lose nothing. Audit each place the application writes:

In the unit above, ProtectSystem=strict mounts the filesystem read-only for the service, which enforces the rule: a forgotten write to local disk fails in staging rather than silently working until an instance is replaced. Add explicit ReadWritePaths= only for legitimate cache directories.

Stateful nodes can still be treated this way if data sits on a detachable volume with its own lifecycle, but replacement then involves replication, quorum and volume attachment. Most teams do those on a slower schedule, or hand them to a managed service.

Drift detection

Immutability is a policy, and someone will eventually SSH in during an incident. Detect it at four levels:

• Version: have each instance expose /etc/myproject/image-release through a /version endpoint or inventory agent, and alert when it differs from the channel for longer than a rollout takes.
• Files: run aide --check --config=/etc/aide/aide.conf on a timer and ship the result off the host. The cleanup script and first boot change some files after the baseline, such as host keys and the machine ID, so exclude those and other paths that change legitimately, or the alert will be ignored.
• Packages: run dpkg --verify to find modified package files, and compare dpkg-query -W output against the list saved at build time.
• Infrastructure: run terraform plan -detailed-exitcode on a schedule to catch changes to load balancers, networks and launch configurations, which an image cannot protect.

The strongest control is a maximum instance age. If nothing lives longer than a fixed period, a manual change cannot persist beyond it, and you continually prove that replacement works. A simple rule handles interactive access: any host with an SSH login gets tagged as tainted and is replaced within a day. Fix the cause in the image repository, since an unrecorded production fix disappears at the next rollout.

Patching cadence

Split images into two layers when build time becomes a problem: a base image with the OS and hardening, and an application image built from it. Application commits then rebuild only the second layer. The periods below are starting points to adjust against your compliance rules:

Rebuild on the schedule even when nothing seems urgent. A pipeline used every week works when you need it in an emergency, while one used twice a year fails on expired credentials, moved mirrors or stale tests. Kernel patches also take effect automatically, since every new instance boots the new kernel.

Image testing

Test the artifact in three stages, each cheaper to fail than the next.

Static. Mount the image read-only and scan it, before it ever boots:

guestmount -a output/myproject-web-20261105-1412-a1b2c3d/*.qcow2 -i --ro /mnt/image
trivy rootfs --scanners vuln,secret --severity HIGH,CRITICAL \
  --ignore-unfixed --exit-code 1 /mnt/image
guestunmount /mnt/image

Boot. Start the image with production-shaped user-data, built with cloud-localds seed.iso user-data meta-data, on an isolated network with test instances of its dependencies. Then assert the result with Goss:

# tests/goss.yaml
service:
  myproject:
    enabled: true
    running: true
  nftables:
    enabled: true
package:
  unattended-upgrades:
    installed: false
file:
  /etc/ssh/sshd_config.d/10-hardening.conf:
    exists: true
    contents:
      - "PasswordAuthentication no"
  /etc/sudoers.d/90-cloud-init-users:
    exists: false
user:
  packer:
    exists: false
port:
  tcp:8000:
    listening: true
http:
  http://localhost:8000/healthz:
    status: 200
    timeout: 3000

Replacement. In staging, run the actual rollout under light load and watch for failed requests. This catches what the first two stages cannot, such as a drain timeout shorter than your longest request.

Failure modes worth a dedicated test: two clones sharing a machine ID (cleanup skipped), an instance that passes /healthz before workers are ready (check too shallow), and an image that boots only with build-time network access (missing packages hidden by the Packer environment).

Rollback

Rollback is a normal deployment with an older image. Read previous_image_id from the channel, set it as current, and run the same rollout. Using the same path means rollback is exercised constantly and gets no special permissions. A separate rollback script tends to be untested when you need it.

Limits to check in advance:

• Schema: the old release must run against the current schema, or rollback fails on the first query.
• Vulnerabilities: the previous image may contain the hole you just patched. Roll back for availability, then ship a fixed image promptly.
• Configuration: user-data and secrets are versioned separately from the image. Pair each image with a compatible configuration, since a renamed variable breaks an old image too.
• Retention: an aggressive image cleanup job can delete the rollback target. Protect any image still listed in channel history.

Verifying it works

Run these checks against a staging environment before trusting the pipeline:

# Template is valid and formatted
packer init image/
packer fmt -check image/
packer validate -var-file=ci.pkrvars.hcl image/

# Build produced a manifest with the expected version
jq '.builds[-1].custom_data' manifest.json

# On a booted instance
cloud-init status --long
systemctl is-active myproject
cat /etc/myproject/image-release
cat /etc/machine-id
goss -g /tmp/goss.yaml validate    # copied in by the test harness
dpkg --verify

# On two clones, these values must differ
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
cat /etc/machine-id

# During a rollout, from a separate host
while true; do
  curl -s -o /dev/null -w '%{http_code}\n' https://app.example.com/healthz
  sleep 1
done | sort | uniq -c

1. Build the same commit twice and confirm that both images pass the tests and that each gets a unique version.
2. Search a mounted image for known test secrets and for the build user's key; expect no matches.
3. Terminate one instance at random under load. Traffic should stay up and nothing should be lost.
4. Promote a deliberately broken image to staging. The rollout must stop at the first instance, and the fleet must keep serving.
5. Roll back to the previous image using the normal pipeline, and confirm the version endpoint reports it on every instance.
6. Edit a file on one host, then confirm that drift detection raises an alert and that replacing the host removes the change.
7. Stop the secret store in staging. Existing instances should keep serving and new ones should fail health checks without taking traffic.
8. Check that every running instance is younger than your maximum age, and that the rebuild schedule produced an image this week.