Ansible at Scale: Roles, Collections, Dynamic Inventory, Vault, and Testing with Molecule

Ansible stays simple for ten hosts and stops being simple around a few hundred, when inventories change hourly, several teams share roles, and one bad variable can restart a whole fleet. This guide covers the structure and habits that keep a large Ansible codebase predictable: a sane project layout, roles with clear interfaces, pinned collections, inventory generated from the cloud API, secrets that never reach a log, rolling updates that stop themselves, and tests that run before anything touches production.

Project layout and toolchain pinning

A layout should answer three questions without reading any YAML: what environments exist, what playbooks can be run, and where a given behaviour is implemented. Keep one inventory directory per environment so that a wrong -i flag is obvious, and keep playbooks thin. Logic belongs in roles.

myproject/
  ansible.cfg
  requirements.txt
  collections/
    requirements.yml
  inventories/
    staging/
    production/
      aws_ec2.yml
      static.yml
      group_vars/
        all/
          vars.yml
          vault.yml
        role_web.yml
  playbooks/
    site.yml
    rolling_deploy.yml
  roles/
    myapp/
      defaults/main.yml
      handlers/main.yml
      meta/main.yml
      meta/argument_specs.yml
      molecule/default/
      tasks/main.yml
      tasks/migrate.yml
      templates/
      vars/debian.yml
      vars/redhat.yml

Pin the control node as well as the code. A playbook is only reproducible if the same ansible-core and the same collection versions run it, so install into a virtual environment from a committed file. Current ansible-core releases require a recent Python on the control node (2.22 needs 3.13 or newer), and managed nodes have their own minimum, so check the support matrix before an upgrade rather than after.

# requirements.txt
ansible-core>=2.22,<2.23
ansible-lint>=26.11,<27.0
molecule>=26.11,<27.0
molecule-plugins[docker]>=26.11,<27.0

The ansible-core constraint is narrower than the others because its minor releases can change behaviour. Version 2.19 introduced a new templating engine with stricter rules: a conditional must now produce a boolean, so when: myvar on a string fails where it once passed. Treat each minor upgrade as a project with a test run. Add boto3 for the AWS collection and hvac for HashiCorp-style vaults, pinned the same way.

Commit a project-level configuration so that every engineer and CI job behaves the same way:

# ansible.cfg
[defaults]
inventory = inventories/staging
roles_path = roles
collections_path = collections
forks = 30
gathering = smart
fact_caching = ansible.builtin.jsonfile
fact_caching_connection = .cache/facts
fact_caching_timeout = 3600
callback_result_format = yaml
callbacks_enabled = ansible.posix.profile_tasks
interpreter_python = auto_silent

[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=60s

Defaulting inventory to staging is deliberate: production then requires a typed -i inventories/production. Ansible ignores an ansible.cfg in a world-writable directory. Run ansible-config dump --only-changed to see what is actually in effect.

Role design: defaults, vars, and precedence

Treat a role as a function. Its public interface is defaults/main.yml and meta/argument_specs.yml. Everything else is implementation. Prefix every variable with the role name, because Ansible has one flat variable namespace per host and port in one role will silently collide with port in another. The var-naming[no-role-prefix] rule in ansible-lint enforces this.

# roles/myapp/defaults/main.yml
---
myapp_user: myapp
myapp_root: /srv/app
myapp_port: 8080
myapp_log_level: info
myapp_exec_start: "{{ myapp_root }}/current/bin/myapp serve --port {{ myapp_port }}"

# roles/myapp/vars/debian.yml
---
myapp_packages:
  - ca-certificates
  - libpq5

# roles/myapp/meta/argument_specs.yml
---
argument_specs:
  main:
    short_description: Install and run the myapp service
    options:
      myapp_db_password:
        type: str
        required: true
        no_log: true
        description: Database password written to the environment file.
      myapp_port:
        type: int
        description: TCP port the service listens on.
      myapp_log_level:
        type: str
        choices: [debug, info, warning, error]
        description: Application log level.

Argument specs run as an implicit first task and fail with a readable message, which replaces the hand-written assert blocks that used to open most roles. They validate; they do not set values. Keep actual defaults in defaults/main.yml.

The rule for placing a variable is about who may change it. Anything a consumer may tune goes in defaults/, the lowest-priority location, so any inventory value wins. Anything that is a fact about the platform, such as package names per OS family, goes in vars/, where inventory cannot accidentally override it.

Precedence that matters in practice

There are over twenty precedence levels. These are the ones that cause incidents, from lowest to highest:

Two more details. ansible_group_priority changes merge order between sibling groups but only works in inventory sources, not in group_vars files. And leave hash_behaviour at its default of replace: global merging makes results depend on load order. If you need a merge, do it explicitly with the combine filter at the point of use. ansible-inventory --host shows inventory-level values only; use ansible.builtin.debug inside the play for values set later.

Tasks and handlers

# roles/myapp/tasks/main.yml
---
- name: Load OS-specific variables
  ansible.builtin.include_vars: "{{ ansible_facts['os_family'] | lower }}.yml"

- name: Install runtime packages
  ansible.builtin.package:
    name: "{{ myapp_packages }}"
    state: present

- name: Create service group
  ansible.builtin.group:
    name: "{{ myapp_user }}"
    system: true

- name: Create service account
  ansible.builtin.user:
    name: "{{ myapp_user }}"
    group: "{{ myapp_user }}"
    system: true
    shell: /usr/sbin/nologin
    home: "{{ myapp_root }}"
    create_home: false

- name: Create application directories
  ansible.builtin.file:
    path: "{{ item }}"
    state: directory
    owner: "{{ myapp_user }}"
    group: "{{ myapp_user }}"
    mode: "0750"
  loop:
    - "{{ myapp_root }}"
    - "{{ myapp_root }}/shared"
    - "{{ myapp_root }}/releases"

- name: Render environment file
  ansible.builtin.template:
    src: app.env.j2
    dest: "{{ myapp_root }}/shared/app.env"
    owner: root
    group: "{{ myapp_user }}"
    mode: "0640"
  no_log: true
  diff: false
  notify: Restart myapp

- name: Install systemd unit
  ansible.builtin.template:
    src: myapp.service.j2
    dest: /etc/systemd/system/myapp.service
    owner: root
    group: root
    mode: "0644"
  notify:
    - Reload systemd
    - Restart myapp

- name: Enable and start service
  ansible.builtin.systemd_service:
    name: myapp
    enabled: true
    state: started
  ignore_errors: "{{ ansible_check_mode }}"

# roles/myapp/templates/myapp.service.j2
[Unit]
Description=myapp
After=network-online.target

[Service]
User={{ myapp_user }}
EnvironmentFile={{ myapp_root }}/shared/app.env
ExecStart={{ myapp_exec_start }}
Restart=on-failure

[Install]
WantedBy=multi-user.target

# roles/myapp/handlers/main.yml
---
- name: Reload systemd
  ansible.builtin.systemd_service:
    daemon_reload: true

- name: Restart myapp
  ansible.builtin.systemd_service:
    name: myapp
    state: restarted
  ignore_errors: "{{ ansible_check_mode }}"

Unpacking a release under releases/ and switching the current symlink follows the same pattern and is omitted here. Handlers run in the order they are defined, not the order they were notified, so Reload systemd must sit above Restart myapp. They run at the end of each section of a play, once per host, however many tasks notified them. If a later task fails, pending handlers are lost for that host and the next run sees no change, so the restart never happens. Set force_handlers = True or call ansible.builtin.meta: flush_handlers before risky steps.

Use import_tasks when the file list is fixed: imports are resolved at parse time, so tags and --list-tasks work. Use include_tasks when the file depends on a runtime value.

Idempotence and check/diff mode

An idempotent play reports changed=0 on its second run. If every run shows 40 changes, nobody can spot the one that matters. The usual offenders are command and shell tasks, which always report changed, and templates containing timestamps.

- name: Read current schema version
  ansible.builtin.command:
    cmd: "{{ myapp_root }}/current/bin/myapp db version"
  register: myapp_schema
  changed_when: false
  check_mode: false

- name: Apply migrations
  ansible.builtin.command:
    cmd: "{{ myapp_root }}/current/bin/myapp db migrate"
  register: myapp_migrate
  changed_when: "'applied' in myapp_migrate.stdout"
  when: myapp_schema.stdout | trim != myapp_target_schema

check_mode: false makes a read-only probe run even under --check. Without it, the task is skipped, myapp_schema.stdout is undefined, and the conditional on the next task fails. Prefer a real module to command; failing that, use creates or removes.

Run --check --diff as a preview of a change, and know its limits. Check mode cannot see the effects of tasks it simulated: if a package is not actually installed, a later task that reads its config file fails or gives a wrong answer. That is why the service tasks above use ignore_errors: "{{ ansible_check_mode }}", since the unit file does not exist yet on a fresh host. The ansible-lint ignore-errors rule accepts this specific form. Modules without check mode support are skipped, so a clean check run does not prove a clean real run.

Diff output prints file contents, so anything secret in a template appears in CI logs unless the task sets diff: false. no_log: true hides failure messages too, so keep it on as few tasks as possible.

FQCN and collections

Since the split of Ansible into ansible-core and collections, a module name has three parts: namespace, collection, and plugin. Always write ansible.builtin.copy, not copy. A short name is resolved through a search list, so an installed collection can shadow it. The play-level collections: keyword narrows the search but does not remove the ambiguity. The fqcn lint rule flags the short form, and ansible-lint --fix rewrites most cases.

# collections/requirements.yml
---
collections:
  - name: amazon.aws
    version: ">=12.1.0,<13.0.0"
  - name: ansible.posix
    version: ">=2.2.2,<3.0.0"
  - name: community.general
    version: ">=14.0.0,<15.0.0"
  - name: community.hashi_vault
    version: ">=7.1.0,<8.0.0"

ansible-galaxy collection install -r collections/requirements.yml -p collections/
ansible-galaxy collection list -p collections/

Ranges let minor updates arrive, but the install you tested and the install CI performs next month can differ. For full reproducibility, pin exact versions and update them through a scheduled pull request, or vendor the resolved set into an execution environment image built with ansible-builder. A private mirror or pre-built image also removes public Galaxy from the release path.

Dynamic inventory and group_vars

At scale, a hand-kept host list is wrong within hours. Inventory plugins query the source of truth at runtime. The filename matters: the AWS plugin only accepts files ending in aws_ec2.yml, and the Azure and GCP plugins (azure.azcollection.azure_rm, google.cloud.gcp_compute) require similar suffixes.

# inventories/production/aws_ec2.yml
---
plugin: amazon.aws.aws_ec2
regions:
  - eu-west-1
filters:
  instance-state-name: running
  tag:Environment: production
  tag-key: Role
hostnames:
  - private-ip-address
compose:
  ansible_host: private_ip_address
keyed_groups:
  - key: tags.Role
    prefix: role
  - key: placement.availability_zone
    prefix: az
groups:
  canary: "tags.get('Canary') == 'true'"
strict: true
cache: true
cache_plugin: ansible.builtin.jsonfile
cache_connection: .cache/inventory
cache_timeout: 300

An instance tagged Role=web lands in role_web. strict: true makes a bad compose or keyed_groups expression fail rather than quietly dropping a host; the tag-key filter ensures every returned instance has the tag being keyed on. Group names built from values containing hyphens, such as zone names, are subject to the TRANSFORM_INVALID_GROUP_CHARS setting, so run --graph to see the names you actually get before referencing them.

Pointing -i at the directory merges every source in it, in ASCII filename order, and loads group_vars/ beside them. A static file adds stable abstractions on top of generated groups:

# inventories/production/static.yml
---
all:
  children:
    frontend:
      children:
        role_web:
        role_lb:

Variables then live in group_vars/role_web.yml and are keyed to the tag rather than to host names, so a replacement instance picks up configuration without a code change.

Failure modes to plan for:

• Silent empty groups. A mistyped tag leaves role_web empty, and the play is skipped with a warning and a zero exit code. The result looks like success. Gate deploys with a check on the expected host count.
• Stale cache. A five-minute cache can target terminated instances or miss new ones. Pass --flush-cache to ansible-playbook for deploys.
• API dependency. Throttling or an outage of the cloud API now blocks your ability to fix things. A short-lived cache and a documented static fallback both help.

Ansible Vault and alternatives

Vault encrypts data at rest in the repository with AES-256. A widely used convention keeps secrets findable: put encrypted values in vault.yml with a vault_ prefix, and reference them from a plain vars.yml.

# group_vars/all/vars.yml (plain)
myapp_db_password: "{{ vault_myapp_db_password }}"

# group_vars/all/vault.yml (encrypted with ansible-vault)
vault_myapp_db_password: "replace-me"

ansible-vault create --vault-id prod@prompt inventories/production/group_vars/all/vault.yml
ansible-vault rekey --vault-id prod@prompt inventories/production/group_vars/all/vault.yml
ansible-vault encrypt_string --vault-id prod@prompt --stdin-name vault_api_token

Use one vault ID per environment so that a staging password cannot open production data. In CI, use a client script instead of a prompt: --vault-id prod@scripts/vault-pass-client.sh. An executable whose name ends in -client receives the vault ID as an argument and prints the password to stdout, so it can fetch it from your CI secret store. Vault's weaknesses are operational: one shared password, rotation that means re-encrypting and redistributing, no record of who read what, and merge conflicts on whole-file encrypted blobs.

- name: Read database credentials
  ansible.builtin.set_fact:
    myapp_db_password: >-
      {{ lookup('community.hashi_vault.vault_kv2_get', 'myapp/db',
                engine_mount_point='secret').secret.password }}
  no_log: true
  delegate_to: localhost
  run_once: true

Variables are lazy, so a lookup placed directly in group_vars runs every time it is referenced, on the control node, once per host and task. Resolve once with set_fact, as above; with run_once, the fact applies to all hosts in the current batch. -vvv output and fact caches can also expose values, so keep verbose runs out of shared CI logs.

Rolling updates with serial and max_fail_percentage

By default a play runs each task on all hosts in parallel, so a bad release reaches everything at once. serial splits the play into batches that complete in full before the next starts. A list ramps batch size, so the first host works as a canary.

# playbooks/rolling_deploy.yml
---
- name: Migrate schema once before any host changes
  hosts: role_web[0]
  become: true
  tasks:
    - name: Run migrations
      ansible.builtin.import_role:
        name: myapp
        tasks_from: migrate

- name: Rolling update of web tier
  hosts: role_web
  become: true
  serial:
    - 1
    - "10%"
    - "25%"
  max_fail_percentage: 10
  order: shuffle
  pre_tasks:
    - name: Drain host from load balancers
      community.general.haproxy:
        state: disabled
        host: "{{ inventory_hostname }}"
        backend: myapp
        socket: /run/haproxy/admin.sock
        drain: true
        wait_interval: 5
        wait_retries: 60
      delegate_to: "{{ item }}"
      loop: "{{ groups['role_lb'] }}"
  roles:
    - role: myapp
  post_tasks:
    - name: Wait for local health endpoint
      ansible.builtin.uri:
        url: "http://127.0.0.1:{{ myapp_port }}/healthz"
        status_code: 200
      register: myapp_health
      until: myapp_health.status == 200
      retries: 30
      delay: 2

    - name: Return host to load balancers
      community.general.haproxy:
        state: enabled
        host: "{{ inventory_hostname }}"
        backend: myapp
        socket: /run/haproxy/admin.sock
        wait: true
      delegate_to: "{{ item }}"
      loop: "{{ groups['role_lb'] }}"

tasks/migrate.yml holds the two migration tasks from the idempotence section, with myapp_target_schema set in group_vars. Migrating first assumes the schema change is backward compatible, because old code runs against the new schema until the last batch finishes. Handlers are flushed between roles and post_tasks, so the restart happens before the health check. A host that fails the check stays drained, which is the outcome you want.

The semantics have sharp edges:

• max_fail_percentage is evaluated per batch and must be exceeded, not met. With a batch of 10 and a value of 10, one failure continues and two abort. With a batch of 1, one failure aborts. Small batches make the setting all-or-nothing.
• If every host in a batch fails, the play stops regardless of settings.
• any_errors_fatal: true stops the whole play after the current task finishes on the batch. Use it where partial state is dangerous, such as database clusters.
• run_once combined with serial runs once per batch, not once per play. That is why the migration lives in its own play. --limit is applied after the [0] subscript, so a limited run can match no host and skip the migration.

throttle caps concurrency on a single task, such as an artifact download that would saturate a shared service.

Performance: forks, pipelining, and mitogen

Measure first. The profile_tasks callback enabled earlier prints the slowest tasks. Then work through these settings, roughly in order of payoff:

Prefer passing a list to ansible.builtin.package over looping one package at a time, since a loop makes one module call per item. Run the control node close to the managed hosts.

Mitogen for Ansible replaces the SSH transport with a persistent Python process on each target and can cut execution time on module-heavy plays. It installs as a strategy plugin, using strategy_plugins and strategy = mitogen_linear. The caveats are real. It hooks into ansible-core internals, so support lags new core versions, which conflicts with staying current. Its behaviour around become, connection variables, and some plugins differs from stock Ansible, and failures are harder to diagnose. Check the compatibility notes for your exact versions, try pipelining and profiling first, and treat it as an optional accelerator you can remove without changing any playbook.

Linting with ansible-lint

Lint in CI on every change. Profiles offer a graded path: start lower on an old codebase and raise the bar over time. The available profiles are min, basic, moderate, safety, shared, and production.

# .ansible-lint
---
profile: production
exclude_paths:
  - collections/
  - .cache/
skip_list:
  - galaxy[no-changelog]
warn_list:
  - experimental
offline: true

Rules worth understanding rather than silencing include no-changed-when, risky-file-permissions (a missing mode leaves the result to umask and module defaults), no-handler, and command-instead-of-module. Exempt a single line with a trailing noqa comment and a reason, not with a global skip. Lint says nothing about whether a role works, which is what Molecule is for.

Testing roles with Molecule

Molecule creates a throwaway instance, applies the role, applies it again to check idempotence, and runs assertions. Testing a systemd service in a container needs an image with an init system, so build one once:

# roles/myapp/molecule/default/Dockerfile
FROM debian:13
RUN apt-get update \
 && apt-get install -y --no-install-recommends python3 sudo systemd systemd-sysv
CMD ["/lib/systemd/systemd"]

docker build -t myproject/debian13-systemd:latest roles/myapp/molecule/default

# roles/myapp/molecule/default/molecule.yml
---
role_name_check: 1
dependency:
  name: galaxy
  enabled: false
driver:
  name: docker
platforms:
  - name: debian13
    image: myproject/debian13-systemd:latest
    pre_build_image: true
    pull: false
    command: /lib/systemd/systemd
    privileged: true
    cgroupns_mode: host
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:rw
verifier:
  name: ansible

# roles/myapp/molecule/default/converge.yml
---
- name: Converge
  hosts: all
  become: true
  vars:
    myapp_db_password: molecule-only
    myapp_exec_start: /bin/sleep infinity
  pre_tasks:
    - name: Refresh apt cache
      ansible.builtin.apt:
        update_cache: true
        cache_valid_time: 3600
  roles:
    - role: myapp

# roles/myapp/molecule/default/verify.yml
---
- name: Verify
  hosts: all
  become: true
  gather_facts: false
  tasks:
    - name: Collect service state
      ansible.builtin.service_facts:

    - name: Read environment file metadata
      ansible.builtin.stat:
        path: /srv/app/shared/app.env
      register: myapp_env

    - name: Assert service and file state
      ansible.builtin.assert:
        that:
          - ansible_facts.services['myapp.service'].state == 'running'
          - ansible_facts.services['myapp.service'].status == 'enabled'
          - myapp_env.stat.exists
          - myapp_env.stat.mode == '0640'

The stub ExecStart shows why it pays to expose that value in defaults: the role can be tested without a real release. Dependency installation is disabled because this role uses only ansible.builtin; a role that needs collections should have them installed in the CI step before Molecule runs. The role_name_check: 1 setting downgrades the fully qualified role name check to a warning, which suits roles not published to Galaxy.

cd roles/myapp
molecule test                     # full lifecycle, including destroy
molecule converge                 # apply and keep the instance
molecule verify                   # run assertions only
molecule login --host debian13        # shell into the instance
molecule test --destroy never     # keep the instance after a failure
molecule destroy

Know what this leaves untested. A privileged container shares the host kernel, so kernel modules, sysctl, firewall rules, and boot ordering behave differently. Rolling logic and real load balancers need a staging environment. Molecule's configuration schema has moved over time, with a top-level ansible section superseding parts of provisioner, so check the docs for your installed version when you upgrade.

Verifying it works

Run these in order before merging, and again before a production deploy.

# Toolchain and effective configuration
ansible --version
ansible-config dump --only-changed
ansible-galaxy collection list -p collections/

# Inventory matches reality
ansible-inventory -i inventories/production --graph
ansible-inventory -i inventories/production --host 10.0.1.25 --yaml
ansible -i inventories/production role_web -m ansible.builtin.ping

# Static checks
ansible-lint
ansible-playbook -i inventories/production playbooks/site.yml --syntax-check
ansible-playbook -i inventories/production playbooks/rolling_deploy.yml --list-hosts --list-tasks

# Role behaviour
(cd roles/myapp && molecule test)

# Preview, then a canary
ansible-playbook -i inventories/production playbooks/site.yml --check --diff --limit canary
ansible-playbook -i inventories/production playbooks/rolling_deploy.yml --limit canary

# Idempotence on a real host: expect changed=0
ansible-playbook -i inventories/production playbooks/site.yml --limit canary

• The --graph output contains every expected group, with the expected host count, and no group is unexpectedly empty.
• git grep -L '\$ANSIBLE_VAULT' -- '*vault.yml' prints nothing, showing that no vault file is committed unencrypted.
• A deliberately failed health check in staging stops the rollout at the first batch and leaves that host drained.
• Output from a --diff run in CI contains no secret values.