Module 09 — Dynamic Inventory & Cloud

Updated 20 August 2026

Module 09 · Dynamic Inventory & Cloud

Every module so far assumed you know which servers exist. In any cloud environment you do not — the list changes while you are reading it. This module makes the inventory a query rather than a file.

🧠 concept → 🧪 exercise → ✅ expected result (hidden) → 🎯 interview questions (answers hidden)

Prerequisite: Modules 01–08. You saw a preview of aws_ec2 in Module 01 Part B1 — this is the full treatment.


Part A · Why static inventory fails

A1 · The problem

The analogy. Think of a printed telephone directory. It was completely accurate on the day it went to the printer. By the time it lands on your doorstep people have moved, changed numbers and been disconnected — and the book has no way of telling you which of its entries are now wrong. You only find out when you dial and nobody answers, or when the person you needed was never in it at all.

A static inventory file is that directory; a dynamic inventory is looking the number up live at the moment you dial. The contacts app from Module 01 B1 worked fine when you were adding servers by hand. Once an autoscaling group can create and destroy machines while you are reading the file, a printed list is not just inconvenient — it is wrong by design.

Diagram source
flowchart TD
    A["STATIC inventory<br>hosts.yml"] --> A1["You edit it by hand"]
    A1 --> A2["Autoscaling adds 12 instances"]
    A2 --> A3["❌ File is now wrong<br>New hosts unmanaged<br>Terminated hosts still listed"]
    B["DYNAMIC inventory<br>aws_ec2.yml"] --> B1["Plugin queries the cloud API<br>at the moment you run"]
    B1 --> B2["Autoscaling adds 12 instances"]
    B2 --> B3["✅ Next run includes them<br>automatically<br>Terminated hosts disappear"]
    style A3 fill:#FEE2E2,stroke:#DC2626
    style B3 fill:#D1FAE5,stroke:#059669,stroke-width:2px
A static inventory is a cache of reality that nothing invalidates. The moment autoscaling, spot instances, or anyone using the cloud console exists, the file is wrong and nobody is told.

The two failure modes are asymmetric and both bad: missing hosts silently go unpatched and unmonitored, and stale hosts make every run report unreachable errors until people learn to ignore red output — which is how a genuinely unreachable production host gets missed.

A2 · Plugins, not scripts

ApproachDetail
Inventory plugin⭐ The modern way. A YAML config file declaring plugin: and its options. Supports caching, keyed_groups, compose, and runs in-process
Inventory scriptLegacy. Any executable that prints inventory JSON when called with --list. Still supported, still works, but no caching and no shared features
You will still meet scripts — the old ec2.py is in a lot of legacy repositories. It works, but every plugin feature (caching, keyed_groups, compose, constructed) had to be reimplemented inside each script, which is exactly why plugins replaced them.

If asked to modernise one: the plugin config is usually 15 lines of YAML replacing several hundred lines of Python.

A3 · How a plugin is found

The analogy. Think of a form that only gets processed if the filename is right.

You fill it in perfectly and email it in. Nothing comes back — no rejection, no error, just silence — because the department's system only picks up files named a particular way, and yours was not.

Silence is the worst possible response, because everything you can see says you did it correctly.

The aws_ec2 plugin behaves exactly like this. Name the file aws.yml instead of prod.aws_ec2.yml and you get an empty inventory, no error, and a run that "succeeds" having done nothing at all.

plain text
# ansible.cfg
[inventory]
enable_plugins = auto, host_list, yaml, ini, toml, amazon.aws.aws_ec2, constructed
cache = True
cache_plugin = jsonfile
cache_connection = /tmp/ansible_inventory_cache
cache_timeout = 3600
The filename rule that silently breaks everything. The aws_ec2 plugin requires its config file to end in aws_ec2.yml or aws_ec2.yaml. Name it inventory/aws.yml and you get an empty inventory with no error at all — Ansible simply does not recognise the file as belonging to that plugin.

This is the single most common dynamic-inventory support question, and the reason it is so confusing is that everything looks correct. Other cloud plugins have equivalent conventions: azure_rm.yml, gcp_compute.yml.

🧪 Exercise A3.1 — Reproduce the empty-inventory trap
bash
mkdir -p inventory
cat > inventory/wrongname.yml <<'EOF'
plugin: amazon.aws.aws_ec2
regions:
  - ap-southeast-1
EOF
cp inventory/wrongname.yml inventory/prod.aws_ec2.yml

ansible-inventory -i inventory/wrongname.yml --graph
ansible-inventory -i inventory/prod.aws_ec2.yml --graph
ansible-inventory -i inventory/wrongname.yml --graph -vvvv 2>&1 | grep -i 'declined\|skipping\|parse'
Expected result — click to reveal

Wrong filename:

plain text
@all:
  |--@ungrouped:

Correct filename:

plain text
@all:
  |--@aws_ec2:
  |  |--10.0.1.15
  |  |--10.0.1.22
  |  |--10.0.2.8
  |--@ungrouped:

And the reason, only visible at -vvvv:

plain text
auto declined parsing /home/zaeem/lab/inventory/wrongname.yml as it did not pass its verify_file() method

An empty inventory is not an error to Ansible. "No hosts matched" is a legitimate outcome, so the run exits successfully having done nothing at all. In CI that is a green pipeline that patched zero servers.

The diagnostic worth memorising: when a dynamic inventory returns nothing, run -vvvv and grep for declined. It tells you which plugin refused the file and why, and nine times out of ten the answer is the filename.

Build the habit: always ansible-inventory --graph after touching an inventory config, before running any playbook against it. It costs nothing and it is the only thing standing between you and a silent no-op.

🎯 Interview questions — Dynamic inventory basics

Q. Why is a static inventory unacceptable in a cloud environment?

It is a hand-maintained cache of reality that nothing invalidates. The moment autoscaling, spot instances or console-created resources exist, the file is wrong and nobody is notified.

Both failure modes are bad: missing hosts go silently unpatched and unmonitored, and stale entries produce unreachable errors on every run until people learn to ignore red output — which is how a genuinely down production host gets overlooked.

Dynamic inventory makes the host list a query against the source of truth, evaluated at run time.

Q. Inventory plugin or inventory script?

Plugin. It is a YAML config declaring plugin: and its options, runs in-process, and gets caching, keyed_groups, compose and constructed for free.

Scripts are the legacy mechanism — any executable printing inventory JSON for --list. Still supported and still common in older repositories, but every feature had to be reimplemented inside each script, which is why plugins replaced them. Modernising one typically replaces several hundred lines of Python with about fifteen lines of YAML.

Q. Your aws_ec2 inventory returns no hosts and no error. What do you check?

The filename first — it must end in aws_ec2.yml or aws_ec2.yaml, or the plugin silently declines the file and you get an empty inventory with no error.

Then -vvvv and grep for declined, which names the plugin that refused it and why. Then credentials, then region, then whether the filters match anything.

The dangerous part is that an empty inventory is a successful run — in CI, a green pipeline that configured nothing.


Part B · aws_ec2 in depth

B1 · A complete, production-shaped config

The analogy. Think of moving house. Every box gets a label — Kitchen, Books, Bedroom — so the movers can put each one in the right room without opening any of them, and so you can find the kettle on the first night. And the box you forgot to label is the one that ends up in the garage and stays there for two years.

Cloud tags are those labels, and keyed_groups is the movers reading them. An untagged server is the unlabelled box: it exists, it costs money, and nothing ever gets done to it — no patching, no config, no monitoring — because it never lands in any group your playbooks target.

yaml
# inventory/production.aws_ec2.yml
---
plugin: amazon.aws.aws_ec2

regions:
  - ap-southeast-1
  - ap-southeast-2

# Only instances we should manage
filters:
  instance-state-name: running
  tag:Environment: production
  tag:ManagedBy: ansible

# What is the host called in the inventory?
hostnames:
  - tag:Name
  - private-ip-address          # fallback if the Name tag is missing

# Turn tags into groups - THE most valuable feature
keyed_groups:
  - key: tags.Role
    prefix: role
    separator: "_"              # tags.Role=web  ->  group role_web
  - key: tags.Environment
    prefix: env
  - key: placement.availability_zone
    prefix: az                  # -> az_ap_southeast_1a
  - key: instance_type
    prefix: type

# Conditional groups from an expression
groups:
  large_instances: "instance_type.startswith('m5.4x') or instance_type.startswith('c5.9x')"
  needs_patching: "'PatchGroup' in (tags | default({}))"

# Build extra host variables
compose:
  ansible_host: private_ip_address
  ec2_instance_id: instance_id
  ec2_launch_time: launch_time
  app_tier: tags.Role | default('untagged')

strict: false                   # do not fail if a compose expression is undefined

cache: true
cache_plugin: jsonfile
cache_connection: /tmp/aws_inv_cache
cache_timeout: 900

B2 · keyed_groups — the feature that matters most

The analogy. Think of a conference where everyone wears a coloured lanyard. Nobody maintains a list of who the speakers are — the lanyard is the list. You look around the room and the red ones are speakers, whether there are twelve of them or two hundred. People arrive and leave all day and it keeps working, because there was never a list to update.

keyed_groups turns tags into groups in exactly that way, which is why hosts: role_web stays correct as an autoscaling group grows and shrinks overnight. It also has a consequence worth saying out loud in an interview: your tagging discipline becomes your inventory. Sloppy tags are no longer a documentation problem, they are an outage waiting to happen.

Diagram source
flowchart LR
    A["EC2 instance<br>tags:<br>Role=web<br>Environment=production<br>AZ=ap-southeast-1a"] --> B["keyed_groups"]
    B --> C["group: role_web"]
    B --> D["group: env_production"]
    B --> E["group: az_ap_southeast_1a"]
    C --> F["hosts: role_web<br>in your playbook"]
    style B fill:#DDD6FE,stroke:#7C3AED,stroke-width:2px
    style F fill:#D1FAE5,stroke:#059669,stroke-width:2px
This is the sentence to say in an interview: "keyed_groups turns EC2 tags into Ansible groups automatically, so hosts: role_web stays correct as the autoscaling group grows and shrinks — nobody edits an inventory file, and the tag becomes the contract between the infrastructure team and the automation."

The consequence is organisational as much as technical: your tagging discipline becomes your inventory. An untagged instance is an unmanaged instance, which is a good property once the team understands it.

OptionEffect
keyThe instance attribute or tag to group by — tags.Role, instance_type, placement.availability_zone
prefixPrepended to the group name. Without it, a tag value of web creates a bare group web
separatorBetween prefix and value. Default _
parent_groupMakes every generated group a child of this one
default_valueGroup name used when the key is missing — catches untagged instances
trailing_separatorSet false to avoid role_ for an empty value
Group names are sanitised. Ansible group names cannot contain dots, hyphens or spaces, so ap-southeast-1a becomes ap_southeast_1a and a tag value of web-tier becomes web_tier. If your playbook says hosts: role_web-tier it will match nothing.

Always confirm the real names with ansible-inventory --graph rather than assuming what the sanitiser produced.

B3 · hostnames and compose

The analogy. Think of a contact saved on your phone.

The screen says "Mum". The phone dials +44 7700 900123. You read the name; the network needs the number, and confusing the two means nothing connects.

You would never try to dial the word "Mum".

hostnames is the name on the screen — what appears in your output. ansible_host is the number actually dialled. Omit the compose: ansible_host line and Ansible tries to dial "web01", which is not a number.

yaml
hostnames:
  - tag:Name                    # readable, if you tag consistently
  - private-ip-address          # fallback
  - dns-name

compose:
  ansible_host: private_ip_address     # what Ansible actually connects to
hostnames and ansible_host are different things, and conflating them is a classic error.

hostnames decides the label — what appears in output, and what inventory_hostname returns.

ansible_host (usually set via compose) decides the address Ansible connects to.

So you get readable output showing web01 while connecting to 10.0.1.15. Without the compose line, Ansible tries to resolve web01 as a DNS name and fails.

This is the same inventory_hostname versus real address distinction from Module 02 Part B4, now with a cloud cause.

B4 · Credentials and permissions

bash
# The standard boto3 credential chain, in order:
# 1. Environment variables
export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_REGION=ap-southeast-1
# 2. ~/.aws/credentials  (a named profile)
export AWS_PROFILE=production
# 3. An EC2 instance profile / IAM role  <- best for a control node in AWS
# 4. IRSA, if running in EKS
Never put credentials in the inventory file. The plugin accepts aws_access_key and aws_secret_key options, and using them means committing keys to Git.

Use the standard chain: an IAM role on the control node in AWS, or a named profile locally, or OIDC federation in CI. The IAM policy needs only ec2:DescribeInstances, ec2:DescribeTags, ec2:DescribeRegions and similar read actions — inventory should never hold write permissions.

B5 · Caching the inventory

plain text
[inventory]
cache = True
cache_plugin = jsonfile
cache_connection = /tmp/ansible_inventory_cache
cache_timeout = 900
bash
ansible-inventory -i inventory/production.aws_ec2.yml --graph --flush-cache
Inventory caching is separate from fact caching (Module 08 Part C2). Fact caching stores what a host is; inventory caching stores which hosts exist.

It matters because the AWS API call is slow and rate-limited — with several thousand instances, DescribeInstances paginates and can take many seconds on every single ad-hoc command.

The trade is the same: a newly launched instance will not appear until the cache expires, so --flush-cache belongs in any run that must catch new hosts.

🧪 Exercise B5.1 — Inspect what a dynamic inventory actually produced
bash
ansible-inventory -i inventory/production.aws_ec2.yml --graph
ansible-inventory -i inventory/production.aws_ec2.yml --graph --vars | head -40
ansible-inventory -i inventory/production.aws_ec2.yml --list > /tmp/inv.json
python3 -c "import json;d=json.load(open('/tmp/inv.json'));print('\n'.join(sorted(k for k in d if k!='_meta')))"
ansible-inventory -i inventory/production.aws_ec2.yml --host web01
ansible -i inventory/production.aws_ec2.yml role_web --list-hosts
Expected result — click to reveal
plain text
@all:
  |--@aws_ec2:
  |  |--web01
  |  |--web02
  |  |--db01
  |--@az_ap_southeast_1a:
  |  |--web01
  |  |--db01
  |--@az_ap_southeast_1b:
  |  |--web02
  |--@env_production:
  |  |--web01
  |  |--web02
  |  |--db01
  |--@role_db:
  |  |--db01
  |--@role_web:
  |  |--web01
  |  |--web02
  |--@type_t3_medium:
  |  |--web01
  |  |--web02
  |--@ungrouped:
json
{
    "ansible_host": "10.0.1.15",
    "app_tier": "web",
    "ec2_instance_id": "i-0abc123def456",
    "instance_type": "t3.medium",
    "placement": {"availability_zone": "ap-southeast-1a"},
    "private_ip_address": "10.0.1.15",
    "tags": {"Environment": "production", "Name": "web01", "Role": "web"}
}

Read three things out of this.

  1. The group names are sanitisedaz_ap_southeast_1a, not az_ap-southeast-1a, and type_t3_medium from t3.medium. Writing hosts: type_t3.medium in a playbook would match nothing. Never guess these; read them from --graph.
  2. ansible_host is 10.0.1.15 while the inventory name is web01. That is hostnames and compose doing their separate jobs — readable output, correct connection target.
  3. Every EC2 attribute is available as a host variable, so instance_type, tags, placement and the rest are usable in when: conditions and templates directly. A playbook can branch on when: instance_type.startswith('m5') with no extra work.

--host <name> is the fastest way to answer "why did this host get that value?" and it makes no connection to the host at all.

🎯 Interview questions — aws_ec2

Q. What is keyed_groups and why does it matter?

It generates Ansible groups from instance attributes or tags — tags.Role=web becomes a group role_web with a prefix.

It matters because hosts: role_web then stays correct as an autoscaling group grows and shrinks, with nobody editing an inventory file. The tag becomes the contract between whoever provisions infrastructure and whoever automates it.

The organisational consequence worth naming: your tagging discipline becomes your inventory, and an untagged instance is an unmanaged one.

Q. What is the difference between hostnames and ansible_host here?

hostnames sets the inventory label — what shows in output and what inventory_hostname returns. ansible_host, usually set through compose, is the address Ansible connects to.

So hostnames: [tag:Name] with compose: {ansible_host: private_ip_address} gives readable output naming web01 while connecting to 10.0.1.15. Omit the compose line and Ansible tries to DNS-resolve web01 and fails.

Q. How should the inventory plugin authenticate to AWS?

Through the standard boto3 chain — environment variables, a named profile, or best, an IAM role attached to the control node, or OIDC federation in CI.

Never aws_access_key in the inventory file, since that commits credentials to Git.

The policy should be read-only: ec2:DescribeInstances, ec2:DescribeTags, ec2:DescribeRegions. An inventory has no reason to hold write permissions.

Q. Why would you cache a dynamic inventory, and what does it cost?

The cloud API call is slow and rate-limited — with thousands of instances DescribeInstances paginates and adds seconds to every ad-hoc command, not just playbook runs.

The cost is staleness: a newly launched instance does not appear until the cache expires. So --flush-cache belongs in any run that must catch new hosts, and the timeout should reflect how fast your estate actually changes.

Note this is separate from fact caching — inventory caching stores which hosts exist, fact caching stores what they are.

Q. Your playbook says hosts: role_web-tier and matches nothing. Why?

Group names are sanitised — hyphens, dots and spaces become underscores. A tag value of web-tier produces the group role_web_tier, and ap-southeast-1a becomes ap_southeast_1a.

Always read the real names from ansible-inventory --graph rather than assuming what the sanitiser produced.


Part C · Other sources and composition

C1 · The other cloud plugins

yaml
# inventory/prod.azure_rm.yml
plugin: azure.azcollection.azure_rm
include_vm_resource_groups:
  - production-rg
auth_source: auto
keyed_groups:
  - key: tags.role
    prefix: role
yaml
# inventory/prod.gcp_compute.yml
plugin: google.cloud.gcp_compute
projects:
  - my-project-id
zones:
  - asia-southeast1-a
auth_kind: serviceaccount
keyed_groups:
  - key: labels.role
    prefix: role
yaml
# inventory/k8s.yml
plugin: kubernetes.core.k8s
connections:
  - kubeconfig: ~/.kube/config
    namespaces: [production]
The shape is identical across providersplugin:, a scoping option (regions, resource groups, projects), keyed_groups, compose, filters. Learn aws_ec2 properly and the others take an afternoon. The differences are terminology: AWS tags, GCP labels, Azure tags on differently-shaped objects.

C2 · The constructed plugin — groups from facts

keyed_groups works on data the cloud API returns. The constructed plugin works on facts and variables — so you can group by anything Ansible knows.

yaml
# inventory/02-constructed.yml   (numbered so it is parsed AFTER the cloud source)
---
plugin: constructed
strict: false

keyed_groups:
  - key: ansible_facts['distribution']
    prefix: os                       # -> os_Ubuntu, os_RedHat
  - key: ansible_facts['distribution_major_version']
    prefix: osver

groups:
  low_memory: "ansible_facts['memtotal_mb'] | default(0) | int < 2048"
  systemd_hosts: "ansible_facts['service_mgr'] | default('') == 'systemd'"

compose:
  is_production: "'env_production' in group_names"
constructed needs facts, and inventory is built before facts are gathered. So it only works with fact caching enabled (Module 08 Part C2) — the facts come from a previous run's cache.

On a first run against a new host there are no cached facts, so the constructed groups are empty. That is not a bug, it is the ordering, and knowing it saves an hour of confusion.

C3 · Merging several sources

bash
# Repeat -i
ansible-playbook site.yml \
  -i inventory/production.aws_ec2.yml \
  -i inventory/onprem.yml \
  -i inventory/02-constructed.yml

# Or point at a DIRECTORY - every file inside is parsed, alphabetically
ansible-playbook site.yml -i inventory/
plain text
inventory/
  01-production.aws_ec2.yml     <- cloud hosts
  02-onprem.yml                 <- static legacy hosts
  03-constructed.yml            <- groups derived from facts
  group_vars/
  host_vars/
Numeric filename prefixes control parse order, and that ordering matters: constructed must run after the sources whose hosts it is grouping. It is a small convention that prevents a genuinely confusing class of bug, and pointing -i at the directory then loads the adjacent group_vars and host_vars as well — the Module 06 layout, still working.

C4 · Terraform and Ansible

The analogy. Think of building a house and then furnishing it. The builder puts up the walls and keeps the plans — so if the house has to come down, they know exactly what was built and can undo it in the right order. The decorator paints the rooms and hangs the curtains. They need the house to already exist, and they keep no record whatsoever of the foundations.

Terraform is the builder, holding the plans as state. Ansible is the decorator, holding none. The two work best when the builder puts a number on the door and the decorator finds the house by that number — which in practice means Terraform writes the tags and Ansible discovers hosts by tag. Trying to make either one do the other's job is the mistake this section exists to prevent.

Diagram source
flowchart LR
    A["Terraform<br>creates infrastructure<br>and TAGS it"] --> B["AWS<br>the source of truth"]
    B --> C["Ansible aws_ec2 plugin<br>queries by tag"]
    C --> D["Ansible configures<br>what it found"]
    E["❌ Terraform local-exec<br>calling ansible-playbook"] -.->|"avoid"| D
    style A fill:#DDD6FE,stroke:#7C3AED
    style C fill:#D1FAE5,stroke:#059669,stroke-width:2px
    style E fill:#FEE2E2,stroke:#DC2626
PatternAssessment
Dynamic inventory by tag⭐ The right answer. Terraform tags; Ansible queries the cloud API. Fully decoupled — either can run alone
terraform output -json → inventoryWorkable. Couples Ansible to Terraform's outputs and state location
cloud.terraform.terraform_state pluginReads the state file directly as an inventory source. Neat, but needs state access
local-exec calling ansible-playbook❌ Avoid. Terraform cannot see whether config succeeded, cannot re-run it, and a failed provisioner taints the resource
The answer that shows you understand the boundary: "Terraform owns the lifecycle of resources and records them in state. Ansible owns what is installed and configured on them and holds no state. I keep them decoupled by having Terraform apply tags and Ansible discover hosts by tag through dynamic inventory — so I can re-run configuration without touching infrastructure, and vice versa. I avoid local-exec provisioners because Terraform then owns a step it cannot observe, retry, or roll back."
🧪 Exercise C4.1 — Compose a cloud source with a constructed source
bash
mkdir -p inventory
cat > inventory/01-prod.aws_ec2.yml <<'EOF'
plugin: amazon.aws.aws_ec2
regions: [ap-southeast-1]
filters:
  instance-state-name: running
hostnames: [tag:Name, private-ip-address]
keyed_groups:
  - key: tags.Role
    prefix: role
    default_value: untagged
compose:
  ansible_host: private_ip_address
EOF

cat > inventory/02-constructed.yml <<'EOF'
plugin: constructed
strict: false
groups:
  needs_attention: "'role_untagged' in group_names"
keyed_groups:
  - key: ansible_facts['distribution']
    prefix: os
EOF

ansible-inventory -i inventory/ --graph
ansible -i inventory/ needs_attention --list-hosts
Expected result — click to reveal

First run, no cached facts:

plain text
@all:
  |--@aws_ec2:
  |  |--web01
  |  |--web02
  |  |--i-0f9e8d7c6b5a4
  |--@role_web:
  |  |--web01
  |  |--web02
  |--@role_untagged:
  |  |--i-0f9e8d7c6b5a4
  |--@needs_attention:
  |  |--i-0f9e8d7c6b5a4
  |--@ungrouped:

Note there are no os_* groupsconstructed needed ansible_facts and none were cached. Enable fact caching, run a playbook once, then re-run --graph and os_Ubuntu appears.

Two genuinely useful things happened here.

default_value: untagged caught an instance with no Role tag and put it in role_untagged. Without it, that instance would be in no role group at all and would silently miss every hosts: role_* play — invisible drift.

The needs_attention group turns that into a report. ansible -i inventory/ needs_attention --list-hosts is now a one-command audit of "which instances are not tagged properly", which you can run in CI and fail the build on.

Notice also the untagged instance is named i-0f9e8d7c6b5a4 — the hostnames fallback to instance ID, because it had no Name tag either. Untagged infrastructure is visible and ugly by design, which is the point.

🎯 Interview questions — Composition and Terraform

Q. What is the constructed inventory plugin and what is its prerequisite?

It creates groups and variables from facts and existing variables, rather than from cloud API data — so you can group by OS, memory size, service manager, or any computed expression.

Its prerequisite is fact caching, because inventory is built before facts are gathered. Without a cache the constructed groups are empty on a first run, which is ordering rather than a bug.

Parse order matters too: name the file so it is read after the sources whose hosts it groups.

Q. How do you combine cloud hosts with legacy on-premises hosts?

Multiple inventory sources — repeat -i, or point -i at a directory where every file is parsed. A dynamic aws_ec2.yml alongside a static onprem.yml, optionally with a constructed.yml unifying them into common groups.

Use numeric filename prefixes to control parse order, and keep group_vars/host_vars adjacent so the Module 06 layout still applies.

Q. How do Terraform and Ansible fit together?

Terraform owns resource lifecycle and holds state; Ansible owns configuration and holds none.

Keep them decoupled: Terraform applies tags, and Ansible discovers hosts by tag through dynamic inventory. Either can then run independently — reconfiguring does not touch infrastructure and vice versa.

Avoid local-exec provisioners calling ansible-playbook: Terraform then owns a step it cannot observe, retry or roll back, and a failed provisioner taints the resource.

Q. What does default_value on a keyed_group buy you?

It catches instances missing that tag and puts them in a named group instead of leaving them in none.

Without it, an untagged instance belongs to no role_* group and silently misses every play targeting one — invisible drift. With it, you can build a needs_attention group and turn "which instances are tagged wrongly?" into a single command you run in CI.


Part D · Putting it together

D1 · Production practice

HabitWhy
Filename must end aws_ec2.yml / azure_rm.yml / gcp_compute.ymlOtherwise the plugin declines the file and you get an empty inventory with no error
ansible-inventory --graph after every inventory changeThe only thing standing between you and a silent no-op run
Read group names from --graph, never guess themSanitisation turns web-tier into web_tier and t3.medium into t3_medium
compose: ansible_host: private_ip_addressWithout it Ansible tries to DNS-resolve the inventory label and fails
default_value on keyed_groups, plus a needs_attention groupTurns untagged instances from invisible drift into a one-command audit
IAM role on the control node; read-only Describe permissionsCredentials in the inventory file means credentials in Git
filters: instance-state-name: runningOtherwise terminated and stopped instances appear and every run reports unreachable
Inventory caching with a timeout matched to your churnDescribeInstances is slow and rate-limited on every ad-hoc command
--flush-cache on any run that must catch new hostsA cached inventory cannot see an instance launched five minutes ago
Numeric filename prefixes when merging sourcesconstructed must parse after the sources it groups
Terraform tags, Ansible queries by tag — never local-execKeeps the two decoupled and keeps Terraform from owning a step it cannot observe
Treat tagging as a contract and enforce it in CIYour tagging discipline is your inventory once keyed_groups is in use

D2 · Capstone exercise

Attempt this without looking anything up. It exercises plugin config, keyed_groups, compose, composition and auditing together.

Brief. Build a dynamic inventory for a mixed estate:

  1. Running EC2 instances in two regions, production only
  2. Readable host names, but connections must use the private IP
  3. Groups derived from Role, Environment and availability zone tags
  4. Instances missing a Role tag must be visible, not silently unmanaged
  5. Legacy on-premises hosts merged into the same inventory
  6. An OS-derived group, and an explanation of why it may be empty at first
  7. One command that reports every badly tagged instance, suitable for CI
Model answer — attempt it first, then click
yaml
# inventory/01-production.aws_ec2.yml     -- requirement 1
---
plugin: amazon.aws.aws_ec2
regions:
  - ap-southeast-1
  - ap-southeast-2
filters:
  instance-state-name: running
  tag:Environment: production

hostnames:                                # requirement 2 - the LABEL
  - tag:Name
  - private-ip-address

compose:                                  # requirement 2 - the ADDRESS
  ansible_host: private_ip_address
  ec2_id: instance_id

keyed_groups:                             # requirement 3
  - key: tags.Role
    prefix: role
    default_value: untagged               # requirement 4
  - key: tags.Environment
    prefix: env
  - key: placement.availability_zone
    prefix: az

strict: false
cache: true
cache_plugin: jsonfile
cache_connection: /tmp/aws_inv_cache
cache_timeout: 900
yaml
# inventory/02-onprem.yml                 -- requirement 5
---
all:
  children:
    onprem:
      hosts:
        legacy01:
          ansible_host: 192.168.10.11
        legacy02:
          ansible_host: 192.168.10.12
      vars:
        ansible_user: sysadmin
yaml
# inventory/03-constructed.yml            -- requirements 4, 6
---
plugin: constructed
strict: false

keyed_groups:
  - key: ansible_facts['distribution']    # requirement 6
    prefix: os

groups:
  needs_attention: "'role_untagged' in group_names"   # requirement 4
plain text
# ansible.cfg
[inventory]
enable_plugins = auto, yaml, ini, amazon.aws.aws_ec2, constructed
cache = True
cache_plugin = jsonfile
cache_connection = /tmp/ansible_inventory_cache
cache_timeout = 900

[defaults]
inventory        = ./inventory
gathering        = smart                  # requirement 6 - constructed needs cached facts
fact_caching     = jsonfile
fact_caching_connection = /tmp/ansible_facts
fact_caching_timeout    = 7200
bash
# Requirement 7 - the CI audit
ansible -i inventory/ needs_attention --list-hosts

The seven decisions:

  1. filters: instance-state-name: running — requirement 1. Without it, terminated and stopped instances appear and every run reports unreachable errors, which trains people to ignore red output.
  2. hostnames and compose: ansible_host are separate — requirement 2. Label versus address. Omit the compose line and Ansible tries to DNS-resolve web01.
  3. default_value: untagged — requirement 4, and the pivot of the whole answer. It converts an invisible gap into a named group.
  4. needs_attention in constructed, not in the AWS plugin — because it is derived from group membership, which only exists after the cloud source has been parsed. Hence the 03- prefix.
  5. Numeric filename prefixes — parse order. constructed must come last.
  6. gathering = smart plus fact caching — requirement 6. os_* groups need facts, and inventory is built before facts are gathered, so they come from the cache. On a genuinely first run they will be absent, and that is correct behaviour rather than a bug.
  7. inventory = ./inventory as a directory — every file parsed, plus adjacent group_vars/host_vars.

Verify:

bash
ansible-inventory --graph
ansible-inventory --graph --vars | head -40
ansible-inventory --host web01
ansible role_web --list-hosts
ansible needs_attention --list-hosts          # empty output = well-tagged estate
ansible-inventory --graph --flush-cache       # bypass the cache

And the CI gate:

bash
if [ "$(ansible -i inventory/ needs_attention --list-hosts | tail -n +2 | tr -d ' ')" ]; then
  echo "FAIL: untagged instances found"; exit 1
fi

D3 · Command reference — everything from this module

Commands from Module 09. ⭐ marks genuinely daily-use.

Inspecting a dynamic inventory

bash
ansible-inventory -i inventory/prod.aws_ec2.yml --graph            # ⭐ the group tree
ansible-inventory -i inventory/ --graph                            # ⭐ merged sources
ansible-inventory -i inventory/ --graph --vars                     # ⭐ tree plus variables
ansible-inventory -i inventory/ --list                             # full JSON
ansible-inventory -i inventory/ --list --output /tmp/inv.json      # ⭐ save for diffing
ansible-inventory -i inventory/ --host web01                       # ⭐ one host, resolved
ansible-inventory -i inventory/ --graph --flush-cache              # ⭐ bypass the cache
ansible -i inventory/ role_web --list-hosts                        # ⭐ does the pattern match?
ansible -i inventory/ needs_attention --list-hosts                 # ⭐ tagging audit

Debugging an empty inventory

bash
ansible-inventory -i inventory/x.yml --graph -vvvv 2>&1 | grep -i 'declined\|parse'   # ⭐
ls inventory/                                    # ⭐ does the filename end in aws_ec2.yml?
ansible-config dump | grep -i enable_plugins     # is the plugin enabled?
ansible-galaxy collection list | grep amazon     # ⭐ is the collection installed?
aws sts get-caller-identity                      # ⭐ are credentials working at all?
aws ec2 describe-instances --max-items 1         # do the permissions work?

Cloud credentials

bash
export AWS_PROFILE=production                    # ⭐ named profile
export AWS_REGION=ap-southeast-1
aws sts get-caller-identity                      # ⭐ confirm before blaming Ansible
az login && az account show                      # Azure
gcloud auth application-default login            # GCP

Inventory plugin config keys

yaml
plugin: amazon.aws.aws_ec2       # ⭐ required first line
regions: [ap-southeast-1]        # ⭐ scope
filters:                         # ⭐ narrow the query server-side
  instance-state-name: running
  tag:Environment: production
hostnames:                       # ⭐ the LABEL
  - tag:Name
  - private-ip-address
compose:                         # ⭐ derived host variables
  ansible_host: private_ip_address
keyed_groups:                    # ⭐ THE feature
  - key: tags.Role
    prefix: role
    default_value: untagged      # ⭐ catches untagged instances
groups:                          # conditional groups from an expression
  large: "instance_type.startswith('m5.4x')"
strict: false                    # do not fail on an undefined compose expression
cache: true                      # ⭐
cache_plugin: jsonfile
cache_timeout: 900
plain text
[inventory]
enable_plugins   = auto, yaml, ini, amazon.aws.aws_ec2, constructed   # ⭐
cache            = True
cache_plugin     = jsonfile
cache_connection = /tmp/ansible_inventory_cache
cache_timeout    = 900
The three-command check on any dynamic inventory, before you trust it with a playbook:
bash
ansible-inventory -i inventory/ --graph        # do the hosts and groups look right?
ansible-inventory -i inventory/ --host <one>   # is ansible_host set correctly?
ansible -i inventory/ <pattern> --list-hosts   # does my playbook's pattern match?

None of them connect to anything. Together they catch the empty-inventory trap, the missing ansible_host, and the sanitised group name — which between them account for most dynamic-inventory failures.


D4 · Official documentation

LinkCovers
Working with dynamic inventoryConcepts, plugins versus scripts, enabling plugins
Inventory pluginsHow plugins are discovered and configured
amazon.aws.aws_ec2 inventoryEvery option — filters, hostnames, keyed_groups, compose, caching
ansible.builtin.constructed inventoryBuilding groups from facts and variables
azure.azcollection.azure_rm inventoryThe Azure equivalent
google.cloud.gcp_compute inventoryThe GCP equivalent
ansible-inventory CLI--graph, --list, --host, --vars, --flush-cache
Developing inventory pluginsWriting your own, for a CMDB or internal API
cloud.terraform collectionReading Terraform state as an inventory source

D5 · Self-assessment

1. Why does a static inventory fail in the cloud, and what are the two failure modes?

It is a hand-maintained cache of reality that nothing invalidates.

Missing hosts go silently unpatched and unmonitored. Stale entries produce unreachable errors on every run until people learn to ignore red output — which is how a genuinely down host gets missed. Both are bad; the first is silent, the second corrodes trust in your output.

2. Your aws_ec2 inventory returns nothing and no error. First three checks?

The filename — must end aws_ec2.yml/aws_ec2.yaml, or the plugin declines it silently. Then -vvvv | grep declined to see which plugin refused it and why. Then credentials with aws sts get-caller-identity, region, and whether the filters match anything.

An empty inventory is a successful run, which in CI is a green pipeline that configured nothing.

3. What is keyed_groups and what is its organisational consequence?

It turns instance attributes and tags into Ansible groups — tags.Role=web becomes role_web — so hosts: role_web stays correct as an autoscaling group changes size, with nobody editing a file.

The consequence: your tagging discipline becomes your inventory. An untagged instance is an unmanaged one, and the tag becomes the contract between whoever provisions and whoever automates.

4. hostnames versus ansible_host — the distinction.

hostnames sets the inventory label shown in output and returned by inventory_hostname. ansible_host, set via compose, is the address Ansible connects to.

Readable output naming web01 while connecting to 10.0.1.15. Without the compose line, Ansible tries to DNS-resolve web01.

5. hosts: type_t3.medium matches nothing. Why?

Group names are sanitised — dots, hyphens and spaces become underscores. t3.medium produces type_t3_medium, and ap-southeast-1a produces az_ap_southeast_1a.

Read the real names from ansible-inventory --graph rather than assuming.

6. What does default_value on a keyed_group achieve?

Instances missing that tag land in a named group instead of none at all. Without it they belong to no role_* group and silently miss every play targeting one — invisible drift.

With it, you can build a needs_attention group and turn tagging compliance into a single command that fails a CI build.

7. What does constructed do, and why might its groups be empty?

It builds groups and variables from facts and existing variables rather than cloud API data.

Its groups are empty when there are no cached facts, because inventory is built before facts are gathered. It therefore requires fact caching, and on a genuinely first run against new hosts the groups will be absent — ordering, not a bug.

8. How do you merge cloud and on-premises hosts, and why do filenames matter?

Repeat -i, or point -i at a directory where every file is parsed. Numeric prefixes control parse order, which matters because constructed must run after the sources whose hosts it groups.

Pointing at the directory also loads the adjacent group_vars and host_vars.

9. Inventory caching versus fact caching?

Inventory caching stores which hosts exist; fact caching stores what those hosts are. They are configured separately.

Inventory caching matters because DescribeInstances is slow and rate-limited on every ad-hoc command. Both trade freshness for speed, and both need --flush-cache when correctness matters more.

10. How do Terraform and Ansible fit together, and what do you avoid?

Terraform owns resource lifecycle and holds state; Ansible owns configuration and holds none. Keep them decoupled: Terraform applies tags, Ansible discovers by tag through dynamic inventory, so either can run independently.

Avoid local-exec calling ansible-playbook — Terraform then owns a step it cannot observe, retry or roll back, and a failed provisioner taints the resource.

11. How should the inventory plugin authenticate, and with what permissions?

The standard boto3 chain — ideally an IAM role on the control node, or OIDC federation in CI, or a named profile locally. Never keys in the inventory file.

Read-only permissions: ec2:DescribeInstances, ec2:DescribeTags, ec2:DescribeRegions. An inventory has no reason to hold write access.


Next — Module 10 · Custom Modules & Plugins.

When no existing module fits, you write one. Module 10 covers AnsibleModule, check-mode support, custom filter and lookup plugins, and — importantly — when not to write a module.

📚 Sources for the interview questions

Behaviour verified against the current dynamic inventory guide and the amazon.aws.aws_ec2 inventory plugin documentation.

Question selection cross-referenced against publicly published 2026 Ansible interview question sets:

Answers were rewritten and deepened rather than reproduced — published versions are usually correct but shallow, and the added operational detail is what differentiates a candidate in the room.

Spotted a mistake or want something added? Send me a note.