Module 13 — Troubleshooting & Debugging

Updated 20 August 2026

Module 13 · Troubleshooting & Debugging

A systematic method for diagnosing Ansible failures, rather than guessing. Everything here is the accumulated "why is this broken" from the previous twelve modules, organised into a procedure you can follow under pressure.

🧠 concept → 🧪 exercise → ✅ expected result (hidden) → 🎯 interview questions (answers hidden)

Prerequisite: Modules 01–12.


Part A · Classify the failure first

A1 · The four categories

The analogy. Think of a doctor before running any tests at all. The first thing they ask is where it hurts — because a headache and a broken ankle do not get the same first move, and starting the wrong examination wastes the whole appointment. The question sounds far too simple to be useful, and it is the single thing that saves the most time.

Ansible failures come in four kinds, and each has a completely different first move. Debugging your playbook logic when the error says UNREACHABLE is examining the ankle when the pain is in the head — your module never even ran, so nothing in your YAML can possibly be the cause. Classify first, investigate second.

Diagram source
flowchart TD
    A["Something failed"] --> B{"What does the<br>output actually say?"}
    B -->|"ERROR! before any PLAY"| C["PARSE ERROR<br>YAML or playbook structure"]
    B -->|"UNREACHABLE!"| D["CONNECTION<br>the module never ran"]
    B -->|"FAILED! with a module msg"| E["EXECUTION<br>the module ran and objected"]
    B -->|"Runs fine, wrong result"| F["LOGIC<br>variables, conditionals, ordering"]
    C --> C1["--syntax-check<br>yamllint<br>read the ^ here marker"]
    D --> D1["Leave Ansible.<br>Reproduce with ssh -vvv"]
    E --> E1["Read the msg field.<br>Run the command by hand<br>on the target"]
    F --> F1["debug: var=...<br>type_debug<br>--list-tasks, --start-at-task"]
    style D fill:#FEE2E2,stroke:#DC2626
    style F fill:#FEF3C7,stroke:#D97706
Classify before you investigate. The single most common waste of time is debugging a playbook when the failure is UNREACHABLE — which means the module never executed, so nothing in your playbook can be the cause.

Saying "first I work out which of the four categories it is, because each has a different first move" is a far better interview answer than listing flags.

CategorySignatureFirst move
ParseERROR! before any PLAY [...] header appears--syntax-check, read the ^ here marker, look above it
ConnectionUNREACHABLE!Stop using Ansible. ssh -vvv user@host
ExecutionFAILED! with a msg from the moduleRead msg, then run the equivalent by hand on the target
LogicGreen run, wrong outcomedebug: var=, type_debug, check precedence and ordering

A2 · Reading an error properly

The analogy. Think of arguing about a bill. The total is wrong, and staring at the total tells you absolutely nothing — the answer is always further down the receipt, in the itemised lines showing what was actually rung through. And very often the item is right while the quantity is blank, which is a data-entry problem rather than a pricing one.

In a failure object, msg is the total; cmd and invocation.module_args are the itemised lines. That is why a cmd reading /opt//deploy.sh — with the tell-tale double slash — means a variable rendered empty, not that a file is missing. Read the itemised lines first and most Ansible errors stop being mysterious.

plain text
fatal: [web01]: FAILED! => {
    "changed": false,
    "cmd": "/opt/deploy.sh",
    "msg": "[Errno 2] No such file or directory: b'/opt/deploy.sh'",
    "rc": 2,
    "stderr": "",
    "stdout": ""
}
FieldRead it for
msg⭐ The actual reason. Read this before anything else
rcExit code — command/shell only
stderr / stderr_lines⭐ What the command itself complained about
cmd⭐ The command after templating — often reveals an empty variable
invocation.module_args⭐ The resolved arguments. Where templating mistakes become visible
cmd and invocation.module_args are the fields people skip and should not. They show the values after Jinja2 rendering. A task that fails with "no such file" and a cmd of /opt//deploy.sh tells you instantly that a variable rendered empty — which is a variable problem, not a file problem.
🧪 Exercise A2.1 — Produce all four failure types deliberately
bash
# 1 - PARSE: missing space after the colon
printf -- '---\n- hosts: all\n  tasks:\n    - debug:\n        msg:broken\n' > f1.yml
ansible-playbook f1.yml 2>&1 | head -12

# 2 - CONNECTION: a host that does not exist
ansible-playbook -i 'nosuchhost.invalid,' -m ping all 2>&1 | head -6

# 3 - EXECUTION: a command that is not there
ansible localhost -m ansible.builtin.command -a "/opt/definitely-missing.sh" 2>&1 | head -8

# 4 - LOGIC: green run, wrong answer
cat > f4.yml <<'EOF'
---
- hosts: localhost
  gather_facts: false
  vars:
    should_run: "false"
  tasks:
    - name: This runs, and it should not
      ansible.builtin.debug: {msg: "I ran"}
      when: should_run
EOF
ansible-playbook f4.yml
Expected result — and what each tells you — click to reveal

1 — Parse. No PLAY [...] header at all:

plain text
ERROR! We were unable to read either as JSON nor YAML...
Syntax Error while loading YAML.
  mapping values are not allowed in this context
The offending line appears to be:
        msg:broken
           ^ here

The run never started. Nothing about your logic is relevant. Note "but may be elsewhere" — YAML errors are often reported a line or two after the real mistake, so read upward from the marker.

2 — Connection:

plain text
nosuchhost.invalid | UNREACHABLE! => {
    "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname",
    "unreachable": true
}

"unreachable": true is the discriminator. The module never ran. Leave Ansible entirely and reproduce with ssh -vvv.

3 — Execution:

plain text
localhost | FAILED | rc=2 >>
[Errno 2] No such file or directory: b'/opt/definitely-missing.sh'

Connection and authentication worked. The module ran and objected. This is a target-state or arguments problem.

4 — Logic — the dangerous one:

plain text
TASK [This runs, and it should not] ***
ok: [localhost] => {"msg": "I ran"}

PLAY RECAP: localhost : ok=1  changed=0  failed=0

Exit code 0. Green. Completely wrong. should_run is the string "false", and every non-empty string is truthy — Module 02 Part D5.

Categories 1–3 announce themselves. Category 4 does not, which is why it needs debug: var= and type_debug rather than a flag, and why it is the category that reaches production.

🎯 Interview questions — Classifying failures

Q. Walk me through how you debug a failing playbook.

Classify first, because each category has a different first move.

ParseERROR! before any PLAY header: run --syntax-check, read the ^ here marker and look above it, since YAML errors are reported late.

ConnectionUNREACHABLE!: leave Ansible and reproduce with ssh -vvv; the module never ran, so the playbook cannot be the cause.

ExecutionFAILED! with a msg: read msg, stderr and cmd, then run the equivalent by hand on the target.

Logic — green run, wrong outcome: debug: var=, type_debug, and check variable precedence and task ordering.

Q. Which fields of a failure object do you read, in what order?

msg first — the actual reason. Then stderr/stderr_lines for what the command complained about. Then cmd and invocation.module_args, which show the values after templating.

That last pair is what people skip. A "no such file" error with cmd: /opt//deploy.sh is a variable that rendered empty — a variable problem wearing a file problem's clothes.

Q. Which failure category is most dangerous, and why?

Logic. The other three announce themselves with a red line and a non-zero exit. A logic error produces a green run, exit code 0, and the wrong outcome — so CI passes and nobody investigates.

The canonical example is when: myvar firing because myvar is the string "false", which is truthy. It needs type_debug and debug: var=, not a verbosity flag.


Part B · The tools

B1 · The verbosity ladder

The analogy. Think of the zoom control on a map. The country view tells you the journey exists. Zoom in and you get the roads. Zoom in further and you get individual house numbers. Nobody navigates a city at country zoom, and nobody plans a road trip at house-number zoom — each level is right for a different question.

-vvv is the house-number level: the exact SSH command, the key it used, the user it connected as. It answers most connection questions on its own. It is also exactly why you do not run production jobs at that zoom, because at house-number level it prints every argument passed to every module — including the ones you encrypted in Module 07.

LevelAdds
-vFull task results, including for successful tasks
-vvTask inputs and which file each task came from
-vvvConnection detail — the exact SSH command, the key used, the temp paths
-vvvvConnection plugin internals, and why an inventory plugin declined a file
-vvv is the level worth remembering, because it answers "which key did it actually use, to which address, as which user" — the three questions behind most connection failures.
bash
ansible web01 -m ping -vvv 2>&1 | grep -o 'IdentityFile=[^]*'
ansible web01 -m ping -vvv 2>&1 | grep 'ESTABLISH SSH'

⚠️ Never run production playbooks at -vvv routinely — it prints full module arguments, which is exactly the leak Module 07 Part B3 warned about. no_log still protects, but non-secret-but-sensitive data is exposed.

B2 · The interactive debugger

The analogy. Think of a dish that fails at the very last step, after two hours of preparation. You do not want to throw the lot away and start again from the shopping. You want to stop right there, taste it, adjust the seasoning and try that one step again — with the two hours of prep still sitting on the counter, untouched.

That is strategy: debug and its r command: inspect, adjust, and redo the failed task without repeating everything that led up to it. What it does not do is edit your recipe book for you. Whatever you changed in the debugger to make the step work still has to be written back into the playbook afterwards, or the next run fails in exactly the same place.

yaml
- hosts: web
  strategy: debug              # drop into the debugger on ANY task failure
  tasks:
    - name: Risky task
      ansible.builtin.command: /opt/thing.sh
      debugger: on_failed       # or: always, never, on_skipped, on_unreachable
plain text
(debug) p task_vars['inventory_hostname']    # print a variable
(debug) p task.args                          # the task's arguments
(debug) p result._result                     # the full failure object
(debug) task.args['cmd'] = '/opt/correct.sh' # MODIFY the task
(debug) r                                    # redo the task with the change
(debug) c                                    # continue
(debug) q                                    # quit
The debugger's real value is r — redo. You can correct an argument or a variable in place and re-run just that task, without restarting a play that took ten minutes to reach the failure.

It is genuinely underused, and mentioning strategy: debug with task.args[...] = ... then r is a strong signal that you have debugged something painful rather than only read about debugging.

B3 · Inspecting state

yaml
- ansible.builtin.debug: var=myvar                        # ⭐ one variable
- ansible.builtin.debug: var=hostvars[inventory_hostname] # ⭐ EVERY variable this host has
- ansible.builtin.debug: msg="{{ myvar | type_debug }}"   # ⭐ the TYPE
- ansible.builtin.debug: var=ansible_facts                # all gathered facts
- ansible.builtin.debug: var=groups                       # the whole inventory
- ansible.builtin.debug: var=result                       # ⭐ a full registered object

- ansible.builtin.assert:                                 # ⭐ fail early, clearly
    that: [app_port | int > 1024]
    fail_msg: "app_port is {{ app_port }} ({{ app_port | type_debug }})"

- ansible.builtin.pause:                                  # stop and look around
    prompt: "Check the host now, then press enter"
bash
ansible-playbook site.yml --start-at-task "Deploy config"   # ⭐ resume after a fix
ansible-playbook site.yml --step                            # ⭐ confirm each task
ansible-playbook site.yml --list-tasks                      # ⭐ what runs, in order
ansible-inventory -i inventory/ --host web01                # ⭐ resolved inventory vars
ansible-config dump --only-changed                          # ⭐ what config is live
ansible --version                                           # ⭐ which ansible.cfg won

B4 · Logging

plain text
[defaults]
log_path = /var/log/ansible.log        # every run appended, with timestamps
bash
ANSIBLE_LOG_PATH=/tmp/run.log ansible-playbook site.yml
grep -n 'FAILED\|UNREACHABLE' /tmp/run.log
log_path writes the same content the terminal shows — including anything not protected by no_log. On a shared control node that file needs restrictive permissions and rotation, or you have created a long-lived, world-readable record of every run. Useful, and a genuine liability if you set it and forget it.
🧪 Exercise B4.1 — Use the debugger to fix a failure in place
yaml
---
- name: Debugger demo
  hosts: localhost
  gather_facts: false
  strategy: debug
  vars:
    script_path: /opt/missing.sh
  tasks:
    - name: Slow setup you do not want to repeat
      ansible.builtin.pause: {seconds: 5}

    - name: This fails
      ansible.builtin.command: "{{ script_path }}"
      debugger: on_failed
plain text
(debug) p task.args
(debug) p result._result['msg']
(debug) task.args['_raw_params'] = '/bin/echo fixed'
(debug) r
(debug) c
Expected result — click to reveal
plain text
TASK [This fails] **************************************************
fatal: [localhost]: FAILED! => {"msg": "[Errno 2] No such file or directory: b'/opt/missing.sh'", "rc": 2}

[localhost] TASK: This fails (debug)>
plain text
(debug) p task.args
{'_raw_params': '/opt/missing.sh'}

(debug) p result._result['msg']
"[Errno 2] No such file or directory: b'/opt/missing.sh'"

(debug) task.args['_raw_params'] = '/bin/echo fixed'

(debug) r
changed: [localhost]

(debug) c

PLAY RECAP: localhost : ok=2  changed=1  failed=0

The five-second pause did not repeat. That is the whole point — on a real playbook that reached the failure after ten minutes of setup, r re-runs only the failed task with your correction.

p task.args showed the arguments after templating, which is where you see that script_path rendered to something wrong. Combined with p task_vars['script_path'] you can distinguish "the variable is wrong" from "the variable is right and the file is genuinely missing".

⚠️ The fix is not persisted. You have proved what the correct value is; you must still edit the playbook. The debugger is a diagnostic instrument, not a repair tool — and saying that distinction out loud is worth doing, because someone will ask whether r "fixes" anything.

🎯 Interview questions — Debugging tools

Q. What do the verbosity levels give you, and which matters most?

-v full results including successes, -vv task inputs and source files, -vvv connection detail — the exact SSH command, the key used, temp paths — and -vvvv connection-plugin internals, which is also where you see an inventory plugin declining a file.

-vvv is the one to remember, because it answers which key, to which address, as which user.

Caveat: it prints full module arguments, so it is not something to run routinely against production.

Q. What is strategy: debug and what makes it worth using?

An interactive debugger that opens on task failure. You can p task.args, p result._result, inspect task_vars, modify an argument or variable in place, and r to redo just that task.

The value is r: on a long playbook you correct and retry the failing task without repeating ten minutes of setup. Also available per task with debugger: on_failed.

The change is not persisted — it is a diagnostic instrument, not a repair.

Q. How do you resume a long playbook after fixing a failure?

--start-at-task "<exact task name>", which is one of the reasons every task needs a name and one of the reasons to prefer import_tasks over include_tasks — dynamic includes are invisible to it.

Also --limit @site.retry to target only the hosts that failed, and --step to walk through interactively.


Part C · The recurring failures

C1 · Connection failures

ErrorCause and fix
Permission denied (publickey)Wrong local $HOME (often from sudo), wrong ansible_user, key not in that user's authorized_keys, or ~/.ssh permissions too loose. Module 01 Part B2.5
Connection timed outHost down, wrong IP, security group or firewall, no route. Not an Ansible problem at all
Host key verification failedRebuilt host with a new host key. ssh-keygen -R <host>, then ssh-keyscan to re-seed
unix socket path too longControlPersist path overflow with long cloud hostnames. Shorten control_path_dir. Module 08 Part B3
sudo: a password is requiredEscalation, not connection — SSH already worked. Supply -K or fix sudoers
you must have a tty to run sudorequiretty enabled with pipelining on. Module 08 Part B2
Failed to import the required Python libraryMissing library on the control node for a cloud module, or on the target for a local one. Module 06 Part D1
bash
# The connection triage sequence
ssh -vvv [email protected]                                  # ⭐ ALWAYS first
ansible web01 -m ping -vvv 2>&1 | grep -o 'IdentityFile=[^]*'
ansible-inventory -i inventory/ --host web01               # ⭐ what is ansible_host?
ls -ld ~/.ssh && stat -c '%a %n' ~/.ssh/*
ssh web01 'ls -ld ~ ~/.ssh ~/.ssh/authorized_keys'         # ⭐ target-side permissions
sudo journalctl -u sshd -n 50                              # ⭐ ON THE TARGET - the real reason
sshd will never tell a remote party why it rejected them — that would leak information to an attacker. So Permission denied (publickey) is deliberately uninformative.

The real reason is in the target's /var/log/auth.log or journalctl -u sshd, which typically says something like Authentication refused: bad ownership or modes for directory /home/deploy/.ssh. Knowing to look there is what turns a thirty-minute guess into a thirty-second fix.

C2 · Variable and templating failures

The analogy. Think of a jar labelled sugar that is actually full of salt.

You measured correctly. You followed the recipe exactly. Everything you can see is right, and the dish is still wrong.

No amount of re-reading the recipe will find it. You have to taste what is in the jar.

type_debug is tasting it — and it explains most of these failures, because a number stored as text and the word "false" stored where a yes/no belongs both look completely correct written down.

SymptomCause
'myvar' is undefinedNot defined for this host, or include_vars was skipped by --tags. Module 03 Part D1
Value ignored despite being in group_varsA role's vars/main.yml (level 15) beats it. Module 05 Part B1
when: fires when the value is falseIt is the string "false" — truthy. Use | bool. Module 02 Part D5
<generator object do_map at 0x...> in a fileMissing | list after map/select. Module 04 Part B5
<built-in method keys of dict object>Dot notation on a key named keys/items/count. Use brackets. Module 02 Part A2
Dictionary keys disappear when overriddenHigher precedence replaces, not merges. Use combine(recursive=True). Module 02 Part D2
hostvars['db01'][...] undefineddb01's facts were never gathered this run. Module 02 Part B4
Template task always reports changedTimestamp, unsalted password_hash, or unsorted iteration. Module 04 Part D3
yaml
# The three-step variable diagnosis
- ansible.builtin.debug: var=hostvars[inventory_hostname]   # 1. what DOES this host have?
- ansible.builtin.debug: var=myvar                          # 2. what is the value?
- ansible.builtin.debug: msg="{{ myvar | type_debug }}"     # 3. what TYPE is it?
Step 3 resolves more cases than people expect. A string where a boolean was intended, a string where an integer was intended, a generator where a list was intended — all of them look correct in the YAML and all of them are one type_debug away from obvious.

C3 · Ordering and state failures

The analogy. Think of describing a room from a photograph taken this morning. Everything you say is accurate — accurate as of when the picture was taken. Then somebody moved the furniture at lunchtime, and your perfectly accurate description is now simply wrong. You were not careless; you were working from a snapshot and the world moved on after it was taken.

Every bug in this section is that photograph. Facts are gathered at the start of the play, handlers run at the end of it, and a fact cache may be holding yesterday's values entirely. Recognising the shape — "I am acting on information that was true earlier" — is worth far more than memorising the individual fixes, because it generalises to bugs this page does not list.

SymptomCause
Verification fails right after deploying configThe handler has not run yet. meta: flush_handlers. Module 01 Part D3
Custom fact undefined after deploying itFacts were gathered at play start. Re-run setup. Module 02 Part B2
--tags x runs and nothing happensTag on include_role did not reach inner tasks. Use apply:. Module 05 Part C3
Handler never runsNotifying task reported ok, an earlier task failed, or the notify name does not match
Rollout continues past a failureA rescue cleared the failure state. End it with fail. Module 03 Part C2
Stale values on a correct-looking hostFact cache or inventory cache. --flush-cache. Module 08 Part C2
Empty inventory, exit code 0Plugin filename convention. -vvvv \| grep declined. Module 09 Part A3
Notice the pattern across this whole table: Ansible captured state at one moment and you changed it at another. Handlers, facts, caches and tags are all the same shape of bug.

Recognising "this is a point-in-time snapshot problem" generalises far better than memorising seven individual fixes, and it is a much stronger thing to say than naming flush_handlers.

🧪 Exercise C3.1 — Diagnose three failures without changing the playbook
yaml
---
- name: Three planted bugs
  hosts: localhost
  gather_facts: false
  vars:
    deploy_enabled: "false"
    app_config:
      host: 0.0.0.0
      port: 8080
    override:
      port: 9090
    users:
      - {name: alice, active: true}
      - {name: bob, active: false}
  tasks:
    - name: BUG 1 - should be skipped
      ansible.builtin.debug: {msg: "deploying"}
      when: deploy_enabled

    - name: BUG 2 - loses the host key
      ansible.builtin.debug:
        var: app_config | combine(override)

    - name: BUG 3 - renders a generator
      ansible.builtin.debug:
        msg: "{{ users | selectattr('active') | map(attribute='name') }}"

Diagnose each with debug/type_debug before reading the answers.

Expected result and diagnosis — click to reveal
plain text
TASK [BUG 1 - should be skipped] ***
ok: [localhost] => {"msg": "deploying"}          <-- ran anyway

TASK [BUG 2 - loses the host key] ***
ok: [localhost] => {"app_config | combine(override)": {"host": "0.0.0.0", "port": 9090}}

TASK [BUG 3 - renders a generator] ***
ok: [localhost] => {"msg": "<generator object do_map at 0x7f3a1c4d5e40>"}

Bug 1 — diagnose with type_debug:

yaml
- ansible.builtin.debug: msg="{{ deploy_enabled | type_debug }}"   # -> "str"

The string "false" is truthy. Fix: when: deploy_enabled | bool.

Bug 2 — this one is subtle, because the output looks right. host survived here since override has only port at the top level. Add a nested dictionary to app_config and the nested level is replaced wholesale. Fix: combine(override, recursive=True).

The diagnosis technique is to compare shapes:

yaml
- ansible.builtin.debug: var=app_config
- ansible.builtin.debug: var=app_config | combine(override, recursive=True)

Bug 3 — visible immediately in the output. map is lazy. Fix: append | list.

The general lesson: two of the three produced a green run. Only bug 3 announced itself, and it did so by writing a Python repr where a value should be. Bugs 1 and 2 would have shipped — which is why type_debug and shape comparison belong in your reflexes rather than your notes.

🎯 Interview questions — Common failures

Q. Permission denied (publickey) — walk me through it.

Reproduce with ssh -vvv first, because if plain SSH fails, Ansible cannot possibly succeed.

Then: is $HOME what you think — did someone run sudo ansible-playbook and send it looking in /root/.ssh? Is ansible_user right? Is the public key in that user's authorized_keys? Are permissions correct — ~ no looser than 755, ~/.ssh 700, authorized_keys 600?

And the step most people miss: sshd deliberately will not say why it refused, so read journalctl -u sshd on the target, which typically states bad ownership or modes for directory.

Q. A variable set in group_vars is being ignored. Why?

Most likely a role's vars/main.yml at precedence level 15 beating inventory at level 6 — a role design bug, since anything user-facing belongs in defaults/.

Other candidates: the group_vars filename does not exactly match the group name (silently ignored), it is defined for a different host, or an include_vars was skipped because of --tags without tags: always.

Diagnose with ansible-inventory --host <name> for the inventory layer and debug: var=hostvars[inventory_hostname] for the runtime picture.

Q. Several ordering bugs share one shape. What is it?

Ansible captured state at one point in time and something changed it afterwards. Handlers run at the end of the play, so a verification task tests the old config. Facts are gathered at play start, so a custom fact deployed mid-play is undefined. Fact and inventory caches hold yesterday's values.

Recognising the shape generalises; memorising flush_handlers, re-running setup and --flush-cache individually does not.

Q. A playbook succeeds but produces the wrong result. Where do you start?

debug: var=hostvars[inventory_hostname] to see everything the host actually has, then debug: var=<the variable>, then {{ var | type_debug }}.

Type is the answer more often than value — a string "false" that is truthy, a string "80" compared numerically, a generator where a list was expected. All three look correct in the YAML.

Then check precedence with ansible-inventory --host, and ordering with --list-tasks.


Part D · Putting it together

D1 · Production practice

HabitWhy
Classify the failure before investigating itDebugging a playbook when the error is UNREACHABLE is wasted time by definition
Reproduce connection failures with plain ssh -vvv firstThe native client gives a far clearer error than Ansible relays
Read msg, then stderr, then cmd and invocation.module_argsThe last two show values after templating — where empty variables become visible
Check journalctl -u sshd on the target for auth failuressshd will not tell the client why; the real reason is only in its own log
type_debug before assuming a value is wrongType is the cause more often than value
assert early with the value and its type in fail_msgTurns a future debugging session into a message
Name every task, and prefer import_tasks--start-at-task needs names, and cannot see inside dynamic includes
strategy: debug on long playbooks you are actively developingr retries the failed task without repeating the setup
Never run production at -vvv routinelyIt prints full module arguments
log_path with restrictive permissions and rotation, or not at allOtherwise it becomes a long-lived readable record of every run
Suspect caches whenever a correct-looking playbook uses stale valuesFact cache and inventory cache both fail this way, silently

D2 · Capstone exercise

A diagnostic exercise rather than a build. Work through it as though it were an incident.

Scenario. A colleague reports: "The nightly patching job used to work. Since Tuesday it reports success but the servers are not being patched. And two new servers are not being touched at all."

Produce your diagnostic sequence, and name the most likely cause of each symptom.

Model answer — attempt it first, then click

Step 1 — separate the two symptoms. They are almost certainly different faults. "Reports success but does nothing" and "two hosts untouched" have different causes, and treating them as one is how incidents get long.

Step 2 — the untouched hosts: is it an inventory problem?

bash
ansible-inventory -i inventory/ --graph | grep -c .
ansible -i inventory/ patch_targets --list-hosts        # are the new hosts in the pattern?
ansible-inventory -i inventory/ --host newhost01        # does it resolve at all?
ansible-inventory -i inventory/ --graph --flush-cache   # ⭐ is it a stale inventory cache?

Most likely causes, in order: a stale inventory cache (Module 09 Part B5); the new hosts are missing the tag that keyed_groups uses, so they landed in no role_* group (Module 09 — and default_value would have made this visible); or an inventory plugin filename problem producing a partial list.

Step 3 — "reports success but does nothing": is it tags?

bash
ansible-playbook patch.yml --list-tasks                 # ⭐ what would actually run?
ansible-playbook patch.yml --tags patch --list-tasks    # ⭐ compare the two

The prime suspect is a tagged run where the tag no longer reaches the tasks — someone converted an import_role to an include_role and the tag now stops at the include, so the play runs, reports ok=1, and executes nothing (Module 05 Part C3). --list-tasks with and without the tag shows this immediately.

Step 4 — is it a conditional that silently became true or false?

yaml
- ansible.builtin.debug: var=hostvars[inventory_hostname]
- ansible.builtin.debug: msg="{{ patching_enabled | type_debug }}"

A safety flag passed as -e patching_enabled=false arrives as the string "false" and is truthy; conversely a when: patching_enabled | bool against an undefined variable skips everything. Either produces a green run that does nothing (Module 02 Part D5).

Step 5 — is it a rescue swallowing failures?

bash
grep -n 'rescue\|ignore_errors\|failed_when' patch.yml roles/*/tasks/*.yml

A rescue block that logs and does not re-raise turns real failures into rescued=1, failed=0 and a zero exit code (Module 03 Part C2). Check the recap line in the job history: rescued and ignored counts are the tell.

Step 6 — what changed on Tuesday?

bash
git log --since="last Tuesday" --oneline -- patch.yml roles/ inventory/
git log --since="last Tuesday" -p -- requirements.yml     # ⭐ a collection version bump

The question that solves most incidents. And requirements.yml deserves its own look — an unpinned collection upgrading underneath you changes behaviour with no commit to your playbooks at all (Module 06 Part C1).

Step 7 — confirm the fix before trusting it.

bash
ansible-playbook patch.yml --check --diff --limit newhost01
ansible-playbook patch.yml --list-tasks --tags patch

The reasoning being assessed here is not the commands. It is: separate the symptoms, check inventory before logic, use --list-tasks to establish what would run before asking why it did not work, distrust caches, and always ask what changed. A candidate who reaches for -vvv first is guessing.


D3 · Command reference — everything from this module

Commands from Module 13. ⭐ marks genuinely daily-use.

Triage

bash
ansible --version                                    # ⭐ which ansible.cfg won
ansible-config dump --only-changed                   # ⭐ what config is live, and its source
ansible-playbook site.yml --syntax-check             # ⭐ parse only
ansible-playbook site.yml --list-tasks               # ⭐ what would run, in order
ansible-playbook site.yml --list-tasks --tags patch  # ⭐ what the TAG would run
ansible-playbook site.yml --list-hosts               # ⭐ blast radius
ansible-inventory -i inventory/ --graph              # ⭐ groups and membership
ansible-inventory -i inventory/ --host web01         # ⭐ resolved inventory vars
ansible-inventory -i inventory/ --graph --flush-cache  # ⭐ bypass a stale cache

Connection debugging

bash
ssh -vvv [email protected]                            # ⭐ ALWAYS first
ansible web01 -m ping -vvv                           # ⭐
ansible web01 -m ping -vvv 2>&1 | grep -o 'IdentityFile=[^]*'   # ⭐ which key?
ansible web01 -m ping -vvv 2>&1 | grep 'ESTABLISH SSH'
ssh-keygen -R web01                                  # ⭐ clear a changed host key
ssh-keyscan -H web01 >> ~/.ssh/known_hosts
ssh web01 'ls -ld ~ ~/.ssh ~/.ssh/authorized_keys'   # ⭐ target-side permissions
ssh web01 'sudo journalctl -u sshd -n 50'            # ⭐ THE REAL REASON for auth failures
ansible web01 -m ansible.builtin.command -a "id" -b  # is become working?

Variable and logic debugging

yaml
- ansible.builtin.debug: var=myvar                          # ⭐
- ansible.builtin.debug: var=hostvars[inventory_hostname]   # ⭐ everything this host has
- ansible.builtin.debug: msg="{{ myvar | type_debug }}"     # ⭐ THE type check
- ansible.builtin.debug: var=result                         # ⭐ a full registered object
- ansible.builtin.debug: var=ansible_facts
- ansible.builtin.debug: var=groups
- ansible.builtin.debug: var=group_names

- ansible.builtin.assert:                                   # ⭐ fail early and clearly
    that: [app_port | int > 1024]
    fail_msg: "app_port={{ app_port }} type={{ app_port | type_debug }}"

- ansible.builtin.pause: {prompt: "Inspect the host, then continue"}

The debugger

yaml
strategy: debug              # ⭐ play level - debugger on any failure
debugger: on_failed          # ⭐ task level - also: always, never, on_unreachable
plain text
(debug) p task.args                       # ⭐ arguments AFTER templating
(debug) p task_vars['myvar']              # ⭐ a variable's value
(debug) p result._result                  # ⭐ the full failure object
(debug) task.args['_raw_params'] = '...'  # ⭐ modify in place
(debug) r                                 # ⭐ REDO just this task
(debug) c                                 # continue
(debug) q                                 # quit

Verbosity and logging

bash
ansible-playbook site.yml -v                 # results, including successes
ansible-playbook site.yml -vv                # + inputs and source files
ansible-playbook site.yml -vvv               # ⭐ + connection detail
ansible-playbook site.yml -vvvv              # + plugin internals, 'declined' messages
ANSIBLE_LOG_PATH=/tmp/run.log ansible-playbook site.yml    # ⭐
grep -n 'FAILED\|UNREACHABLE\|rescued' /tmp/run.log        # ⭐
ANSIBLE_CALLBACKS_ENABLED=profile_tasks,timer ansible-playbook site.yml   # ⭐ timing

Recovery and partial runs

bash
ansible-playbook site.yml --start-at-task "Deploy config"   # ⭐ resume after a fix
ansible-playbook site.yml --limit @site.retry               # ⭐ only failed hosts
ansible-playbook site.yml --step                            # ⭐ confirm each task
ansible-playbook site.yml --check --diff --limit web01      # ⭐ prove the fix safely
ansible-playbook site.yml --flush-cache                     # ⭐ discard cached facts

"What changed?"

bash
git log --since="last Tuesday" --oneline -- playbooks/ roles/ inventory/   # ⭐
git log -p -- requirements.yml                # ⭐ an unpinned collection bump
ansible-galaxy collection list                # ⭐ what versions are actually installed
git diff HEAD~1 -- group_vars/                # ⭐ a variable change
The five-command opening move on any Ansible incident:
bash
ansible --version                              # which config is live
ansible-inventory -i inventory/ --graph        # does it see the right hosts
ansible-playbook site.yml --list-tasks         # what would actually run
ansible <host> -m ping -vvv                    # can it connect, with which key
git log --since="<when it last worked>" --oneline

None of them change anything, and between them they cover configuration, inventory, scope, connectivity and recent change — which is where the cause almost always is.


D4 · Official documentation

LinkCovers
Playbook debuggerstrategy: debug, debugger:, every debugger command
Ansible FAQThe canonical answers to recurring errors
Error handling in playbooksfailed_when, block/rescue, any_errors_fatal
Connection methods and detailsSSH options, ControlPersist, connection variables
ansible.builtin.debugvar: versus msg:, verbosity thresholds
ansible.builtin.assertthat, fail_msg, success_msg
Configuration — log_pathPersistent run logging
Callback pluginsprofile_tasks, timer, output formatting

D5 · Self-assessment

1. Name the four failure categories and the first move for each.

Parse (ERROR! before any PLAY) → --syntax-check, read the ^ here marker and look above it.

Connection (UNREACHABLE!) → leave Ansible, ssh -vvv.

Execution (FAILED! with a msg) → read msg/stderr/cmd, then run it by hand on the target.

Logic (green run, wrong result) → debug: var=, type_debug, check precedence and ordering.

2. Why can a playbook never be the cause of an UNREACHABLE error?

Because the module never executed — the failure occurred before any of your code ran. It is network, DNS, firewall, authentication, or file permissions on ~/.ssh.

3. Which failure fields do you read, in order, and which are most overlooked?

msg, then stderr/stderr_lines, then cmd and invocation.module_args — the last two being the overlooked ones, because they show the values after templating. A cmd containing /opt//deploy.sh is an empty variable, not a missing file.

4. sshd says Permission denied (publickey) and nothing else. Where is the real reason?

In the target's own log — journalctl -u sshd or /var/log/auth.log — typically Authentication refused: bad ownership or modes for directory.

sshd deliberately will not tell the client why, because that would leak information to an attacker.

5. What single shape do the handler, custom-fact, cache and tag bugs share?

Ansible captured state at one point in time and something changed it afterwards. Handlers run at play end, facts are gathered at play start, caches hold previous values, and tags decide what runs before it runs.

Recognising the shape generalises; memorising the individual fixes does not.

6. What does type_debug solve that inspection of the YAML does not?

Type confusion, which is invisible in source. A string "false" that is truthy, a string "80" compared numerically, a lazy generator where a list was expected — all look correct written down and all behave wrongly.

7. What makes strategy: debug worth using, specifically?

The r command. You inspect with p task.args and p result._result, modify an argument or variable in place, and redo only the failed task — without repeating however long the playbook took to reach it. The change is diagnostic and not persisted.

8. A job reports success but changed nothing. Name three possible causes.

A tag on an include_role that never reached the inner tasks (ok=1 and nothing else). An empty inventory from a plugin filename problem — "no hosts matched" is a successful run. A rescue block that logged and did not re-raise, giving rescued=1, failed=0.

Also a when: on a string "false" or an undefined variable skipping everything.

9. What is the most valuable question in any Ansible incident?

"What changed?" git log --since=<when it last worked> across playbooks, roles and inventory — and separately git log -p -- requirements.yml, because an unpinned collection can upgrade underneath you and change behaviour with no commit to your own code.

10. Why not run production playbooks at -vvv as a habit?

It prints full module arguments. no_log still protects marked secrets, but sensitive-but-unmarked data is exposed to the terminal, the CI log and any log aggregation. Use it deliberately for diagnosis, not routinely.


Next — Module 14 · Full Interview Simulation.

The final module: rapid-fire conceptual rounds, scenario design questions, a broken-playbook debugging exercise, and system-design-style automation questions — assembled from everything in Modules 01–13.

📚 Sources for the interview questions

Behaviour verified against the current playbook debugger documentation and the Ansible FAQ.

Question selection cross-referenced against publicly published 2026 Ansible interview question sets:

Answers were rewritten and deepened rather than reproduced — published versions are usually correct but shallow, and the added operational detail is what differentiates a candidate in the room.

Spotted a mistake or want something added? Send me a note.