🛡️Module 14 — Error Handling and Defensive Scripting

step 1
verdict: 1
before
bash: line 1: TYPO_VAR: unbound variable
verdict: 127
pipefail verdict: 1

What to read out of it: -e stopped the parade at false — step 2 never printed, and the script's verdict is the failure's (Module 3's truthful-verdict goal, enforced for free). -u turned the silent empty expansion into a named error — unbound variable, with the variable's name — killing Module 2's whole bug family (and Module 9's ${var:-} defaults remain the sanctioned way to say "empty is fine here"). pipefail made false | true finally confess (compare Module 4's Exercise 4.8: same pipeline, verdict 0). Three demos, three defaults inverted.

The direct answer: set -u makes expansion of unset variables a fatal error (typos and missing config die loudly instead of expanding to ""); set -o pipefail makes a pipeline return the first non-zero stage's status instead of only the last stage's. They address different failure channels — -e alone sees only each command's verdict, and a pipeline "command" lies without pipefail; that difference is LinuxTeck's question.

Going deeper: -u coexists with intentional defaults via Module 9's operators ("${TIMEOUT:-30}" is not an unset expansion — the default satisfies it), which is exactly how scripts stay strict and configurable. pipefail changes the verdict, while PIPESTATUS still holds the per-stage detail when you need to know which stage.

The details that separate candidates: the grep-under-pipefail consequence — a no-match grep (exit 1, Module 3) now fails the pipeline, so "no matches is fine" must be said explicitly: grep pattern file || true; and knowing -u has a special-parameter exception ("$@" is safe when there are zero arguments) — the two corners where strictness meets legitimate emptiness.

The direct answer, as layers: strict mode at the top (set -euo pipefail); Module 9's ${var:?} guards for required inputs; explicit command || handle where a failure has a local answer (retry, fallback, skip); an ERR trap for located last words (trap '…$LINENO…' ERR, Part C); an EXIT trap for cleanup (Module 13); and meaningful exit codes out the bottom (Module 3).

Going deeper: the philosophy that organizes the layers — strict mode is the safety net, not the error handling: it guarantees unhandled failures stop the script; the || branches are the actual handling, written only where you have something real to do about the failure. Scripts that "handle" everything with || true have removed the net and installed nothing.

The details that separate candidates: the retry-with-backoff loop for transient failures (for i in 1 2 3; do cmd && break; sleep $((i*2)); done — Modules 6+3 composed) presented as selective handling; and naming the observability half — errors logged with timestamps and line numbers to stderr/file, because at 3 a.m. "the script stopped" is a fact but "line 141, exit 7, at 02:13:22" is a diagnosis.

false inside if: allowed
survived the && chain
substitution failure, handled locally
still alive at the end

What to read out of it: four failures, zero deaths — every one sat in a guard position. Line 2's survival is the one to memorize (false && true under -e: left-of-&& is examined, and the chain's overall failure is forgiven with it). The || echo on the substitution shows the sanctioned pattern: touch the verdict and you own it — which is what "handled" means. The closing set +e matters when experimenting interactively: strict mode in your login shell turns every typo into a closed terminal.

The direct answer: because it's not supposed to — -e exempts commands in tested positions (if/while conditions, &&/|| operands, negated commands), since a non-zero status there is the question's answer, not an error. if grep -q x file must be allowed to "fail" or conditionals could not exist under strict mode.

Going deeper: recite the exemption list from the manual (conditions, chain operands except the last, non-final pipeline stages sans pipefail, !) and the corollary that bites: everything inside the body of a function called from a condition is also exempt — if my_complex_check; then silences -e for the whole call tree of my_complex_check, which is why deep checks are better run as plain guarded commands than as if-conditions.

The details that separate candidates: the whole-call-tree fact (few candidates know -e's exemption is dynamic, not lexical); the local n=$(cmd) masking as -e's most common real-world bypass (with the split-declaration fix); and the mature framing — these edges are why some teams reject set -e in favor of explicit || die everywhere, and being able to argue both policies calmly is worth more than either policy.

step 1
error at line 5: grep -q missing /etc/hosts (exit 1)
verdict: 1

What to read out of it: the failing line's number, its text, and its verdict, on stderr, unprompted — this is Module 8's die() upgraded from "you must call me" to "I catch what you forgot." The 3 a.m. difference is total: without the trap, the log ends at "step 1" and the investigation starts from nothing; with it, the investigation starts finished. Add $(date) and the script's name ($0) to taste — the handler is just a bash string, and everything from twelve modules is available inside it.

lock acquired; working
second run refused: already locked

What to read out of it: the subshell (Module 12 — a stand-in for a second cron invocation) opened the same lock file and was bounced immediately — -n's fail-fast, which for cron overlap is exactly right (exit 0, quietly: "my twin is on duty" is not an error). Release the lock by closing your shell (or exec 9>&-) and the second taker would succeed. One exec + one flock line at the top of any cron script retires the entire overlapping-runs incident class.

The direct answer: append stderr at the invocation (./job.sh 2>> /var/log/job.err — Module 4), or self-log inside: exec 2>> "$LOGFILE" rewires the script's own stderr permanently (Module 12's exec-without-command), and an ERR trap adds structured lines — timestamp, $0, $LINENO, $BASH_COMMAND, $? — at each failure.

Going deeper: keep the two channels honest — results still to stdout, diagnostics to the (now file-backed) stderr — so capturing output remains clean; timestamp every line (a log() helper, Module 8) because "when" is half of every incident; and append (>>) always, with rotation left to logrotate rather than the script.

The details that separate candidates: exec 2> >(tee -a "$LOG" >&2) for log-and-still-see (process substitution, Module 12) — plus knowing when not to bother: cron already mails/captures output, and systemd journals it; integrating with the platform's logging beats reinventing it, and saying so is the senior answer.

The direct answer: flock — exec 9>/var/lock/job.lock; flock -n 9 || exit 0 at the top; the kernel holds the lock for the process's lifetime and releases on any exit, crash and SIGKILL included.

Going deeper: why hand-rolled lock files fail — delete-at-end skips on kill (stale lock, Module 13's E2), and check-then-create is a race (two runs both see "no lock" and both proceed); flock's acquisition is atomic and its release is kernel-guaranteed, closing both holes. The -n vs waiting choice: cron overlap wants fail-fast-exit-0; a queue-worker wrapper might prefer waiting with a timeout (-w 60).

The details that separate candidates: the one-liner self-locking wrapper from flock's man page (flock -n /var/lock/job.lock ./job.sh in the crontab itself — no script changes at all); putting locks on a local filesystem (flock over NFS is a different, murkier story); and the observability touch — logging "skipped: previous run active" so silent skips are countable, because a job that skips every night is an incident wearing camouflage.

The direct answer, as the six-line preamble: #!/bin/bash; set -euo pipefail; ERR trap logging timestamp + $0 + $LINENO + $BASH_COMMAND + $? to the job's log; : "${REQUIRED_VAR:?...}" per input; tmpdir=$(mktemp -d) immediately followed by the EXIT-trap cleanup; exec 9>lock; flock -n 9 || exit 0 if cron-driven. Then the work, with explicit || handle only where a failure has a local answer.

Going deeper: justify each line by the failure it intercepts (typo'd variable → -u; lying pipeline → pipefail; overlap → flock; mid-run kill → EXIT trap; mystery stop → ERR trap) — the reasoning is the answer; the lines are just its residue.

The details that separate candidates: naming strict mode's exemptions unprompted (and the local masking) as the holes that remain; ShellCheck in CI as the enforcement that keeps the preamble present next quarter; and the closing judgment that this is a template, stamped not retyped — teams that template the preamble have it everywhere, teams that don't have it somewhere.

Diagnosis. -u meeting a genuinely optional variable (E2 row 2): the script always read $PROXY_HOST bare, silently empty on proxyless hosts — invisible pre-strictness, fatal now. The strictness isn't wrong; the script never declared the variable optional.

Work the steps: find the bare expansions (grep -n 'PROXY_HOST' sync.sh); confirm the intent (empty = no proxy) with the owners.

Fix and prevention: declare the optionality: proxy="${PROXY_HOST:-}" once at the top, then use "$proxy" (and Module 9's :+ trick shines here: curl ${proxy:+--proxy "$proxy"} … — the flag exists only when the value does). The general migration rule: adopting -u is an audit of which variables are required (:?), defaulted (:-), or presence-flags (:+) — the errors are the audit finding you.

Diagnosis. The local masking (B1's second surprise): the substitution's failing verdict (curl's, or jq's) was overwritten by local's own success, so -e saw nothing; version became empty; and everything downstream built artifacts named like app-.tar.gz (Module 9 Ticket 1's cousin). ShellCheck flags exactly this as SC2155 — if it had been in CI, the ticket wouldn't exist.

Work the steps: reproduce: f(){ local v=$(false); echo "alive, v=[$v]"; }; f under set -e — alive. Then the split version — dead, as intended.

Fix and prevention: local version; version=$(curl -fsS "$META_URL" | jq -r .tag) — now the assignment carries the pipeline's verdict (with pipefail covering the jq stage) and -e fires. Add a [ -n "$version" ] || die "empty version" belt to the suspenders, and put ShellCheck in CI so the next masking never merges.

Diagnosis. E2 row 3 verbatim: grep's documented "searched fine, found nothing" (exit 1 — Module 3's model citizen) now fails the pipeline under pipefail, and "clean logs" became indistinguishable from "scan broke." The paging is the meaning change surfacing, not a malfunction.

Work the steps: confirm the code: on a no-ERROR night, grep ERROR app.log; echo $? → 1; distinguish from a real failure (missing file → 2 — the 1-vs-2 discipline from Module 3 C1).

Fix and prevention: state the intent: grep ERROR app.log || [ $? -eq 1 ] || exit 1 (tolerate exactly no-match, still fail on 2), or the simpler commented || true when "couldn't even search" is acceptable to lump in — teams choose; the comment is mandatory either way. Then fix the alert: page on the scan's own failure, report-not-page on clean nights. Clean nights are the goal, not an incident.

Diagnosis. The hand-rolled lock, both classic holes pre-drilled (Part C's interview answer): check-then-create races (two Saturday runs both pass [ -f ] before either creates), and delete-at-end leaks a stale lock on any kill (Module 13: -9 skips cleanup) — after which no renewal runs until a human notices the certificate expiry graph. The fix would trade an overlap incident for an outage generator.

Work the steps: demonstrate the race conceptually (two interleaved check-then-creates on a whiteboard beats any prose) and the stale-lock path with a kill -9 during a test run.

Fix and prevention: flock: exec 9>/var/lock/renew.lock; flock -n 9 || { log "renewal already active, skipping"; exit 0; } — atomic acquisition, kernel-guaranteed release, stale-proof by construction. Move the lock off /tmp (tmpfiles cleaners delete idle locks) to /var/lock. And log-and-count the skips (E1): a renewal that skips every run is the camouflage incident this module keeps warning about.