🛑Module 13 — Signals and Traps

after TERM: wait verdict 143
after KILL: wait verdict 137

What to read out of it: two dead sleepers, two different death certificates — 143 = 128+15 (TERM), 137 = 128+9 (KILL). This is Module 3's reserved-range table completing itself: any $? above 128 names its signal by subtraction. Operational reading practice: a service exiting 137 was either kill -9ed by a human or — far more often — by the kernel's OOM killer; a 143 died politely, probably at the hands of an orchestrator's ordinary shutdown. The number is the incident's first clue.

The direct answer: SIGTERM (15, kill's default) is a catchable request — the process can flush buffers, close connections, remove temp files, then exit. SIGKILL (9) is uncatchable kernel-level termination — instant, unconditional, and cleanup-free.

Going deeper: -9's true cost is everything a graceful path would have done: half-written files, un-flushed buffers, orphaned locks (Module 14's lock files outlive their owner!), un-removed temp dirs, dropped in-flight requests. The professional sequence: kill, wait a grace period, verify, escalate to -9 only for a process that ignored polite mail — which is literally what docker stop automates (TERM, 10 seconds, KILL — Module 12 Ticket 4's whole plot).

The details that separate candidates: the 128+N death certificates (137 vs 143 telling you which death occurred, OOM-kill included); knowing SIGKILL cannot be trapped by design (the system's one guaranteed eviction); and the honest edge that even -9 cannot kill a process stuck in uninterruptible kernel sleep (D state, usually dead storage underneath) — naming that marks real operational scars.

working in /tmp/tmp.tAiMj6TyLh
cleaning up /tmp/tmp.tAiMj6TyLh
script verdict: 1

What to read out of it (your random suffix differs): the script failed — verdict 1 — and the cleanup line printed anyway: EXIT fired on the failure path with no if-statements, no duplicated rm at every exit point. Verify with ls /tmp/tmp.tAiMj6TyLh (your suffix): No such file or directory — the directory really is gone. Single-line takeaway: acquire, trap, then work — in that order, always.

tick 1
caught TERM mid-work; finishing current step first
tick 2
tick 3
loop done
verdict: 0

What to read out of it (line order near the kill may vary slightly): the TERM arrived during tick 2's sleep — and the handler message appears only after that sleep finished (the delayed-trap fact, observed). Because this demo's handler only announces (no stop flag), the loop ran to completion and exited 0 — a deliberate simplification so you can see the delivery timing; the production worker above adds stop=1 so the loop condition performs the stopping. Un-trapped, the same kill would have printed nothing and left verdict 143. The difference between 143 and "drained cleanly" is four lines of trap.

The direct answer (the trap half — Module 14 owns the set-flags half): trap 'cleanup_commands' EXIT guarantees cleanup on every exit path; trap 'handler' TERM INT converts kill-requests into graceful shutdown; trap 'echo "failed at line $LINENO" >&2' ERR (with Module 14's set -e) turns silent failures into located ones.

Going deeper: the acquire-trap-work ordering; EXIT as the pseudo-signal that fires on every ending except SIGKILL; and the flag-not-work handler pattern (stop=1 in the trap, the loop does the stopping) — handlers should be tiny, because they interrupt your script at awkward moments.

The details that separate candidates: knowing traps are reset in subshells (a $( ) does not inherit your EXIT trap — Module 12's copies are shallower than they look); that trap -p prints current traps (debuggable state); and the SIGKILL honesty — trap-based cleanup is a strong promise, not an absolute one, so critical resources also get startup-time stale-sweeps.

The direct answer: Ctrl-C makes the terminal send SIGINT to the foreground job (the whole process group — every stage of a foreground pipeline gets it). Default action: terminate, exit code 130. Background jobs are not in the foreground group — Ctrl-C never touches them, and bash additionally starts them with SIGINT ignored (job control off), so even an INT trap inside won't fire.

Going deeper: this is why kill %1 or kill $pid (SIGTERM) is the way to stop a background job, and why worker scripts trap TERM rather than INT for programmatic shutdown; INT is the human foreground signal, TERM the automation signal — a division of labor worth stating in exactly those words.

The details that separate candidates: the process-group subtlety (Ctrl-C killing a pipeline kills all its stages at once — which is what makes interrupting a | b | c feel atomic); 130 = 128+2 as the fingerprint of a Ctrl-C'd run in CI logs; and Ctrl-Z as SIGTSTP (stop, resumable with fg/bg — Module 12's rescue sequence) rather than a kill — three keystrokes, three different signals, candidates who can name all three rarely get a second signals question.

The direct answer, as a sequence: confirm delivery target (ps -o pid,ppid,stat,comm — right PID? right process?); check state — Z means already dead, un-reaped (kill the parent's negligence, not the zombie; Module 12); D means uninterruptible I/O sleep (no signal will land — investigate the storage/NFS underneath); T means stopped (resume with SIGCONT, then re-evaluate). A live process ignoring TERM may trap it — escalate deliberately: kill -9, then perform the cleanup its skipped handlers would have done (locks, temp files, partial output).

Going deeper: name the delivery subtleties — permissions (you can signal only your own processes unless root), process groups (a negative PID targets a whole group — how orchestration stops a pipeline), and the difference between "signal sent" (kill exits 0) and "process acted on it" (nothing promises that).

The details that separate candidates: the D-state honesty (even -9 waits for the kernel — the fix is below the process, not a bigger signal); and closing the loop with prevention — a service that "always needs -9" has a broken TERM handler, and that is the bug to file. Interviewers ask this question to hear a system-shaped answer, not a bigger hammer.

Diagnosis. Cleanup-at-the-end (E2 row 2): by hand the happy path runs to the bottom; at night, the scheduler's timeout TERMs the long ETL mid-flight, and the bottom of the script is never reached. The leftovers are a calendar of every night the job overran.

Work the steps: correlate leftover timestamps with the scheduler's kill log; reproduce with Exercise 13.3's shape (TERM the script mid-loop, watch the tail never run).

Fix and prevention: promote the cleanup into trap 'rm -rf "$workdir"' EXIT registered at acquisition (13.2's pattern) — TERM now cleans too. Add a startup sweep for the existing debris (find /tmp -maxdepth 1 -name 'etl.*' -mtime +1 … — Module 11's armored pipeline) because past leaks don't fix themselves, and SIGKILL nights will still slip one through.

Diagnosis. No graceful shutdown anywhere: the app neither traps TERM nor drains; the kill defaults to instant-ish death mid-request, and sleeping after the death settles nothing — the requests are already gone. The sleep is cargo cult where a protocol belongs (D1's flowchart, unimplemented on the receiving side).

Work the steps: verify with the death certificate: app exit status 143, instantly, with connection errors clustered at deploy timestamps. Confirm the app has no TERM handler (its logs say nothing at shutdown).

Fix and prevention: implement the receiving half: trap TERM → stop accepting new work → finish in-flight items → deregister → exit 0 (the C1 worker shape, application-sized). Then delete the sleep and let the orchestrator's grace period do its real job: bounding the drain, not replacing it. Measure success as "zero dropped requests during a deploy," not "fewer complaints."

Diagnosis. Three sins compounding: -9 as first resort (no drain, hence the corruption — half-processed messages die with the workers); pattern-matching kills (-f worker matches any command line containing the substring — the monitoring agent's config path did); and killing on a schedule instead of on a condition (healthy workers die every five minutes too).

Work the steps: demonstrate the overmatch harmlessly: pgrep -af worker — read everything the pattern actually selects. Trace one corruption to a kill timestamp.

Fix and prevention: replace the cron hammer with the protocol: detect stuckness (no heartbeat/progress in N minutes — a Module 5 guard on a Module 11-parsed status), then TERM the specific PIDs (from a pidfile or supervisor, never a substring), grace, verify, escalate. And fix the root cause the hammer was hiding — the workers' actual hang. Every -9 in cron is an incident postponed and a root cause protected.

Diagnosis. 130 = 128 + 2 = SIGINT — something is Ctrl-C-ing (or programmatically INTing) the builds. Tests don't fail with 130 on their own merits; the code is the pattern the failures lack. Suspects: a CI timeout mechanism that sends INT, a runner's cancellation (a human or auto-cancel on new commits), or a wrapper's own trap.

Work the steps: read the certificates first (this module's core habit): confirm 130 across runs; correlate timestamps with pipeline cancellations, force-pushes, and the runner's timeout settings; check whether the CI system documents INT-then-KILL as its cancellation protocol (most do — it is D1's flowchart again).

Fix and prevention: if auto-cancel is the cause, that is a feature misread as flakiness — label such runs cancelled, not failed, in reporting. If timeouts, fix the slow tests or the limit. The durable lesson for the runbook: any exit code over 128 is a death certificate — subtract 128, name the signal, and go find the sender before blaming the deceased.