📣Module 04 — Signals, Sessions & Job Control

$ kill -l
 1) SIGHUP	 2) SIGINT	 3) SIGQUIT	 4) SIGILL	 5) SIGTRAP
 6) SIGABRT	 7) SIGBUS	 8) SIGFPE	 9) SIGKILL	10) SIGUSR1
11) SIGSEGV	12) SIGUSR2	13) SIGPIPE	14) SIGALRM	15) SIGTERM
16) SIGSTKFLT	17) SIGCHLD	18) SIGCONT	19) SIGSTOP	20) SIGTSTP
...

$ kill -l | wc -w
124

  1 = HUP
  2 = INT
  3 = QUIT
  9 = KILL
 15 = TERM
 17 = CHLD
 18 = CONT
 19 = STOP

What to read out of it.

wc -w counts 124 words, because kill -l prints a number and a name for each one — so that is 62 signals, numbered 1 to 64. Numbers 32 and 33 are reserved by the threading library and are not shown.

Of those, the first 31 are the standard signals. The rest are "real-time" signals that ordinary system administration almost never touches.

You already know two of these numbers from Module 02. When a container exits with 137, that is 128 + 9 — and 9 is right there in the list as SIGKILL. Exit code 143 is 128 + 15, which is SIGTERM. The list you are looking at is what turns those exit codes into a diagnosis.

Notice SIGCHLD at 17. That is the signal a parent gets when a child dies — the notification that a child is waiting to be reaped, from Module 02's Section C3. It was there all along.

SIGTERM -> exit code 143   (128 + 15)
SIGINT  -> exit code 130   (128 + 2)
SIGHUP  -> exit code 129   (128 + 1)
SIGQUIT -> exit code 131   (128 + 3)
SIGKILL -> exit code 137   (128 + 9)

(you will also see a "Terminated" / "Killed" job message from the shell between lines)

What to read out of it.

Every one of those exit codes is 128 + the signal number from the table in Exercise A1.1. The arithmetic works every time, because a process killed by a signal never chose an exit status — it never got the chance — so the shell reports what happened to it instead.

Notice the exercise signals a separate sleep, not the shell itself. That is deliberate: a shell that kills itself with SIGINT or SIGQUIT behaves differently — a non-interactive bash ignores SIGQUIT outright, and an interactive one abandons the loop on SIGINT. Signalling a victim keeps the arithmetic clean.

These five numbers are worth recognising instantly:

• 143 — a normal, polite shutdown. Usually fine.
• 137 — SIGKILL. In a container, nearly always the memory limit or an expired grace period.
• 130 — somebody pressed Ctrl-C.
• 129 — SIGHUP. A terminal disconnected, or a reload went wrong.
• 131 — SIGQUIT, which also leaves a core dump behind if core dumps are enabled.

default  -> still running? no
ignore   -> still running? yes
[handler ran]
catch    -> still running? yes

What to read out of it.

Same signal, sent the same way, three completely different outcomes. Nothing about the sending changed — only what the receiver had arranged in advance.

The third case is the one worth staring at. The handler ran, printed its message, and the process kept running. Catching a signal does not mean the process stops. It means the process gets to decide, and here it decided to do nothing much.

This is exactly why a shutdown script cannot assume kill worked. Module 02 made that point; now you can see all three reasons it might not.

$ grep -E 'SigIgn|SigCgt|SigBlk' /proc/7801/status
SigBlk:	0000000000010000
SigIgn:	0000000000000004
SigCgt:	0000000000014002

$ kill -TERM $STUBBORN
caught TERM
   7801 S    bash

$ kill -STOP $STUBBORN
   7801 T    bash

$ kill -KILL $STUBBORN
gone

What to read out of it — and the first line is the most interesting.

SigIgn: 0000000000000004. That is a bitmask. Signal N occupies bit N−1, worth 2^(N−1) — so 0x4 is signal 3, SIGQUIT. (Bash ignores SIGQUIT in background jobs.) There is no bit set for signal 9 or signal 19.

You asked bash to ignore KILL and STOP. Bash accepted the commands without complaint. And the kernel simply did not record them, because it is not possible to record them. The mask shows only what actually took effect.

Then the behaviour follows exactly:

• TERM was caught and printed its message. The arrangement worked.
• STOP put the process into state T, the stopped state from Module 02, despite the trap.
• KILL removed it, despite the trap.

The bitmasks in /proc/PID/status are worth knowing about generally. SigIgn is what a process ignores, SigCgt is what it catches, SigBlk is what it has temporarily blocked.

To read one, remember signal N is bit N−1: SIGHUP (1) is 0x1, SIGQUIT (3) is 0x4, SIGTERM (15) is 0x4000, SIGCHLD (17) is 0x10000. So SigCgt: 0x14002 here is SIGCHLD + SIGTERM + SIGINT, and SigBlk: 0x10000 is bash blocking SIGCHLD while it waits for a child.

On a service that is not responding to shutdown, checking SigCgt tells you whether it has even installed a TERM handler. Exact values vary a little between interactive and script contexts — what matters is which bits are set, not matching these digits.

--- sending TERM while it is being ignored ---
SigPnd:	0000000000000000
SigIgn:	0000000000004006
SigCgt:	0000000000010000
   8102 S    bash

--- now compare with a D-state process ---
sent KILL while in D state; state right after: D

What to read out of it — the two halves show two different things.

First half. SigPnd is all zeros, and the process is still running. An ignored signal is not held pending — it is thrown away the moment it arrives. Ignoring is not deferring. The signal is gone and nothing will ever act on it.

That is an important distinction: ignored means discarded, blocked means deferred. Look at SigIgn and you can see which signals this process is discarding — 0x4004 is signal 3 (0x4, SIGQUIT) plus signal 15 (0x4000, SIGTERM).

Second half, and this is the one that matters operationally. A SIGKILL was sent to a process in D state, and immediately afterwards the process is still there, still in D.

The unrefusable signal did nothing, because the process was not at the boundary where signals get checked. It was inside the kernel waiting for the disk. When the write finished, it left D, hit the boundary, and only then did the KILL take effect.

On a healthy disk this lasts milliseconds and you would never notice. On a hung NFS mount it lasts forever, and that is the unkillable process from Module 02.

All three end a process by default, and the differences are about who is in control.

SIGTERM (15) is the polite request. It is what kill sends by default. The process can catch it and shut down cleanly — finish in-flight requests, flush buffers, close connections, deregister from a load balancer. It can also ignore it entirely.

SIGINT (2) is what Ctrl-C sends. Also catchable, but it comes from the terminal and goes to the whole foreground process group, not to one process. That group behaviour is the real difference from TERM.

SIGKILL (9) cannot be caught, blocked or ignored. The kernel removes the process without telling it. No cleanup of any kind.

The correct order is always TERM, wait, then KILL — which is exactly what systemctl stop does with TimeoutStopSec and Kubernetes does with terminationGracePeriodSeconds.

The details that separate candidates:

• Say why SIGKILL has to be uncatchable. There must be one thing a process cannot argue with, or a program could make itself genuinely unstoppable.
• Name what you lose with -9: unflushed writes, connections dropped mid-request, stale lock and PID files, temp files never cleaned up.
• Know the two cases where -9 still does nothing — a zombie (already dead, nothing to signal) and a D-state process (alive, but not at a point where signals are checked). The precise phrasing is that SIGKILL cannot be refused, but it can fail to be delivered.

No. SIGKILL and SIGSTOP cannot be caught, blocked or ignored. They are handled entirely by the kernel and the process is never informed.

But kill succeeding tells you far less than people assume. It means the signal was marked pending on the target, and nothing more. It does not mean the target noticed, acted, or stopped.

Delivery happens separately, when the kernel is about to return that process to user space. Until then the signal sits as a bit in the process's record.

The details that separate candidates:

• Explain D state properly. A process waiting on storage inside the kernel never reaches the point where pending signals are checked, so even SIGKILL sits there. If the storage never answers — dead disk, hung NFS — it waits forever. You can see the pending bit in SigPnd in /proc/PID/status.
• Distinguish ignored from blocked. An ignored signal is discarded on arrival and is gone. A blocked signal stays pending and is delivered once unblocked. SigIgn and SigBlk show which is which.
• Mention PID 1. For PID 1, signals with a default action of terminate are not delivered at all unless PID 1 installed a handler. This is why a container whose application never handles TERM cannot be stopped politely.
• Standard signals are not queued. Sending the same signal a hundred times to a busy process sets one bit; it may be acted on once. Any design that counts signals is broken.

Two completely different things, and which one applies depends on whether the process has a terminal.

Originally: "the terminal went away." When a session leader exits or a connection drops, the kernel sends SIGHUP to the foreground process group of that terminal. The default action is to terminate, which is why background jobs can die when you close a window. It is literally named after a phone line hanging up.

By convention: "reload your configuration." A daemon has no controlling terminal, so it can never receive HUP for the original reason. That left the signal free, and it was reused. nginx -s reload, systemctl reload for many units, and rsyslog all use HUP to re-read config without restarting and without dropping connections.

The details that separate candidates:

• Explain why one signal ended up with two jobs. Most candidates know both meanings; being able to say why they do not conflict — a daemon has no terminal, so the original meaning cannot reach it — turns trivia into understanding.
• Connect it to log rotation. From Module 03: renaming a log file leaves the daemon writing to a now-nameless inode. logrotate uses a postrotate script to send HUP (or USR1 for nginx) so the daemon reopens the file by name. Without it, the new log file stays empty and the old one keeps growing invisibly.
• Name the ways to survive it: nohup sets the disposition to ignore, disown removes the job from the shell's list so no HUP is sent, and setsid puts the process in a new session with no controlling terminal at all. Three different mechanisms for the same goal — Part D covers them.

$ ps -eo pid,pgid,comm --no-headers | grep -E 'sleep|cat'
   9101    9101 sleep
   9102    9101 cat
   9103    9101 cat

--- after killing just the sleep ---
   9102    9101 cat
   9103    9101 cat

killing whole group 9101
all gone

--- always look first ---
9210 sleep 200

What to read out of it.

Three processes, three different PIDs, and one shared PGID of 9101. The group ID is the PID of the first process in the pipeline — the group leader. That is where process groups come from, and Part C covers it properly.

Killing just the sleep left both cat processes running. This is the thing that catches people out: stopping the first stage of a pipeline does not stop the pipeline. The cat processes are still there waiting for input that will never arrive.

Using the negative PGID took all three out in one go. That minus sign is the whole difference between "this process" and "this job".

And pgrep -af showed exactly what pkill -f was about to hit, before it hit it.

# 1. Normal exit
working file is /tmp/demo.k3Xa9Q
sleeping - send me a signal
[EXIT handler] removing /tmp/demo.k3Xa9Q

# 2. TERM
working file is /tmp/demo.pR7mZt
sleeping - send me a signal
[TERM] shutting down cleanly
[EXIT handler] removing /tmp/demo.pR7mZt
exit code: 143

# 3. KILL
working file is /tmp/demo.9Lw2Vc
sleeping - send me a signal
after SIGKILL, does /tmp/demo.9Lw2Vc still exist?
-rw------- 1 zaeem zaeem 0 Aug 20 14:02 /tmp/demo.9Lw2Vc

What to read out of it — the three cases are the whole lesson.

Normal exit: the EXIT handler ran and the file was removed. Nothing was signalled at all.

TERM: both handlers ran, in order. The TERM handler printed its message and called exit 143, and that exit then triggered the EXIT handler. This is the pattern to copy: signal handlers do the signal-specific part, and EXIT does the cleanup, so cleanup is never duplicated and never missed.

The exit code is 143, which is 128 + 15 from Module 02. It was chosen deliberately in the handler, so the process reports the same code it would have had if it had simply been killed. That matters — a supervisor reading exit codes should not see a different number just because you handled the signal nicely.

KILL: no handler ran, and the temporary file is still there. Not because the script is badly written, but because nothing runs after SIGKILL. This is the concrete cost of kill -9, in one line of output.

--- what does the NEW program (sleep) have? ---
SigIgn:	0000000000000004
SigCgt:	0000000000000000

TERM worked - handler did not survive

--- nohup'd process: what is it ignoring? ---
SigIgn:	0000000000000001
   9455 S    sleep
survived SIGHUP

What to read out of it.

Read the two bitmasks from the first block:

• SigCgt: 0000000000000000 — the new program catches nothing. The TERM handler that was installed a moment earlier is gone. exec replaced the program, and the handler function went with it.
• SigIgn: 0000000000000004 — bit 3 is set, which is signal 3, SIGQUIT. The ignore survived, exactly as the table says.

Then TERM killed the process, proving the handler really was reset rather than merely invisible.

The nohup block shows the same mechanism put to work. SigIgn: 0000000000000001 is bit 1 — SIGHUP. nohup did nothing clever; it set HUP to ignored and execd sleep. The ignore survived, and sleep is now immune to a signal it has never heard of.

started as PID 9601
>>> sending TERM
[TERM] no new work; finishing 1 in-flight
[shutdown] draining for 2 seconds
[shutdown] done
[EXIT] final cleanup
final exit code: 143
ls: cannot access '/tmp/graceful.lock': No such file or directory

What to read out of it — the order of those lines is the whole pattern.

The TERM handler did not exit. It set a flag. The loop noticed the flag, finished the unit of work it was in the middle of, and only then left the loop. Nothing was cut off halfway.

Then the drain — a deliberate pause before exiting. In a real service that is the window in which the load balancer stops sending you traffic.

Then the EXIT handler removed the lock file, on the way out, exactly as in Section B2.

And the exit code is 143, the same code the process would have reported if it had simply been killed by TERM. A supervisor reading exit codes sees a normal shutdown, not a mysterious custom number.

Compare this with a handler that just calls exit immediately: the in-flight work would have been abandoned, the drain would not have happened, and the lock file would still be there if EXIT were not also trapped.

Catch SIGTERM and shut down in this order:

1. Stop accepting new work. Close the listening socket, or fail the readiness probe.
2. Finish requests already in progress. Do not abandon them.
3. Drain. Keep serving for a few seconds while the load balancer or service mesh notices you are going away.
4. Flush and close. Buffers to disk, database connections, open files.
5. Exit with a sensible status — conventionally 128 + 15 = 143.

The supervisor sends TERM, waits, then sends KILL. That wait is TimeoutStopSec in systemd, terminationGracePeriodSeconds in Kubernetes (default 30s), and docker stop -t (default 10s).

The details that separate candidates:

• Point 3 is the one most people miss. In Kubernetes, SIGTERM and endpoint removal happen in parallel. Your pod can still receive new requests for a second or two after TERM. Exiting immediately drops them, on every deploy. A short sleep before shutting down is the standard fix, along with a preStop hook.
• The handler should set a flag, not do the work. A signal handler interrupts the program at an arbitrary point, so only a small set of operations is safe inside one. Set a flag, return, let the main loop shut down cleanly.
• Make the grace period longer than your drain. If they are equal you get SIGKILLed every time, and it looks like a crash.
• Check PID 1. If your entrypoint is a shell script that starts the app as a child, the shell gets TERM and the app never hears it. Use exec in the entrypoint so the app becomes PID 1 — the Module 02 and Module 03 mechanism, applied.

fork copies everything: default stays default, ignored stays ignored, and handlers are inherited, because the child is running the same program so the handler code still exists.

exec is different, and the asymmetry is the point of the question:

• Handlers are reset to default. The handler was a function in the old program's memory, and exec replaced that memory. There is nothing left to call.
• Ignored signals stay ignored. Ignoring is not code — it is a flag on the process — so there is nothing to lose.
• Default stays default.

The details that separate candidates:

• Give the one-line reason, not just the table: "a handler is code and exec throws the code away; ignore is a flag and flags survive."
• Name the practical use: this is exactly how nohup works. It sets SIGHUP to ignored, then execs your command. The ignore survives, so the program is immune to HUP without containing any code about it.
• Name the practical bug: a wrapper script that ignores a signal passes that on to everything it starts. A service that inherits an ignored SIGTERM cannot be stopped politely, and nothing in its own source explains why. grep SigIgn /proc/PID/status is how you find it.

Signal the process group, not the PID: kill -TERM -<PGID>. The leading minus sign means "every process in this group".

A pipeline like a | b | c is three processes sharing one process group, whose ID is the PID of the first process. Killing only the first leaves the others running and waiting for input that will never come.

Find the group with ps -o pgid= -p PID.

The details that separate candidates:

• Know why systemd does not use this. Process groups can be changed by the processes themselves, and a program that calls setsid escapes entirely. So systemd tracks every process of a unit in a cgroup and stops all of them together, which nothing can escape from the inside. That is a much stronger guarantee, and it is why systemctl stop reliably cleans up things a kill would miss.
• Be careful with pkill -f. It matches a substring of the full command line and will happily match more than you intended. Always run pgrep -af <pattern> first to see the targets before sending anything.
• Mention kill 0 and kill -1 as the two easy accidents: kill 0 signals your own process group including yourself, and kill -1 as root reaches nearly every process on the machine.

$ tty
/dev/pts/0

$ ls -l "$(tty)"
crw--w---- 1 zaeem tty 136, 0 Aug 20 14:30 /dev/pts/0

$ who
zaeem    pts/0        2026-08-20 14:12 (10.191.44.1)

$ echo "written straight to the terminal device" > "$(tty)"
written straight to the terminal device

$ stty -a | tr ';' '\n' | grep -E 'intr|quit|erase|susp'
intr = ^C
 quit = ^\
 erase = ^?
 werase = ^W
 susp = ^Z

$ stty -a | head -3
speed 38400 baud; rows 44; columns 178; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; ...
isig icanon iexten echo echoe echok -echonl -noflsh -tostop

What to read out of it.

ls -l shows crw--w----. The leading c means a character device, not a regular file. And it is owned by you — the kernel hands ownership of the pseudoterminal to whoever logged in, which is what stops other users writing to your screen.

echo > $(tty) printed to your screen. That is not a trick or a special case: your terminal genuinely is a file, and writing to it is an ordinary write call to a descriptor. Everything from Module 03 applies.

Now the stty output, which is the mapping table for the red buttons:

• intr = ^C — Ctrl-C produces SIGINT
• quit = ^\ — Ctrl- produces SIGQUIT
• susp = ^Z — Ctrl-Z produces SIGTSTP

These are settings, not laws. stty intr ^X would move interrupt to Ctrl-X, and Ctrl-C would then be an ordinary character.

The second command shows the mode flags, which live on their own line rather than in the control-character list. isig is what makes those keys produce signals at all — turn it off and Ctrl-C becomes an ordinary character. icanon means line mode is on, so your program sees nothing until you press Enter. echo means your typing is shown back to you.

All three are the line discipline, and all three are what vim and less switch off.

$ ps -eo pid,pgid,sid,tpgid,comm --no-headers | grep -E 'sleep|cat|bash'
   3901    3901    3901    9820 bash
   9801    9801    3901    9820 sleep
   9802    9801    3901    9820 cat
   9803    9803    3901    9820 sleep
   9804    9803    3901    9820 cat

--- my shell ---
    PID    PGID     SID   TPGID COMMAND
   3901    3901    3901    9820 bash

stopping group 9801
   9803    9803 sleep
   9804    9803 cat

What to read out of it — read the four columns as four different questions.

PGID groups the jobs. 9801 and 9802 share PGID 9801. 9803 and 9804 share PGID 9803. Two jobs, two groups, and the PGID of each is its first process's PID.

SID is the same for everything — 3901, your shell's PID. All of this belongs to one login session, which is Section C3.

TPGID is the interesting one. It is the process group that currently owns the terminal — anything you type goes to that group and nothing else. Section C4 is entirely about this column.

It is the same on every line of one ps run, because they were all sampled at the same instant. It changes between runs (9820, then 9831), because the foreground group at that moment is whichever job is running ps. The number is not a fixed property of your session; it is a snapshot of who is holding the keyboard.

Then killing group 9801 removed both of its processes and left the other job completely untouched. One command, one job, precise.

$ ps -o pid,pgid,sid,tty,comm -p $$
    PID    PGID     SID TT       COMMAND
   3901    3901    3901 pts/0    bash

$ ps -o pid,pgid,sid,tty,comm -p $NORMAL
    PID    PGID     SID TT       COMMAND
  10011   10011    3901 pts/0    sleep

$ ps -o pid,ppid,pgid,sid,tty,comm -p $ESCAPED
    PID    PPID    PGID     SID TT       COMMAND
  10015       1   10015   10015 ?        sleep

What to read out of it — compare the three lines column by column.

Your shell: PID, PGID and SID are all 3901. It is its own group leader and its own session leader. Terminal pts/0.

The normal background job: its own group (10011), but SID is still 3901. It is a different crew on the same shift. Still attached to pts/0.

The setsid job: everything changed.

• SID = 10015, its own — a new session.
• TT = ? — no controlling terminal at all. There is no radio to hear.
• PPID = 1 — its parent is gone, so it was re-parented, exactly as Module 02 described for orphans.

That third line is what a daemon looks like. Not hiding from signals, not ignoring anything — simply not on the list any more. Nothing sent to your session can reach it, because it is not in your session.

terminal is owned by group: 3901
background job is in group: 10201

background job still alive? yes
[BACKGROUND JOB got SIGINT]

   10240 T    bash

What to read out of it — three separate results, all worth having.

One. The terminal is owned by group 3901 (your shell). The background job is in group 10201. Different numbers, which is the whole reason for what follows.

Two. Sending INT to the foreground group left the background job completely untouched. It is not that the job ignored it — the signal never went anywhere near it. This is exactly what happens when you press Ctrl-C with a job in the background.

Three. Aiming at the job's own group reached it immediately, and its handler ran. The job was perfectly capable of receiving SIGINT all along. It simply was not being sent one.

And the last line is the other rule. The background job that ran read is in state T — stopped. It tried to read from the terminal while not in the foreground, and was stopped by SIGTTIN.

That is a genuinely useful thing to recognise: a background job that mysteriously stops itself is almost always one that tried to prompt for input. fg gives it the microphone and it carries on.

The character never reaches your program. The terminal's line discipline intercepts it and converts it into a signal.

Step by step: you press Ctrl-C, the line discipline sees the configured interrupt character (stty -a shows intr = ^C), and because ISIG is enabled it sends SIGINT to every process in the foreground process group of that terminal. The default action for SIGINT is to terminate, which is why things stop.

Two consequences worth stating:

• It goes to a whole group, not one process. That is why Ctrl-C stops an entire pipeline rather than just the first command.
• Background jobs receive nothing. They are not in the foreground group, so the signal never reaches them.

The details that separate candidates:

• It is catchable, which is why some programs ask "press Ctrl-C again to confirm" and why vim does not exit. Contrast with SIGKILL, which cannot be caught.
• The key mapping is configurable, not fixed. stty intr ^X moves it, and full-screen programs turn ISIG off entirely so they can handle keys themselves.
• Ctrl-Z sends SIGTSTP and Ctrl- sends SIGQUIT — same mechanism, different characters. And SIGTSTP is catchable while SIGSTOP is not, which is exactly why a program can decline Ctrl-Z but nothing can decline kill -STOP.

A process group is a set of processes that are signalled together. The shell puts each job in its own group, so a pipeline a | b | c is one group of three. The PGID is the PID of the first process. kill -TERM -<PGID> signals all of them.

A session is a collection of process groups belonging to one login. Its leader is normally your shell, and it may have one controlling terminal. Within a session, exactly one group at a time is the foreground group — the one that receives keyboard signals.

So: session = one login, process group = one job, process = one command.

The details that separate candidates:

• Name the columns: ps -eo pid,pgid,sid,tty,tpgid. TPGID is the group that currently owns the terminal, which is the single most useful and least known of the four.
• Explain how a daemon escapes. setsid creates a new session with no controlling terminal, so nothing aimed at the old session can reach it. The signature in ps is PID = PGID = SID with TT showing ?.
• Say why setsid needs a fork first. It fails if the caller is already a process group leader, so the standard daemon pattern is fork, let the parent exit, and have the child — which cannot be a group leader — call setsid.
• Explain why systemd does not rely on any of this. A process can move itself out of a group or a session at will, so signalling a group is not a reliable way to stop everything a service started. systemd puts every process of a unit in a cgroup, which nothing can escape from the inside.

Several causes share this symptom, and a good answer works through them:

1. It is waiting for input it will never get. Under cron there is no terminal. Anything that prompts — sudo without -n, ssh confirming a host key, a package manager asking to confirm — blocks forever or is stopped by SIGTTIN. Redirect stdin from /dev/null so a stray read fails immediately rather than waiting.
2. The environment is almost empty. cron gives you a minimal PATH and none of your shell startup files. A command that works interactively is simply not found. Use absolute paths, or set PATH explicitly at the top of the script.
3. No controlling terminal. Anything that requires one — ssh -t, some interactive tools, anything calling tput — behaves differently or fails.
4. A different umask, from Module 03, so files come out with unexpected permissions.

The details that separate candidates:

• Name the diagnostic: run it the way cron does, with env -i /bin/sh -c '/path/to/script', rather than trying to reproduce it in your own shell where the environment is doing half the work for you.
• Recognise the stopped-job signature. A process sitting in state T that nobody stopped is almost always SIGTTIN — a background job that tried to read the terminal.
• Say what to do instead. For anything that matters, a systemd timer is better than cron: real logging through the journal, proper environment control, dependency ordering, and no silent failures because output was mailed to a mailbox nobody reads.

running (PID 10502)
sending INT (what Ctrl-C sends):
  -> caught SIGINT, not stopping
sending TSTP (what Ctrl-Z sends):
  -> caught SIGTSTP, not suspending
   10502 S    bash

now SIGSTOP, which cannot be caught:
   10502 T    bash

What to read out of it.

Both keyboard signals were caught and both were declined. The state is still S — sleeping normally, exactly as before. Neither Ctrl-C nor Ctrl-Z achieved anything, and that is entirely legitimate.

Then SIGSTOP moved it to T immediately. No handler ran, because none could. This is the pair from Section A4 in action: TSTP is a request, STOP is not.

This explains a difference you have probably noticed. vim catches Ctrl-Z so it can restore your terminal settings before suspending, which is why your screen looks normal afterwards. If Ctrl-Z sent SIGSTOP, vim would freeze mid-redraw and leave your terminal in a mess.

$ jobs -l
[1]+ 10701 Stopped                 sleep 300
state now: T

after bg:   S

stopped:    T
after CONT: S

--- what does a DIFFERENT shell see? ---
(that was a different shell - empty)
--- but the kernel still knows ---
    PID    PGID     SID STAT COMMAND
  10701   10701    3901 S    sleep

What to read out of it.

The state moved T → S → T → S, and kill -CONT did exactly what bg did. bg is not special: it sends SIGCONT and updates the shell's own note. That is all of it.

Then the important half. A different shell running jobs printed nothing, while ps found the process immediately, still in session 3901.

The job did not go anywhere. The list is what was missing. jobs, %1, fg and bg are conveniences your shell provides on top of process groups. The kernel has never heard of any of them.

plain job:    dead

SigIgn:	0000000000000001
nohup job:    alive

still in jobs list? 0
still running:      alive
now send HUP directly - predict the result first:
disowned job: dead

What to read out of it — the third case is the one worth understanding.

Plain job: SIGHUP arrived, default action is terminate, it died. As expected.

nohup job: SigIgn: 0000000000000001 — bit 1 set, which is SIGHUP. It genuinely ignores the signal, so sending HUP directly did nothing. This is the Section B3 mechanism at work: set to ignore, then exec, and the ignore survives.

disowned job: removed from the jobs list, still running — and it died when sent SIGHUP directly.

That is the distinction most people get wrong. disown does not make a process immune to SIGHUP. It only takes it off the shell's list so the shell does not send one. If a HUP reaches it by any other route, it dies exactly as before.

So the two tools solve the same problem in genuinely different ways:

• nohup changes the process — it now ignores HUP from anyone.
• disown changes the shell — it no longer sends HUP to that job.

my shell: SID=3901  TTY=/dev/pts/0

TOOL      PID     PPID    SID     TTY     IGNORES-HUP
nohup     11201   3901    3901    pts/0   yes
setsid    11205   1       11205   ?       no
disown    11209   3901    3901    pts/0   no

--- the supervised alternative ---
Running as unit: demo-job.service
● demo-job.service - /bin/bash -c echo "job started"; sleep 30
     Loaded: loaded (/run/user/1000/systemd/transient/demo-job.service; transient)
     Active: active (running) since Wed 2026-08-20 15:02:11 +08; 1s ago
   Main PID: 11220 (bash)
      Tasks: 2 (limit: 2273)
     CGroup: /user.slice/user-1000.slice/[email protected]/app.slice/demo-job.service
             └─11220 bash

What to read out of it — take the table one column at a time.

nohup: same SID as your shell, still on pts/0, and ignores HUP: yes. It is still fully part of your session. It just refuses that one signal.

setsid: everything changed. Its own SID (11205). PPID = 1, because its parent exited and it was re-parented, exactly as Module 02 described. TTY = ? — no controlling terminal at all. And note ignores HUP: no. It does not need to. It has left the session that would ever send one. This is what a real daemon looks like.

disown: same SID, same terminal, ignores HUP: no. From the kernel's point of view nothing at all changed. The only thing that changed is a list inside bash.

Three tools, three genuinely different mechanisms, visible in three columns.

Then the systemd-run block. It is running, it has a status you can query by name, its output goes to the journal rather than a stray file, and look at the last two lines: it has its own cgroup, and Tasks: 2 is counting every process inside it. Stopping the unit stops all of them — the guarantee Section B1 said process groups cannot provide.

They solve the same problem at three different points in the chain.

nohup sets SIGHUP to ignored and then execs the command. Because ignore survives exec, the program becomes immune to HUP without containing any code about it. It also redirects output to nohup.out. The process stays in your session with your controlling terminal.

disown removes the job from the shell's job list, so the shell does not send SIGHUP on the way out. From the kernel's point of view the process is unchanged — same session, same terminal — and it is not immune to SIGHUP from any other source.

setsid creates a new session with a new process group and no controlling terminal. Nothing aimed at the old session can reach it. This is the strongest of the three, and it is how daemons are made.

The details that separate candidates:

• The disown distinction is the separator. Most candidates say all three "make a process immune to hangup". disown does not — it stops one sender. Prove it: disown a job, then kill -HUP it, and it dies.
• Explain why nohup works at all — the ignore-survives-exec rule from fork/exec. That turns it from a magic command into a consequence of something you already know.
• Name what to use instead. All three leave an unsupervised process: nothing restarts it, nothing collects its output, nothing knows it exists. systemd-run --user --unit=name cmd gives detachment plus journal logging, status by name, a clean stop, and a cgroup so everything it started stops with it. For interactive work, tmux.

Because of a chain with two separate steps, and knowing both is the point:

1. The terminal goes away and the kernel sends SIGHUP to the foreground process group — which includes your shell, the session leader.
2. Bash then sends SIGHUP to all of its own jobs before exiting. Default action is terminate, so they die.

Step 2 is done by the shell, not the kernel. That is precisely why there are several places to intervene: make the job ignore HUP (nohup), take it off the shell's list (disown), or put it in a different session entirely (setsid).

The details that separate candidates:

• Say that bash kills the jobs, not the kernel. Most answers stop at "the kernel sends SIGHUP", which cannot explain why disown works at all.
• Mention shopt -s huponexit, which is off by default in bash on most distributions. That is why background jobs sometimes survive a logout with no nohup. The honest answer to "do they die?" is "it depends", followed by what it depends on.
• Distinguish a clean logout from a dropped connection — they are not identical paths and the behaviour can differ.
• Give the real recommendation: tmux or screen for anything interactive you want to return to, systemd-run for anything that should simply finish. Both beat a detached, unsupervised process.

T means stopped, so something sent SIGSTOP, SIGTSTP, SIGTTIN or SIGTTOU. The last two are the interesting answers, because nobody sent them deliberately.

SIGTTIN is sent automatically when a background process tries to read from the terminal. SIGTTOU is the equivalent for writing, when that is configured. The kernel stops the process rather than letting two programs fight over one keyboard.

So the usual cause is a background job that tried to prompt for something — sudo asking for a password, ssh asking to confirm a host key, a package manager asking to confirm.

Other causes: someone ran kill -STOP, a debugger attached, or Ctrl-Z in a job whose shell has since gone.

The details that separate candidates:

• Reach for SIGTTIN, because it is the one that happens by accident and the one most candidates have never heard of. It is the standard explanation for "the script hangs under cron but works by hand".
• Say that a stopped process frees nothing. It holds all its memory, its file descriptors and any locks, while using no CPU. A process left in T can hold a database lock indefinitely and hang things that look completely unrelated.
• Give the fix and the prevention: kill -CONT resumes it, or fg if it is your job. Prevent it by redirecting stdin from /dev/null and using non-interactive flags (sudo -n, ssh -o BatchMode=yes) so a stray read fails immediately instead of stopping the job.