👣Module 02 — Processes, fork/exec & Process States

$ echo "My shell is PID $$"
My shell is PID 3901

$ grep -E '^(Name|State|Pid|PPid|Uid|Gid|Threads|VmRSS)' /proc/$$/status
Name:	bash
State:	S (sleeping)
Pid:	3901
PPid:	3898
Uid:	1000	1000	1000	1000
Gid:	1000	1000	1000	1000
VmRSS:	   5504 kB
Threads:	1

$ wc -l < /proc/$$/status
58

What to read out of it.

Name: bash — the program this process is running.

State: S (sleeping) — your shell is asleep. It is waiting for you to type. It is using no CPU at all. This is normal for almost every process, as you saw in Module 01.

Pid: 3901 and PPid: 3898 — this process has a number, and it has a parent. Section A2 is all about that second number.

Uid has four numbers, not one. They are not all the same thing. Section A3 explains why that matters.

58 lines. The kernel is tracking 58 separate facts about one idle shell. Multiply that by the 243 processes from Module 01 and you can see why process management is a real job and not an afterthought.

$ ps -o pid,ppid,comm -p $$
    PID    PPID COMMAND
   3901    3898 bash

$ pstree -ps $$
systemd(1)---sshd(812)---sshd(3893)---sshd(3898)---bash(3901)---pstree(4402)

$ ps -o pid,ppid,comm -p 1
    PID    PPID COMMAND
      1       0 systemd

$ cat /proc/sys/kernel/pid_max
4194304

What to read out of it.

The pstree line is the best thing on this page. Read it left to right and you can see your entire login story: the kernel started systemd, which started sshd, which accepted your connection, which started your shell, which started pstree so it could print this.

Notice pstree(4402) on the end. The command you ran to look at the tree is itself in the tree. There is no outside.

ps -o pid,ppid,comm -p 1 shows PPID 0. There is no process 0 — that is just how the kernel says "no parent". PID 1 is the root.

pid_max is 4,194,304 here. Once PIDs reach that number they wrap back around to the beginning and start reusing free numbers. On older systems the default was only 32,768, which wrapped in hours on a busy box.

$ id
uid=1000(zaeem) gid=1000(zaeem) groups=1000(zaeem),27(sudo)

$ ps -o pid,ruser,euser,comm -p $$
    PID RUSER    EUSER    COMMAND
   3901 zaeem    zaeem    bash

$ ls -l /usr/bin/passwd
-rwsr-xr-x 1 root root 68208 Mar 23  2026 /usr/bin/passwd

$ find /usr/bin /usr/sbin -perm -4000 -type f 2>/dev/null
/usr/bin/passwd
/usr/bin/chsh
/usr/bin/su
/usr/bin/sudo
/usr/bin/mount
/usr/bin/umount
/usr/bin/newgrp
/usr/bin/chfn
/usr/bin/gpasswd

What to read out of it.

For your shell, RUSER and EUSER are both zaeem. No badge in play. Normal.

Now look closely at -rwsr-xr-x on passwd. In the owner's permissions there is an s where you would normally see an x. That single letter is the setuid flag. It means: when anyone runs this, run it as the file's owner — root.

That is why you, an ordinary user, can change your own password even though /etc/shadow is readable only by root. You are not root. passwd is, for a moment.

The find command lists every program on the machine that can do this. There are nine here. Each one is a program that any user can start and that will run with root's powers.

A process is a running instance of a program. The program is a passive file on disk; the process is that program actually executing, with its own memory, its own open files, and its own place in the instruction stream.

The kernel keeps a record for each one — usually called the process control block. It holds the PID, the parent's PID, the current state, the saved CPU registers, the memory map, the table of open file descriptors, the user and group IDs, CPU time used, and scheduling priority.

Go further than the textbook answer:

• Say why the saved registers matter. They are what makes preemption possible. The kernel takes the CPU away hundreds of times a second, and the process carries on afterwards as if nothing happened, because its exact position was written down.
• Point at where it is visible. "You can read most of that record from /proc/PID/status, and the memory map from /proc/PID/maps." That turns a definition into something you have actually looked at.

The detail that separates candidates: many people say a program and a process are "basically the same". Give the one-to-many relationship instead — one bash binary on disk, five people logged in, five processes. Then give the operational consequence: you can patch the file on disk while old processes keep running the old code from memory, which is exactly why patching without restarting leaves you vulnerable.

PID 1 is the first user-space process the kernel starts at boot. On modern distributions that is systemd; historically it was SysV init. It is special for three reasons:

1. It has no parent. Its PPID is 0, which is the kernel's way of saying "nobody started this". Everything else on the machine descends from it.
2. It adopts orphans. When a process dies while its children are still running, those children are handed to PID 1. Section C5 shows this happening.
3. It cleans up after the dead. As the adoptive parent it collects the exit status of those adopted children, which stops them piling up as zombies.

The container angle, which is where this question usually goes: in a container, PID 1 is your application, not an init system. Most applications were never written to adopt orphans or collect exit statuses, so in a busy container zombies can accumulate until the process table fills. That is the entire reason docker run --init exists, and why images often use tini or dumb-init.

The detail that separates candidates: PID 1 also cannot be killed by the default signal handling rules. Signals with a default action of "terminate" are simply not delivered to PID 1 unless it has explicitly installed a handler for them. This is why a container whose PID 1 ignores the shutdown signal takes the full grace period and then gets killed the hard way — the classic "why does my container take 30 seconds to stop" ticket.

The real UID is who the user actually is. The effective UID is the identity the kernel uses when it checks permissions. They are normally the same.

They differ when a program file has the setuid bit set. Running such a program sets your effective UID to the file's owner. /usr/bin/passwd is owned by root and is setuid, so while it runs it has root's permissions — which is how an ordinary user can update /etc/shadow, a file only root can write.

You can see the bit as an s in the owner's execute position: -rwsr-xr-x.

The details that separate candidates:

• Name the risk precisely. A setuid-root program gains all of root's powers, not only the one it needs. A single bug in it is a full privilege escalation. This is why find / -perm -4000 is a standard audit, and why an unexpected setuid binary is treated as a compromise indicator rather than a curiosity.
• Mention the modern replacement. Linux capabilities split root's power into around forty separate permissions, so a program that only needs to bind to a low port can be given just that one instead of all of root. ping moved from setuid to CAP_NET_RAW for exactly this reason.
• Know that scripts are excluded. Linux ignores the setuid bit on shell scripts, because the gap between checking the file and running the interpreter could be exploited. Candidates who claim you can make a setuid shell script are showing they have not tried it.

$ echo "My shell PID is: $$"
My shell PID is: 3901

$ ( echo "Inside the brackets, I am PID: $BASHPID" )
Inside the brackets, I am PID: 4127

$ ( echo "Different brackets, different PID: $BASHPID" )
Different brackets, different PID: 4128

$ echo "Outside again, PID: $BASHPID"
Outside again, PID: 3901

What to read out of it.

Every set of brackets produced a new PID. Each one was a real process, created by fork, that ran one echo and then exited.

Notice the numbers went 4127, then 4128. Each fork got the next number in line.

And notice the last line: 3901 again. The original shell never went anywhere. It made copies, the copies did their work and died, and the original carried on. That is the pattern behind almost everything a shell does.

$ grep -E 'clone|execve|exited' /tmp/fork.txt | head -8
4310  execve("/usr/bin/bash", ["bash", "-c", "ls /tmp > /dev/null; echo done"], 0x7ffd...) = 0
4310  clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD) = 4311
4311  execve("/usr/bin/ls", ["ls", "/tmp"], 0x5591...) = 0
4311  +++ exited with 0 +++
4310  +++ exited with 0 +++

What to read out of it. Read the PID at the start of each line — that is which process made the call.

1. 4310 runs execve to become bash. That line is strace itself: it forks, the child marks itself traceable, then execs the command you asked for. It appears with or without -f.
2. 4310 calls clone and gets back 4311. One process just became two. The number it got back is the child's PID, exactly as described above.
3. 4311 — the child — calls execve and becomes ls. That is Section B3.
4. 4311 exits, then 4310 exits.

Notice we needed -f (follow children). Without it you would still see the clone, but none of 4311's lines — the child's execve and exit would simply be missing. Forgetting -f is one of the most common reasons people conclude strace is not working.

$ echo "Variable holds ${#BIG} bytes"
Variable holds 50000000 bytes

$ grep VmRSS /proc/$$/status
VmRSS:	   52268 kB

$ time ( for i in $(seq 1 200); do ( : ); done )

real	0m0.148s
user	0m0.041s
sys	0m0.096s

What to read out of it.

The shell is using about 52 MB of real memory. It was forked 200 times. If fork copied memory, that is roughly 10 GB of copying — on a VM that only has 2 GB of RAM in total.

It took 0.148 seconds, and the machine never came close to running out of memory.

That is copy-on-write, measured. The memory was never copied, because none of those 200 children ever wrote to it. They ran : (do nothing) and exited.

$ bash -c 'echo "The shell is PID: $$"; exec ps -o pid,comm'
The shell is PID: 4455
    PID COMMAND
   3901 bash
   4455 ps

What to read out of it. This is the clearest demonstration in the module.

The shell announced it was PID 4455. Then ps reported PID 4455, running ps. (The 3901 bash line is just your interactive shell — bare ps lists everything on your terminal.)

No new process was created. The count of processes on the machine did not change. One process simply stopped being bash and started being ps, keeping its number.

This is why "exec creates a process" is wrong, and why interviewers like the question. fork creates processes. exec replaces the contents of one.

$ bash -c 'exec /no/such/program; echo "Did you see this line?"'
bash: line 1: /no/such/program: No such file or directory
$ echo "Exit code was: $?"
Exit code was: 127

$ bash -c 'exec /bin/true; echo "Did you see this line?"'
$ echo "Exit code was: $?"
Exit code was: 0

What to read out of it. Look at what did not happen: "Did you see this line?" was never printed, in either case. That is the lesson, and both halves teach it.

• When exec worked (/bin/true), the echo was destroyed along with the rest of the shell before it could run. Nothing to return to.
• When exec failed, exec did return — but a non-interactive shell treats a failed exec as fatal and exits immediately with 127.

127 is worth memorising. It means "command not found". When a CI job or a container exits with 127, the program you asked for was not there — wrong path, missing package, wrong architecture, or a script whose #! line points at an interpreter that does not exist. It is almost never a bug in your code.

$ grep -E 'clone|execve|out.txt|dup2|wait4' /tmp/seq.txt | head -10
4501  execve("/usr/bin/bash", ["bash", "-c", "ls /etc/hostname > /tmp/out.tx"...], ...) = 0
4501  clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|...|SIGCHLD) = 4502
4501  wait4(-1,  <unfinished ...>
4502  openat(AT_FDCWD, "/tmp/out.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
4502  dup2(3, 1)                        = 1
4502  execve("/usr/bin/ls", ["ls", "/etc/hostname"], ...) = 0
4501  <... wait4 resumed>[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 4502

What to read out of it — this is the whole module in eight lines. Follow the two PIDs.

1. 4501 is the shell. It calls clone and gets 4502. Now there are two.
2. 4501 immediately calls wait4 and stops there. The parent is now asleep, waiting. That is Section C3.
3. 4502 — still bash at this moment — opens /tmp/out.txt and gets file number 3.
4. 4502 calls dup2(3, 1). In plain terms: "make my output number 1 point at that file instead of the screen." This is the redirection, and it happens before ls exists.
5. 4502 calls execve and becomes ls. ls writes to output 1 as it always does. The bytes go to the file.
6. The parent's wait4 wakes up and collects the result: exited, status 0. (The child's own exit line is not shown because it does not match our grep pattern — add |exited to see it.)

Look at the order once more. The file was opened by bash, not by ls. That is why > works identically for every command on the system.

fork creates a process by copying the calling one. Afterwards there are two processes running the same program, with different PIDs, both continuing from the line after the fork.

exec replaces the program running inside an existing process. No new process is created. The PID, the parent, and the open file descriptors all survive; the memory is wiped and the new program starts from its beginning.

The one-line version: fork gives you another process, exec gives you another program.

Then explain how they are used together, because that is what the question is really asking. To run a command, a shell forks and then the child execs. The parent stays alive to wait for the result.

The details that separate candidates:

• fork returns twice, exec returns never. fork returns the child's PID to the parent and 0 to the child — that is how each knows which it is. A successful exec cannot return, because the code that called it no longer exists. If exec returns, it failed.
• Prove it rather than assert it. "bash -c 'echo \\$\\$; exec ps' prints the same PID twice — no process was created, one just changed program."
• Name why the split exists: the gap between them is where the shell sets up redirection, pipes and permissions, using ordinary calls in an ordinary process.

No, because of copy-on-write. fork does not copy the memory. Both processes are pointed at the same physical pages, marked read-only. When either one writes, the CPU traps it, the kernel copies that single page, and the writer gets a private copy. Pages that are only ever read are never copied.

The details that separate candidates:

• Say what actually does get copied: the page tables. The cost of fork scales with the size of the address space that has to be described, not with how much data is in it. This is why forking a process with an enormous mapped region is genuinely slow even though nothing is duplicated.
• Give the real production failure. Redis forks to write a snapshot. The child gets a free frozen view — but every page the parent modifies during the save must now really be copied. A Redis holding 8 GB can transiently need close to 16 GB, which is how teams get their cache OOM-killed during the first busy backup after go-live.
• Know the alternatives and why they exist. vfork and posix_spawn skip the address-space duplication for the common "fork then immediately exec" case. Large runtimes prefer posix_spawn for exactly this reason.

127 means "command not found". The shell tried to exec the program and the exec failed because there was nothing at that path. 126 means "found, but could not execute" — usually the execute permission bit is missing, or it is a directory, or the binary is for the wrong architecture.

Where they actually come from, which is the better answer: these are shell conventions, not kernel ones. The kernel returned ENOENT or EACCES from execve; the shell chose to exit with 127 or 126 to report it. So the number is the shell's translation of an errno, which ties straight back to how system calls report failure.

The detail that separates candidates: name the container case, because that is where you will actually meet it. A container exiting instantly with 127 is nearly always one of three things: the binary is genuinely missing from the image; the script's #! line names an interpreter the image does not have (#!/bin/bash inside Alpine, which only ships /bin/sh); or the binary is dynamically linked against a libc the image does not have (a glibc build dropped into an Alpine/musl image). All three look identical from the outside, and ldd inside the image separates them in one command.

Bonus: 128 + N means the process was killed by signal N. So 137 is 128 + 9 — killed by SIGKILL, which in a container almost always means the out-of-memory killer. 143 is 128 + 15, a normal shutdown. Recognising 137 on sight is worth a lot in a Kubernetes interview.

$ echo "Started PID $STUBBORN"
Started PID 4780

$ kill $STUBBORN
$ sleep 1
I am ignoring that
$ ps -o pid,stat,comm -p $STUBBORN
    PID STAT COMMAND
   4780 S    bash

$ kill -9 $STUBBORN
$ sleep 1
$ ps -o pid,stat,comm -p $STUBBORN
    PID STAT COMMAND
[1]+  Killed                  bash -c 'trap ...'

What to read out of it.

kill printed no error. It succeeded — the signal was delivered. The process simply chose to print a message and carry on. ps proves it is still there.

That is the entire point. kill succeeding does not mean the process stopped. It means the message was sent. Scripts that run kill and immediately assume the process is gone are wrong, and this is a very common bug in shutdown scripts.

kill -9 produced no message from the program at all, because the program was never told. The kernel simply removed it. Then ps finds nothing.

true       -> 0
false      -> 1
missing dir -> 2
bad command -> 127
killed -9   -> 137
killed -15  -> 143
exit 300    -> 44

What to read out of it.

true and false are real programs whose entire job is to return 0 and 1. That is how central this convention is.

ls on a missing directory returns 2, not 1. Different tools use different numbers for different failures, so "non-zero" is the only thing you can rely on across all of them.

137 and 143 are the 128 + N rule in action. Subtract 128: signal 9 and signal 15.

And exit 300 gave you 44. The status is only 8 bits, so 300 wraps around (300 − 256 = 44). This is a genuine trap: a script that does exit $(some_count) can report success by accident when the count happens to be 256, 512, or 768.

$ sleep 3 &
[1] 4901
$ echo "Started child $CHILD, now waiting..."
Started child 4901, now waiting...
$ wait $CHILD
$ echo "Child finished with status: $?"
Child finished with status: 0

$ bash -c 'sleep 1; exit 42' &
[1] 4912
$ wait $!
$ echo "That one returned: $?"
That one returned: 42

What to read out of it.

The 42 is the point. That number was chosen inside a completely separate process, which then ceased to exist. The kernel held it, and wait delivered it to the parent. That is the only channel through which a finished process can tell its parent anything.

Notice also that wait blocked. Your shell sat there for three seconds doing nothing. That is normal — the parent was asleep, using no CPU, waiting for an event. In Module 01's terms, it was in state S.

This is also what your shell does every time you run a command in the foreground: fork, exec, wait. It is why you do not get a prompt back until the command finishes, and why adding & (which skips the wait) gives you the prompt immediately.

$ ps -eo pid,ppid,stat,comm | awk -v p="$PARENT" 'NR==1 || ($3 ~ /^Z/ && $2==p)'
    PID    PPID STAT COMMAND
   5033    5032 Z    sleep
   5034    5032 Z    sleep
   5035    5032 Z    sleep

$ echo "Zombie PID is $ZOMBIE"
Zombie PID is 5033

$ kill -9 "$ZOMBIE"
$ sleep 1
$ ps -o pid,ppid,stat,comm -p "$ZOMBIE"
    PID    PPID STAT COMMAND
   5033    5032 Z    sleep

$ kill "$PARENT"
$ sleep 1
$ ps -o pid,stat,comm -p "$ZOMBIE" >/dev/null 2>&1 || echo "Zombie is gone."
Zombie is gone.

What to read out of it — the middle block is the lesson.

kill -9 on the zombie ran without an error, and the zombie is still listed, unchanged. Compare that with Exercise C1.1, where kill -9 removed a running process instantly. The difference is that here there is nothing to remove. kill -9 delivers a signal to a running process; this process finished several seconds ago.

Killing the parent cleared all three immediately. When the parent died, the zombies were handed to PID 1, which collected them at once. That is the fix, every time.

Why three children and not one? Because with a single background child, bash would very likely reap it for you while waiting, and no zombie would appear at all. Starting several makes the effect reliable — and it is closer to the real production shape, which is a parent forking many workers and reaping none of them.

Notice COMMAND still says sleep. The kernel keeps the name so ps can tell you what died. On a real system this is your best clue about which program the parent keeps forgetting to reap.

$ echo "Parent is $PARENT, child is $CHILD"
Parent is 5209, child is 5210

$ ps -o pid,ppid,stat,comm -p "$CHILD"
    PID    PPID STAT COMMAND
   5210    5209 S    sleep

$ sleep 4
$ ps -o pid,ppid,stat,comm -p "$CHILD"
    PID    PPID STAT COMMAND
   5210       1 S    sleep

$ ps -o pid,comm -p "$NEWPARENT"
    PID COMMAND
      1 systemd

What to read out of it.

The PPID changed by itself, from 5209 to 1. Nothing you ran did that. The kernel re-parented the process the moment its original parent exited.

The STAT stayed S throughout. The orphan was never disturbed — it kept sleeping through the whole thing. Losing a parent does not harm a process.

A zombie is a process that has finished but whose exit status has not yet been collected by its parent. Its memory and resources are already released; all that remains is a small kernel record holding the exit status, plus the PID. ps shows it as Z / <defunct>.

An orphan is the opposite situation: a process still running whose parent has died. The kernel re-parents it, so it keeps running normally and will be reaped correctly when it finishes.

The clean contrast to give: a zombie is dead but not cleaned up; an orphan is alive but re-homed. An orphan is harmless. A zombie is harmless individually and dangerous in bulk.

The details that separate candidates:

• Say why zombies matter, precisely. They use no memory and no CPU — but each holds a PID, and pid_max is finite. Enough of them and the machine cannot fork at all, which means you cannot even log in to fix it. Naming the resource that actually runs out is the answer interviewers are listening for.
• You cannot kill a zombie, and explain why: there is nothing running to receive a signal. kill -9 is a no-op. The fix is to make the parent reap, or to kill the parent so PID 1 adopts and reaps it.
• Give the modern re-parenting answer. Orphans go to the nearest subreaper, not necessarily PID 1. systemd registers user sessions and services as subreapers so each unit cleans up its own descendants. "Orphans go to init" is the older, less accurate answer.

Work in this order and say it out loud:

1. Confirm the scale and the trend. ps -eo stat | grep -c Z. A few is normal. What matters is whether it is climbing.
2. Check how close you are to the ceiling. Compare against cat /proc/sys/kernel/pid_max. This tells you whether you have hours or minutes before the machine cannot fork.
3. Find the guilty parent — this is the key step: ps -eo ppid,stat | awk '$2 ~ /^Z/ {print $1}' | sort | uniq -c | sort -rn. It will be one process, and now you know which service has the bug.
4. Clear it now: restart that parent. Its zombies are re-parented to PID 1 or the nearest subreaper and reaped instantly. Do not waste time sending kill -9 to the zombies themselves.
5. Fix it properly: the parent is forking children and never calling wait. Either it should reap them, or it should tell the kernel it does not want the statuses at all.

The details that separate candidates:

• Never say kill -9 as the fix. Volunteering that it cannot work, and why, is worth more than the rest of the answer.
• Name the container version, because that is where this happens now. If PID 1 in a container is your application, it inherits reaping duties it was never written for. docker run --init, or tini/dumb-init as PID 1, fixes it. Kubernetes has shareProcessNamespace considerations too.
• The classic shape: a parent that forks workers and then blocks on something else forever — exactly the pattern in Exercise C4.1.

Plain kill sends SIGTERM (15): "please stop". The process is told, and can catch it to shut down cleanly — finish in-flight requests, flush buffers to disk, close connections, deregister from a load balancer. It can also ignore it.

kill -9 sends SIGKILL, which cannot be caught, blocked or ignored. The process is never told; the kernel removes it. No cleanup happens.

Always TERM first, then KILL only after a timeout. That is exactly what systemctl stop does with TimeoutStopSec, and what Kubernetes does with terminationGracePeriodSeconds.

The details that separate candidates:

• kill succeeding tells you nothing about whether the process stopped. It only means the signal was delivered. Shutdown scripts that assume otherwise are a common bug.
• Name what you lose with -9: unflushed database writes, dropped connections mid-request, stale lock files and PID files left behind, temporary files never cleaned up. "It's faster" is not a reason.
• Know the case where -9 does nothing. A process in uninterruptible sleep (D) cannot be killed by any signal at all, and a zombie cannot either. Those are the two exceptions, and being able to name both is a strong signal of real experience.
• The container angle: if a container ignores TERM, the usual cause is that PID 1 is a shell script that started the real application as a child. The shell receives the signal; the application never hears it. Use exec in the entrypoint so the application becomes PID 1 — which is exactly the exec behaviour from Section B3.

Because in a container, PID 1 is your application, not an init system — and PID 1 carries two responsibilities most applications were never written to handle.

1. Adopting and reaping orphans. Any process whose parent dies inside the container is re-parented to PID 1. If your application ignores those exit statuses, they accumulate as zombies until the PID space is exhausted.
2. Signal handling. The kernel does not apply default "terminate" actions to PID 1. A signal with a default action of terminate is simply not delivered unless PID 1 has explicitly installed a handler. So a PID 1 that ignores SIGTERM cannot be stopped politely at all — the platform waits out the grace period and then uses SIGKILL.

The fix is a small init process as PID 1: docker run --init, or tini / dumb-init in the image. They do nothing but forward signals and reap orphans.

The detail that separates candidates: name the shell-script entrypoint trap, because it is the one you will actually meet. An entrypoint like #!/bin/sh then myapp leaves the shell as PID 1 and the application as a child. TERM goes to the shell, which does not forward it, and the app never learns it should shut down. The one-word fix is exec myapp — the shell is replaced by the application, which becomes PID 1 itself and receives the signals directly. That single keyword is behind a very large number of "my container takes 30 seconds to stop" tickets.

$ ps -o pid,stat,pcpu,wchan:20,comm -p $SLEEPER,$SPINNER
    PID STAT %CPU WCHAN                COMMAND
   5412 S     0.0 hrtimer_nanosleep    sleep
   5414 R    99.3 -                    bash

$ kill -STOP $SPINNER
$ ps -o pid,stat,pcpu,comm -p $SPINNER
    PID STAT %CPU COMMAND
   5414 T    48.5 bash

What to read out of it.

sleep is S at 0.0% CPU. It is doing nothing and costing nothing.

The loop is R at 99.3% CPU. It always wants the CPU, so it is always either on one or queuing for one.

The WCHAN column is the useful one and most people never use it. It shows what the process is waiting for, by naming the kernel function it is parked in. hrtimer_nanosleep means "waiting for a timer to expire" — which is exactly what sleep does. The spinner shows - because it is not waiting for anything.

After kill -STOP, the state is T. The process still exists, still holds all its memory, and is simply not being scheduled. kill -CONT would resume it mid-instruction as if nothing had happened. This is what Ctrl-Z does.

Notice that %CPU did not drop to zero. That column is not a live rate — it is total CPU time divided by total lifetime, so it decays slowly as the clock keeps running while the CPU time stands still. The state letter is what tells you it stopped; the percentage lags behind. This catches people out when they watch %CPU expecting it to fall immediately.

23       5601 D    submit_bio_wait        dd
14       5601 R    -                      dd
 3       5601 S    -                      dd

What to read out of it.

Out of 40 samples, the process was in D state 23 times — most of the time. It was waiting for the disk to physically accept the data.

WCHAN says submit_bio_wait. That is the kernel function that submits a block I/O request and waits for it to complete. The column has told you exactly what it is waiting for: the block layer.

During those 23 samples, that process was unkillable. A kill -9 sent then would have been recorded and only acted on once the write finished.

$ uptime
 12:04:11 up 1:22,  1 user,  load average: 0.08, 0.12, 0.09
$ cat /proc/loadavg
0.08 0.12 0.09 1/243 5702

--- during the writes ---
 12:04:39 up 1:23,  1 user,  load average: 3.42, 1.16, 0.41
CPU idle percentage:
88% idle,  9% iowait
Processes in D state right now:
4

What to read out of it — this is the payoff of Part D.

Load average went from 0.08 to 1.38 in 25 seconds, and it is still climbing. On a 2-CPU machine that already reads like trouble.

And the CPU is 88% idle. Almost nothing is being computed.

The two facts are reconciled by the last line: 4 processes in D state. Each one is waiting on the disk, using no CPU, and each adds 1.0 to the load average.

So why is the figure 1.38 rather than 4.0? Because load average is smoothed, not instantaneous. The 1-minute figure moves towards the true value gradually, and 25 seconds is not long enough to get there. Leave the writes running for three minutes and it climbs towards 4.0. This lag is exactly why a short spike can be invisible in the numbers.

iowait at 9% is the supporting clue. That is CPU time spent idle specifically because something is waiting for I/O.

Look at the three load figures together: 1.38, 0.42, 0.20. The 1-minute figure is the highest, which means this started recently and is still climbing. The reverse order — 15-minute highest — would mean the problem is easing off.

The fourth field of /proc/loadavg is worth knowing too: 1/243 means 1 process currently runnable out of 243 total.

$ ps -eo pid,ppid,stat,pcpu,pmem,wchan:18,etime,comm --sort=-pcpu | head -11
    PID    PPID STAT %CPU %MEM WCHAN              ELAPSED COMMAND
   5414    3901 R    99.3  0.2 -                    00:42 bash
      1       0 Ss    0.1  1.1 ep_poll             01:24:07 systemd
    812       1 Ss    0.0  0.6 ep_poll             01:23:44 sshd
   3901    3898 Ss    0.0  0.2 do_wait                12:03 bash
   5455    3901 R+    0.0  0.1 -                    00:00 ps
   ...   (rows 6 to 10 omitted)

$ ps -eo stat --no-headers | cut -c1 | sort | uniq -c | sort -rn
    198 S
     31 I
      8 R
      2 D
      1 Z

$ ps -eo pid,ppid,stat,wchan:18,comm | awk 'NR==1 || $3 ~ /^[ZD]/'
    PID    PPID STAT WCHAN              COMMAND
   5033    5032 Z    -                  sleep
   5601    3901 D    submit_bio_wait    dd

What to read out of it.

In the first table, look at the suffixes. systemd and sshd are Ss — sleeping session leaders. Your shell is Ss too. The ps command itself is R+ — running, and in the foreground group, because it owns your terminal right now.

WCHAN tells the story of the sleepers for free. systemd and sshd are in ep_poll, waiting on epoll for a connection or an event. Your shell is in do_wait — waiting for a child, which is exactly Section C3. Three sleeping processes, three genuinely different reasons, visible in one column.

The second command is the one to memorise. One line, and you know the health of the machine: 198 sleeping (normal), 8 runnable, 2 in D, 1 zombie. Run it on a healthy host to learn its normal shape, and the abnormal shape becomes obvious.

The third is your triage command. Z and D are the two states that indicate something is wrong, and this prints only those.

They are opposite problems that share a symptom.

D — uninterruptible sleep. The process is alive and waiting inside the kernel for an I/O operation that must not be interrupted. Signals are not delivered at all, so SIGKILL is recorded but not acted on until the I/O completes. If the storage never responds, the process stays there indefinitely.

Z — zombie. The process is already dead. Its memory and resources are gone; only the exit status record remains, waiting for the parent to call wait. There is nothing running to receive a signal, so SIGKILL is a no-op.

The one-line contrast: D is too busy to die, Z is already dead.

The fixes are completely different. For D: fix the storage, or reboot. For Z: make the parent reap, or kill the parent so PID 1 adopts and reaps it.

The details that separate candidates:

• Explain why D exists rather than just naming it. The kernel is holding locks and has a device mid-transfer; interrupting there would corrupt filesystem state or let hardware write into freed memory. It is a correctness guarantee, not an oversight.
• Use WCHAN. ps -eo pid,stat,wchan:25,comm names the kernel function the process is parked in, which usually identifies the subsystem immediately — submit_bio_wait for the block layer, an nfs_ prefix for NFS.
• Mention TASK_KILLABLE. Some newer waits — notably parts of NFS — use a middle state that blocks ordinary signals but still allows SIGKILL, added specifically because unkillable NFS hangs were so painful. So "D is always unkillable" is the older answer; "most D waits are unkillable, and some paths are now killable" is the current one.

The machine is not short of CPU. It is blocked on I/O.

On Linux, load average counts processes in uninterruptible sleep (D) as well as those running or runnable. A process waiting on storage consumes no CPU and still adds to the number. So a high load with an idle CPU means processes are stuck waiting, not competing.

How to confirm it, in order:

1. ps -eo stat | grep -c '^D' — how many are actually blocked.
2. vmstat 1 — expect a high wa (iowait) and a low r column.
3. ps -eo pid,stat,wchan:25,comm | awk '$2 ~ /^D/' — what they are waiting on.
4. iostat -x 1 — is a device at 100% utilisation with high await?
5. dmesg -T | tail — storage errors, controller resets, NFS timeouts.
6. mount | grep nfs — a hung network filesystem is the classic cause.

The details that separate candidates:

• Say it is Linux-specific. Most other Unixes count only runnable processes. This is why the same number means different things on different systems, and it is a detail few candidates know.
• Normalise by CPU count. A load of 20 on a 32-core box is not the same as on a 2-core box. Always ask "load relative to nproc".
• Read the three numbers as a trend, not as one figure — 1, 5 and 15 minutes. The 1-minute figure being highest means it is getting worse; the 15-minute figure being highest means it is recovering. Also note they are smoothed, so a problem that started 20 seconds ago will not yet show its true size.
• State the conclusion plainly: adding CPU or scaling out would not help. Fix the storage.

Work down the states, because each one has a different cause and a different fix:

1. ps -o pid,stat,wchan:25,comm -p PID first. The state letter tells you which problem you have before you try anything.
2. State Z — it is already dead. Signals are pointless. Find the parent with ps -o ppid= -p PID and restart it.
3. State D — it is blocked in the kernel and no signal will reach it. Read WCHAN to identify the subsystem, check dmesg and iostat for storage trouble, and look for a hung NFS or detached volume. Often the only resolutions are restoring the storage or rebooting.
4. State S or R and still alive after TERM — the process is catching SIGTERM and ignoring it, or its handler is stuck. kill -9 will work here. This is the only one of the three where -9 is the answer.
5. It dies and comes straight back — it is not the process, it is the supervisor. systemd with Restart=always, a Kubernetes controller, or a wrapper script is recreating it. Kill the supervision, not the process.

The details that separate candidates:

• Leading with the state letter rather than escalating straight to kill -9 is the whole point of the question. It shows you diagnose before acting.
• Knowing that -9 cannot help in two of the five cases, and being able to say which, is what an experienced interviewer is listening for.
• Volunteering case 5 shows you have actually operated systems. It is very common and it catches people who only think about processes in isolation.

You are marked on whether your conclusion follows from your evidence, not on exact numbers.

a–d, the zombies. Around 20 in state Z. The second command prints one PID with a count next to it — that is the parent, and it is the answer. kill -9 cannot help because those processes are already dead; there is nothing to receive a signal. The fix now is to restart that parent, so its zombies are re-parented to PID 1 (or the nearest subreaper) and reaped at once. The fix in the code is for the parent to call wait for its children, or to tell the kernel it does not want their exit statuses. A complete answer also mentions the ceiling: zombies hold PIDs, and pid_max is finite.

e–f, the stubborn process. kill succeeded — the signal was delivered. The process installed a handler that discards TERM, so it simply ignored it. kill returning 0 means "sent", never "stopped", and code that assumes otherwise is a common shutdown bug. A real platform sends TERM, waits a fixed period, then sends KILL: systemd via TimeoutStopSec, Kubernetes via terminationGracePeriodSeconds (30s by default). This is exactly why some containers always take the full 30 seconds to stop.

g–h, the load. No, it is not a CPU problem, and two numbers prove it: load average is well above 1 while CPU idle is high (with wa elevated). Load average on Linux counts D-state processes as well as runnable ones, so blocked-on-storage work inflates it while using no CPU. The WCHAN column names what they are waiting for — a block-layer function such as submit_bio_wait. Scaling CPU or adding replicas would change nothing.