🧵Module 06 — Threads, Races & Deadlock

$ ps -e --no-headers | wc -l
243

$ ps -eL --no-headers | wc -l
412

$ ps -eLo pid,comm --no-headers | awk '{c[$1" "$2]++} END {for (k in c) print c[k], k}' | sort -rn | head -6
14 812 sshd
 9 1 systemd
 4 756 rsyslogd
 2 640 systemd-udevd
 1 3901 bash

$ ls /proc/1/task
1  128  129  130  131  147  148  149  150

$ grep -E '^(Name|Pid|Threads)' /proc/1/status
Name:	systemd
Pid:	1
Threads:	9

What to read out of it.

243 processes, 412 threads. Nearly 170 more lines of execution than there are processes. Every ps command you ran in Modules 02 to 05 was hiding most of them.

/proc/1/task is the key directory. It lists one entry per thread, and those numbers are thread IDs. Notice the first one is 1 — the same as the PID. Section A2 explains why.

Those nine systemd threads all share PID 1, one address space, and one set of open files. ps without -L shows them as a single line, which is usually what you want and occasionally exactly what is hiding your problem.

$ grep -oE 'CLONE_[A-Z_]+' /tmp/thr.txt | sort -u | head -10
CLONE_CHILD_CLEARTID
CLONE_FILES
CLONE_FS
CLONE_PARENT_SETTID
CLONE_SETTLS
CLONE_SIGHAND
CLONE_SYSVSEM
CLONE_THREAD
CLONE_VM

$ ps -Lo pid,tid,comm -p "$SSHD" --no-headers | head -5
    812     812 sshd

--- /proc entries ---
812

What to read out of it — the flag list is the definition of a thread.

CLONE_VM (share memory), CLONE_FILES (share descriptors), CLONE_SIGHAND (share handlers), CLONE_THREAD (share the thread group). Compare with Module 02's fork, where the flags were essentially just SIGCHLD.

Same system call. Different flags. Completely different thing. That is the whole of thread creation on Linux, and it is why "Linux threads are just processes that share everything" is accurate rather than a simplification.

In the ps -Lo output, PID and TID are the same number for a single-threaded process. Run it against a threaded program and you get one PID repeated down the page with a different TID on each line.

$ time ( for i in $(seq 1 500); do ( : ) ; done )
real	0m0.361s
user	0m0.104s
sys	0m0.244s

$ time python3 -c "...500 threads..."
real	0m0.094s
user	0m0.048s
sys	0m0.041s

What to read out of it.

Threads came out roughly four times faster here, and the gap widens sharply as the process gets bigger — because forking has to duplicate page tables, and a thread does not.

But notice how small both numbers are. Under a millisecond each. Process creation on Linux is genuinely cheap, which is why "threads are faster" is a much weaker argument than people assume, and why nginx and PostgreSQL happily use worker processes.

The honest reason to choose threads is usually not creation speed — it is that they share memory for free. If your workers need constant access to the same large data structure, copying it between processes is the real cost.

expected: 1000
actual:   137

What to read out of it.

Your number will be different from mine, and different again if you run it twice. That variability is the bug — the answer depends on how the workers happened to interleave.

Not a few increments were lost. Nearly all of them were — 137 on an 8-core box, and single digits on a 2-CPU VM, so anywhere from 85% to 99% loss is normal. The fewer CPUs and the more workers, the worse it gets. Every time a worker read the file, did its arithmetic, and wrote back, any other worker that read in that gap was working from a stale value and overwrote the result.

Read the loop again and find the gap. Between value=$(cat ...) and echo ... >, this worker is holding a number that is already out of date. Twenty workers, fifty times each, and the window is wide open every time.

This is genuinely the same bug as two threads incrementing a variable. The shell makes the three steps visible; in a compiled program they are three machine instructions and the window is nanoseconds wide — which makes it rarer, and far harder to find.

expected: 1000
actual:   1000

What to read out of it.

Exactly 1000, every time. Run it ten times and you get 1000 ten times. Compare with Exercise B1.1, which gave a different wrong answer on every run.

Nothing about the arithmetic changed. The only difference is that the read-modify-write now happens inside flock, so no worker can start its read while another is between reading and writing.

You will also notice it is considerably slower. That is not overhead to be optimised away — it is the actual cost of correctness here. Twenty workers are now taking turns rather than running at once, and the critical section has become the bottleneck. That is rule 2 from above, measured.

=== low contention ===
100.00    0.055152          16      3338        79 total

=== high contention ===
  0.10    0.001124           4       240           flock
100.00    1.105608          27     39704      2047 total

=== high contention, the same run with -w (wall clock) ===
 25.58    4.317391       17989       240           flock
100.00   16.880830         425     39686      2047 total

What to read out of it.

Same amount of actual work — 240 lines written either way. The high-contention run made about twelve times as many system calls and took roughly 20 times longer. Be honest about where that syscall count comes from: most of it is the cost of forking and execing flock (and the sh -c it runs) 240 times, roughly 128 syscalls apiece — not the lock. That is the weakness of doing this demo in a shell.

The flock line is the cost — but you have to ask for it, and this is the real lesson of the exercise. By default strace -c summarises system time, CPU spent inside the kernel. A thread blocked on a lock burns no CPU, so the default summary shows flock at about 4 microseconds a call and hides the waiting completely. Add -w and the same 240 calls report about 18,000 microseconds each. That is the queueing, finally visible.

strace -c counts syscalls and kernel CPU; strace -c -w counts the wall clock — and blocking only ever shows up in the second.

In a real threaded program the same shape appears as futex rather than flock, because the lock is in shared memory rather than on a file. The reading is identical: a large call count with a high time-per-call means threads are queueing.

W1: holding A
W2: holding B
W1: now wants B...
W2: now wants A...
W1: TIMED OUT waiting for B
W2: TIMED OUT waiting for A

What to read out of it — read the order of those lines.

Both workers successfully took one lock. Both then asked for the other one. And then nothing happened for five seconds, until the timeouts fired.

Without -w 5, those last two lines never appear. The script hangs forever, and the only way out is to kill it. That is a real deadlock: not slow, not retrying, not consuming CPU — permanently stopped.

Now look at what caused it, because it is only one thing. Worker 1 takes A then B. Worker 2 takes B then A. Change worker 2 to take A then B and the deadlock is impossible — it would simply wait for A at the start while holding nothing, and worker 1 would finish and release both.

That is condition 4 broken by lock ordering, and it is a change to the order of two lines.

=== are they using CPU? ===
  10412 S     0.0 do_wait                  bash
  10415 S     0.0 do_wait                  bash
  10419 S     0.0 locks_lock_inode_wait    flock
  10420 S     0.0 locks_lock_inode_wait    flock

=== sample again 3 seconds later - has anything moved? ===
  10412 S     0.0 do_wait                  bash
  10415 S     0.0 do_wait                  bash
  10419 S     0.0 locks_lock_inode_wait    flock
  10420 S     0.0 locks_lock_inode_wait    flock

=== which files are they holding locks on? ===
l-wx------ 1 zaeem zaeem 64 Aug 20 18:22 200 -> /tmp/dlA
l-wx------ 1 zaeem zaeem 64 Aug 20 18:22 201 -> /tmp/dlB

gone - it was killable

What to read out of it — four separate signals, and together they are conclusive.

State S, CPU 0.0. Both processes are asleep, using nothing. This is the fact that rules out a spin loop or real work, and it is why "the service is hung but the CPU is idle" points at locks rather than load.

WCHAN says locks_lock_inode_wait — on the flock processes. Module 02 introduced this column and this is exactly what it is for: the kernel naming what a process is parked on. Note which process, because this is a real trap. The two shells report do_wait, because they are waiting for their flock child; the child is the one holding the lock and the one blocked on it. Always follow the tree down to the task that is actually stuck. In a threaded program the equivalent line begins futex_ — futex_do_wait on kernel 6.x, futex_wait_queue on 5.x — and the reading is the same.

Two samples, three seconds apart, identical. That is the test that separates deadlock from contention. A contended lock would show different processes parked at different moments, and work would be getting done. Here nothing has moved at all.

The file descriptors show what they hold. 200 -> /tmp/dlA — Module 03's /proc/PID/fd telling you which lock this process is sitting on. Do the same for the other process and you can see the cycle directly.

And kill -9 worked. That is the clean distinction from D state in Module 02: a deadlocked process is in S, so signals reach it normally. Unkillable means storage, not locks.

One caveat worth carrying into production, and it is why the exercise kills the children too. Killing the shell alone does not release the locks, because its flock child inherited the descriptors, and flock(2) releases a lock only when every duplicate descriptor is closed — and fork duplicates them. That is what flock(1)'s -o/--close option is for, and it is the same reason a "dead" service can still be holding a lock through a surviving child.

=== livelock: is the machine busy? ===
%Cpu(s): 45.5 us, 54.5 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  687 root      20   0    7160   5152   2012 S  18.2   0.1   0:00.35 bash
  688 root      20   0    7160   5152   2012 S  18.2   0.1   0:00.36 bash
W2: 5s of retrying, no progress
W1: 5s of retrying, no progress
=== CPU actually consumed by the retry loops ===
0m0.030s 0m0.041s
0m7.291s 0m2.155s
(compare with Exercise C2.1, where CPU was 0.0 and 'times' would show nothing)

What to read out of it.

0.0 id — both CPUs saturated for the whole five seconds, and the shell's times builtin reports 7.3 s of user plus 2.2 s of system time consumed by children in 5 s of wall clock on a 2-CPU box. Compare directly with Exercise C2.1, where the deadlocked processes were S at 0.0 and consumed nothing.

Note that ps %CPU on the two bash lines reads only about 18%: the CPU is being burned inside short-lived flock children, and a parent's %CPU never includes a reaped child's time. That is exactly why times, or the machine-wide %Cpu(s) line, is the honest measurement here.

Same outcome — no work is getting done — and the opposite signature. That single difference is how you tell them apart, and it decides where you look next.

Livelock is usually a retry loop with no backoff. Everyone fails, everyone retries instantly, everyone collides again. The fix is randomised exponential backoff: wait a little, wait longer each time, and add jitter so the retries stop landing together.

=== process level: what does plain ps say? ===
  8495 98.0    5 python3

=== thread level: which one is actually busy? ===
  8495  8495  0.4 Sl   futex_do_wait    python3
  8495  8497 98.0 Rl   -                python3
  8495  8498  0.0 Sl   hrtimer_nanoslee python3
  8495  8499  0.0 Sl   hrtimer_nanoslee python3
  8495  8500  0.0 Sl   hrtimer_nanoslee python3

=== per-thread names, if the program sets them ===
8495: python3
8497: python3
8498: python3
8499: python3
8500: python3

What to read out of it — the two views tell completely different stories.

At the process level: 98.0% CPU with nlwp 5. On a 2-CPU box that is one core fully consumed. It tells you the process is busy — but not why, and not that only one of its five threads can ever make progress. On a 32-core host the same 98% is 3% of the machine, and every host-level dashboard would call it idle.

At the thread level, thread 8497 is at 98.0% in state Rl while every other thread sits at 0.0 in Sl. One thread is saturating a core; the main thread and the three sleepers are all asleep. That is the fact the process view cannot give you.

The trailing l in Sl and Rl is ps telling you the process is multi-threaded — a useful tell on its own.

Read the WCHAN column too, exactly as in Module 02. hrtimer_nanoslee means waiting on a timer — those are the sleep(8) threads. futex_do_wait is the main thread, blocked joining its children on a lock, which is your first sighting of the futex from Section B3 in the wild. The busy thread shows -, because it is not waiting for anything. And the busy thread shows -, because it is not waiting for anything; it is running.

Notice TID 11204 equals the PID. That is the first thread, from Section A2.

And the comm list shows every thread called python3. No per-thread names, which is the limitation from the analogy above. A JVM would show GC Thread#0, C2 CompilerThread0 and so on, and that makes the difference between "a thread is hot" and "the garbage collector is hot".

per-user process/thread limit: 7557
system-wide thread max:        15114
threads in use right now:      412
$ ulimit -s
8192

limit lowered to 50
bash: fork: retry: Resource temporarily unavailable
bash: fork: Resource temporarily unavailable

What to read out of it.

ulimit -s is 8192 KB — 8 MB per thread stack, and that is the number behind the memory row in the table. A 1,000-thread process reserves about 8 GB of address space for stacks alone.

Resource temporarily unavailable is EAGAIN, and it is the same errno Module 01 listed. On a socket it means "no data yet, try again"; on fork or thread creation it means "you have hit a process or thread limit". Same errno, entirely different meaning depending on the call — which is exactly why Module 01 insisted the per-call ERRORS section matters more than a generic table.

Notice bash retried before failing: fork: retry: then fork:. It attempted several times before giving up, which is why this failure sometimes shows up as a slow service before it shows up as a broken one.

A process has its own address space; threads within a process share one. Threads share the memory, the open file descriptors, the working directory, the signal handlers and the PID. Each thread has its own stack, registers, thread ID, errno and signal mask.

The consequences follow directly:

• Communication. Threads share data for free. Processes need pipes, sockets or shared memory.
• Cost. Threads are cheaper to create and switch between, because there is no address space to duplicate and no page-table switch.
• Safety. A crash in one thread takes the whole process down. A crashed process leaves its siblings alone.

The details that separate candidates:

• Say how Linux actually implements it. There is no separate thread object — clone() with CLONE_VM | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD creates a task sharing everything, and fork is the same call sharing nothing. Threads and processes are the same kernel object with different sharing flags.
• Get the identifiers right. Every thread has a unique TID; all threads in a process share a PID, which is really the thread group ID and equals the first thread's TID. ps -eL shows them; /proc/PID/task/ lists them.
• Mention the safety trade as a design decision, not a footnote: browsers use separate processes per tab specifically so one crash cannot take the rest down, despite the higher cost.

A race condition is when the result depends on the order two operations happen to run in, and nothing enforces that order. The classic case is check-then-act, or a read-modify-write on shared data: two threads both read the same value, both increment it, both write it back, and one update is silently lost.

You prevent it by making the critical section indivisible — mutual exclusion via a mutex, or an atomic operation such as compare-and-swap that does the whole read-modify-write in one step.

Two rules matter: every piece of code touching the data must use the same lock, and the critical section must be as short as possible, because everything inside it is single-threaded by definition.

The details that separate candidates:

• Explain why races are so hard, rather than just defining one. They are intermittent, worsen under load, often vanish when you add logging, and the corruption surfaces far from its cause. "Test harder" is not a strategy.
• Give a non-threading example, because that shows you understand the pattern rather than the vocabulary: two requests both checking "does this username exist", both getting no, both inserting. The fix there is a database unique constraint — the same idea, enforced by a different layer.
• Name the lock-free option: atomics and compare-and-swap avoid a lock entirely for simple cases and are how counters are usually done properly.

Deadlock is two or more threads each holding a resource the other needs, so none can proceed — permanently. It requires four conditions simultaneously: mutual exclusion, hold-and-wait, no preemption, and circular wait. Breaking any one prevents it.

In practice you break circular wait: define a global lock ordering and always take locks in that order. If every code path takes lock A before lock B, a cycle cannot form. It costs nothing at runtime and needs no detection.

The second-line defence is breaking no-preemption with timeouts — try_lock with a deadline, so a thread that cannot progress releases what it holds and retries.

The details that separate candidates:

• Naming which condition you would break, and why, rather than reciting all four. The list is easy; the choice is the answer.
• Describe the signature on a live system: service unresponsive, CPU near zero, threads in state S with WCHAN showing futex_wait, and no change across repeated samples. Contrast with D state from Module 02, which is storage and is not killable, and with livelock, which looks similar but burns CPU.
• Name the runtime tool: jstack reports Java-level deadlocks explicitly; kill -QUIT dumps goroutine stacks in Go; py-spy dump for Python.
• Mention how databases handle it: they detect the cycle, abort one transaction, and return a deadlock error the application is expected to catch and retry. That is condition 3 broken deliberately.

Work through the possibilities in order, because each has a distinct signature:

1. Per-thread CPU first. top -H -p <pid> or ps -eLo pid,tid,pcpu,stat,wchan:20 --sort=-pcpu. Process-level CPU is an average and hides a single hot thread completely.
2. Lock contention. Threads in S with WCHAN of futex_wait, and a futex-dominated call count. The threads are queueing, not computing — adding CPU will not help.
3. Blocked on I/O. Threads in D state, high iowait. That is Module 02's territory, not a locking problem.
4. Deadlock. Same as case 2 but it never recovers. Sample twice, thirty seconds apart, and see whether anything moved.
5. Too many threads. Beyond core count, extra threads add context switching and contention without adding parallelism. Check nlwp and the involuntary context switch count from Module 02.

The details that separate candidates:

• "CPU is not saturated" is the clue, not the problem. It rules out compute and points at waiting — for a lock, for I/O, or for another service.
• Distinguish contention from deadlock by whether progress is made, and say how you would test it: two samples separated in time.
• Volunteer the caution from Module 04: strace on a busy production process can slow it tenfold, so confirm with perf or an eBPF tool. Knowing when not to use a tool is worth as much as knowing the tool.

a–b, the race. The increment is three steps — read, add, write — and the window is between the read and the write. Any worker that reads during another's window works from a stale value and overwrites it, so updates are lost. Two fixes: wrap the read-modify-write in a lock (flock), or use an operation that is atomic by construction — an atomic increment, a compare-and-swap, or in a database a single UPDATE ... SET n = n + 1 rather than SELECT-then-UPDATE.

c, the deadlock. State S, CPU 0.0. The shells show WCHAN do_wait because they are waiting on their flock children; the children themselves show locks_lock_inode_wait, which is the lock wait. All four conditions are present, and the one to break is circular wait: make both workers take cap_A before cap_B. A complete answer also names the fallback — timeouts, which break no-preemption and convert a permanent hang into a retry — and distinguishes this from D state (storage, unkillable) and livelock (CPU busy).

d, the average. The process-level %CPU is the sum of every thread's CPU time divided by elapsed time — ps does not divide by thread count — so one thread at ~100% inside a six-thread process still reads as roughly 100%. What the process view cannot tell you is that a single thread is responsible: the thread view shows one thread in Rl near 100% and the rest in Sl at 0.0. Two lessons follow. A process pinned at 100% on a many-core host looks like a quiet machine in host-level metrics. And a thread capped at exactly one core is the signature of a serialised workload, which no amount of extra CPU will fix.

e, the ceilings. Three: the per-user limit (ulimit -u / RLIMIT_NPROC), the system-wide limit (/proc/sys/kernel/threads-max, and relatedly kernel.pid_max), and memory for the 8 MB-per-thread stack reservations. For a systemd service the one that applies is TasksMax in the unit file — /etc/security/limits.conf does not apply to services systemd starts, which is a very common misdiagnosis.