Module 11 — IPC: Pipes, Sockets & Shared Memory
Updated 22 August 2026
Every module so far has treated processes as isolated. They are not — they talk constantly, and almost all of it goes through three mechanisms. This module covers what a pipe really is, why a socket is just a file descriptor, how shared memory avoids copying entirely, and what is actually happening when a program is stuck waiting for something that never arrives.
🧠 concept → 🧩 real-world analogy → 🧪 exercise → ✅ expected result (hidden) → 🎯 interview questions (hidden)
From Module 02 — fork, exec, and that a child inherits its parent's open files.
From Module 03 — file descriptors, the lowest-unused-fd rule, redirection, inodes, and RLIMIT_NOFILE.
From Module 04 — signals and dispositions, and that SIGPIPE's default action is to kill the process.
From Module 08 — that two processes can map the same physical page.
From Module 09 — what Shmem is, and why memory counted inside Cached is not always reclaimable.
Tools used here that are not installed by default on Ubuntu: sudo apt-get install -y lsof strace socat netcat-openbsd.
🪠 Part A · Pipes
A1 · What a pipe actually is
A pipe is a fixed-size buffer in kernel memory with a file descriptor at each end. That is the whole thing. One descriptor can only be written to, the other can only be read from, and the data never touches a disk.
When you type ls | wc -l, the shell calls pipe() to create that buffer, then forks twice. In the first child it makes the pipe's write end become file descriptor 1 and runs ls; in the second it makes the read end become descriptor 0 and runs wc. Neither program knows a pipe is involved — ls writes to stdout exactly as it always does. This is the redirection mechanism from Module 03, applied to a pipe instead of a file.
Two consequences that surprise people:
The stages run at the same time. ls does not finish and then hand its output to wc. Both run concurrently, and the pipe buffer is what lets them run at different speeds.
A pipe has an inode, but no name. ls -l /proc/PID/fd shows entries like pipe:[482913]. That number is an inode in a virtual filesystem, and it is the only way to work out which two processes are connected — matching inode numbers is how you find the other end.
A FIFO, also called a named pipe, is the same object with a filename attached so that unrelated processes can find it. The name is a doorway; the data still lives only in kernel memory and never touches the disk the name sits on.
A kitchen and a dining room share a hatch in the wall with a shelf in it. The chef puts plates on the shelf; the waiter takes them off. Neither ever sees the other.
The shelf holds a fixed number of plates, and that single fact produces almost all pipe behaviour. If the chef is faster, the shelf fills and the chef has to stop and wait — not because anything is broken, but because there is nowhere to put the next plate. If the waiter is faster, the shelf empties and the waiter waits.
Both work at once. The chef does not cook the entire service and then call the waiter. That is why a pipeline starts producing output immediately instead of after the first command finishes.
An anonymous pipe is a hatch built into the wall between two specific rooms when they were built — only those two rooms can use it, and it exists only while they do. A FIFO is a hatch with a label on the corridor side, so anyone who knows the label can walk up and use it. The label is on the wall; the plates are still only ever on the shelf.
Where the analogy stops working. A waiter can see how full the shelf is. Neither end of a pipe can ask how much data is buffered — you find out only by blocking.
🧪 Exercise A1.1 — Find both ends of a pipe
# A pipeline whose two halves stay alive so we can inspect them
( sleep 60 | sleep 60 ) &
sleep 1
# Find the two processes
pgrep -a sleep | tail -2
P1=$(pgrep sleep | tail -2 | head -1)
P2=$(pgrep sleep | tail -1)
# Their file descriptors. Look for pipe:[NNNN].
echo "--- writer (PID $P1) ---"
ls -l /proc/$P1/fd
echo "--- reader (PID $P2) ---"
ls -l /proc/$P2/fd
# The inode number is the join key. Find everything using it.
INODE=$(readlink /proc/$P1/fd/1 | sed 's/[^0-9]//g')
echo "--- everything holding pipe inode $INODE ---"
# Compare each symlink target directly. `ls -l` over many directories prints
# a header per directory, and grep would throw those headers away - leaving
# you knowing that two descriptors exist but not which processes hold them.
for f in /proc/[0-9]*/fd/*; do
[ "$(readlink "$f" 2>/dev/null)" = "pipe:[$INODE]" ] && echo "$f"
done 2>/dev/null
# lsof does the same job in one command, and names the processes
sudo lsof 2>/dev/null | awk -v i="$INODE" '$0 ~ ("FIFO") && $0 ~ i'
kill $P1 $P2 2>/dev/null✅ Expected result — click to reveal
7412 sleep 60
7413 sleep 60
--- writer (PID 7412) ---
total 0
lrwx------ 1 zaeem zaeem 64 Aug 21 14:02 0 -> /dev/pts/0
l-wx------ 1 zaeem zaeem 64 Aug 21 14:02 1 -> 'pipe:[482913]'
lrwx------ 1 zaeem zaeem 64 Aug 21 14:02 2 -> /dev/pts/0
--- reader (PID 7413) ---
total 0
lr-x------ 1 zaeem zaeem 64 Aug 21 14:02 0 -> 'pipe:[482913]'
lrwx------ 1 zaeem zaeem 64 Aug 21 14:02 1 -> /dev/pts/0
lrwx------ 1 zaeem zaeem 64 Aug 21 14:02 2 -> /dev/pts/0
--- everything holding pipe inode 482913 ---
/proc/7412/fd/1
/proc/7413/fd/0
sleep 7412 zaeem 1w FIFO 0,14 0t0 482913 pipe
sleep 7413 zaeem 0r FIFO 0,14 0t0 482913 pipeWhat to read out of this.
Both processes are running the identical command, sleep 60. What makes one the writer and one the reader is only which descriptor the pipe landed on: 1 for the first, 0 for the second. That is the shell's doing, and it happened before sleep started running.
Look at the permission bits on the symlinks: l-wx for the writer's fd 1 and lr-x for the reader's fd 0. Write-only and read-only. A pipe end is genuinely one-directional — the kernel will refuse a read on the write end with EBADF.
Both point at pipe:[482913], the same inode. That is the buffer. Two descriptors, in two processes, referring to one object in kernel memory.
Notice what is not there: no file, nowhere on any disk, and no entry in any directory. find / -inum 482913 will find nothing, because the inode lives in the anonymous pipefs filesystem that is never mounted anywhere you can see.
The practical value. When a process is stuck reading from a pipe and you need to know who is supposed to be writing to it, this is the procedure: read the inode from /proc/PID/fd/0, then compare every process's descriptor targets against it. The loop prints /proc/7412/fd/1 and /proc/7413/fd/0 — the paths carry the PIDs, which is the whole point. lsof does the same job and adds the command names. If nothing else holds the inode, the writer has already exited, and Section A3 explains why the reader is still waiting.
One limit worth stating now. This inode-matching procedure works for anonymous pipes. A FIFO opened by path shows up as the path itself, not as pipe:[N], so for those you search by name instead — sudo lsof /path/to/fifo.
If pgrep sleep picks up other processes, adjust the tail offsets — on a busy machine there may be several.
A2 · Capacity, blocking and atomicity
The buffer has a size, and everything interesting about pipes follows from it.
Capacity is 16 pages — 65,536 bytes on a normal system. It has been 16 pages since Linux 2.6.11. A process can query and change its own pipe's capacity with fcntl's F_GETPIPE_SZ and F_SETPIPE_SZ, but an unprivileged process cannot exceed /proc/sys/fs/pipe-max-size, which defaults to 1 MiB. A process holding CAP_SYS_RESOURCE — one of the fine-grained pieces of root privilege that Module 13 covers — bypasses that ceiling.
Full means the writer blocks. Empty means the reader blocks. That is not an error condition, it is the design: it is how a fast producer is slowed to the speed of a slow consumer, with no coordination code anywhere. This is backpressure, and it is the reason find / | grep something does not consume unbounded memory.
Writes up to PIPE_BUF are atomic. POSIX guarantees that a write of PIPE_BUF bytes or fewer lands as one contiguous run and is never interleaved with another writer's data. On Linux PIPE_BUF is 4096. Above that, the kernel may split your write and another writer's bytes may end up in the middle of it.
The hatch shelf holds sixteen plates. That is capacity.
The chef is faster than the waiter, so the shelf fills up. The chef stands there holding a plate, unable to put it down, until the waiter takes one. Nothing is wrong; the kitchen has simply been slowed to the speed of the dining room. Nobody wrote any code to make that happen — it falls out of the shelf being finite.
Atomicity is a different rule about the same hatch. A single tray, if it is small enough, goes through in one movement and cannot be interrupted halfway. Two chefs pushing small trays through get two intact trays on the other side, in some order.
A tray that is too wide has to go through in pieces — and if the other chef pushes theirs between your pieces, what comes out the other side is half of yours, all of theirs, then the rest of yours. Neither chef did anything wrong and neither can tell it happened.
Where the analogy stops working. A chef can see the shelf is full and go and do something else. A blocked write gives the process no such choice: it is stopped inside the kernel until space appears, unless it deliberately opened the pipe as non-blocking.
🧪 Exercise A2.1 — Measure the pipe's capacity by filling it
# The ceiling an unprivileged process may raise a pipe to
cat /proc/sys/fs/pipe-max-size
# Now measure the DEFAULT capacity. The writer records its progress to a
# file after each 1 KiB write; the reader deliberately does nothing for
# five seconds. Where progress stops is where the pipe filled up.
KB=$(head -c 1024 /dev/zero | tr '\0' 'x')
: > /tmp/progress
(
for i in $(seq 1 200); do
printf '%s' "$KB"
echo "$i" >> /tmp/progress
done
) | ( sleep 5; cat > /dev/null ) &
sleep 2
echo "writer got $(wc -l < /tmp/progress) KiB into the pipe before blocking"
# Which process is blocked, and on what? wchan names the kernel function.
for d in /proc/[0-9]*; do
w=$(cat $d/wchan 2>/dev/null)
# Match the SUFFIX: kernels from 6.15 renamed these to anon_pipe_write.
case "$w" in *pipe_write)
echo "PID ${d#/proc/} state=$(awk '{print $3}' $d/stat) wchan=$w" ;;
esac
done
wait
rm -f /tmp/progress✅ Expected result — click to reveal
1048576
writer got 64 KiB into the pipe before blocking
PID 7841 state=S wchan=pipe_writeWhat to read out of this.
64 KiB. The writer completed exactly sixty-four 1 KiB writes and then stopped. 64 × 1024 = 65,536 bytes, which is the 16-page default from pipe(7). You have just measured a kernel constant with printf and wc.
pipe-max-size reads 1048576 — 1 MiB. That is not the pipe's size; it is the ceiling an unprivileged process may raise its own pipe to with F_SETPIPE_SZ. Programs that move a lot of data through pipes sometimes do exactly that, and it is worth knowing the limit exists before someone asks why their tuning did not take effect.
Now the blocked process. State S — interruptible sleep, from Module 02. Not D, not R. It is not consuming CPU, it is not counted as load in the way a D-state process is, and it will respond to a signal. A blocked pipe writer is invisible in almost every metric: no CPU, no I/O, no memory growth. The only trace is wchan.
wchan reads pipe_write, which names the kernel function the process is parked in. That is the single most useful field for "what is this process actually waiting for", and the loop above is worth remembering — it will also show you pipe_read, wait_for_partner (blocked opening a FIFO), futex_wait (Module 06), and io_schedule (Module 10).
Note the loop matches on the suffix rather than testing for equality. Kernels from 6.15 renamed these functions to anon_pipe_read and anon_pipe_write; Ubuntu 24.04's 6.8 kernel still uses the old names. An equality test would silently print nothing on a newer kernel, which is the worst way for a diagnostic to fail.
If your count reads 65 rather than 64, the reader's sleep started fractionally late and one more write got through. If it reads 200, the reader started draining before the pipe filled — increase the sleep.
🧪 Exercise A2.2 — Break atomicity on purpose
getconf PIPE_BUF /
mkfifo /tmp/atomic
: > /tmp/atomic.out
# CRITICAL: hold a writer reference open for the whole exercise. Without it,
# the first `printf > /tmp/atomic` closes, the FIFO's writer count hits zero,
# `cat` sees EOF and exits - and every later writer then blocks forever
# in open() with no reader.
exec 9<> /tmp/atomic
( timeout 20 cat /tmp/atomic > /tmp/atomic.out ) &
sleep 1
# --- Round 1: 100-byte records, well under PIPE_BUF ---
SMALL_A=$(head -c 100 /dev/zero | tr '\0' 'A')
SMALL_B=$(head -c 100 /dev/zero | tr '\0' 'B')
for i in $(seq 1 40); do
printf '%s\n' "$SMALL_A" > /tmp/atomic &
printf '%s\n' "$SMALL_B" > /tmp/atomic &
done
wait $(jobs -p 2>/dev/null | head -80) 2>/dev/null
sleep 1
echo "small records: $(grep -c 'AB\|BA' /tmp/atomic.out) lines contained both letters"
# --- Round 2: 8 KiB records, twice PIPE_BUF ---
: > /tmp/atomic.out
BIG_A=$(head -c 8192 /dev/zero | tr '\0' 'A')
BIG_B=$(head -c 8192 /dev/zero | tr '\0' 'B')
for i in $(seq 1 40); do
printf '%s\n' "$BIG_A" > /tmp/atomic &
printf '%s\n' "$BIG_B" > /tmp/atomic &
done
wait 2>/dev/null
sleep 1
echo "large records: $(grep -c 'AB\|BA' /tmp/atomic.out) lines contained both letters"
exec 9>&-
rm -f /tmp/atomic /tmp/atomic.out✅ Expected result — click to reveal
4096
small records: 0 lines contained both letters
large records: 3 lines contained both lettersWhat to read out of this.
getconf PIPE_BUF / reports 4096. That is the guarantee boundary, and POSIX requires it to be at least 512 — Linux gives 4096.
With 100-byte records, not one line was corrupted across eighty concurrent writers. Every record arrived whole. That is the atomicity guarantee doing its job, and it is why a design where many processes append short lines to one pipe works reliably.
With 8 KiB records, three lines contained both an A and a B. Those are records that were spliced together mid-write. Three out of eighty — rare, which is exactly what makes this class of bug so hard to pin down. Nobody made an error. No system call failed. No log line records it. The data is simply wrong, in a way that will show up as an unparseable record in a downstream consumer, days later, once in every few million lines.
This is one of the most valuable things in this module, because it explains a whole class of "impossible" bug reports. Multiple containers writing JSON to one log pipe, records occasionally longer than 4 KiB, and one malformed entry a day that nobody can reproduce — this is that bug.
Notice which detail decides it: the record size, not the number of writers. Going from two writers to two hundred does not create the problem, and reducing the writers does not fix it. Keeping every record under 4 KiB does.
A detail worth knowing, because it sharpens the rule. Trace what printf actually does with an 8 KiB string and you see three calls: write(1, ..., 4096), write(1, ..., 4096), write(1, "\n", 1). Bash flushes in 4 KiB chunks, so every individual write here is atomic — the record was spliced because it took three writes, not because the kernel tore one apart. The precise rule is therefore: one write() per record is necessary but not sufficient. The record must also fit within PIPE_BUF.
If your large-record count is 0, the writes happened to serialise — run it again, or raise the loop count. If the small-record count is not 0, check that the cat reader was still running; once it exits, the writes fail rather than interleave.
A3 · How pipes end — EOF one way, SIGPIPE the other
The two ends of a pipe do not behave symmetrically when the other side disappears, and the asymmetry is deliberate.
All writers closed → the reader gets end-of-file. read() returns 0. No error, no signal. The reader has everything there was, and it stops. This is exactly how wc -l knows to print its answer.
All readers closed → the writer gets SIGPIPE. The default action of SIGPIPE is to terminate the process. If the process has arranged to ignore or block the signal, the write() fails with EPIPE instead.
The reasoning is worth stating plainly: a reader with no writer has finished its job, so ending quietly is correct. A writer with no reader is producing output that nothing will ever consume, so continuing is pointless — and the kernel's default is to stop it immediately rather than let it burn CPU forever.
The waiter finishes the service and goes home, closing the hatch from their side. The chef puts a plate on the shelf and it falls straight through onto the floor. There is no point continuing, and the restaurant's rule is that the chef stops work immediately.
Now the other direction. The chef finishes and goes home. The waiter comes to the hatch, finds it empty and the kitchen dark, and knows the service is over. Nothing dramatic happens — they just stop waiting and go and cash up. That is end-of-file.
The asymmetry makes sense once you see it as who still has useful work. A waiter with an empty kitchen is finished. A chef with no waiter is cooking for nobody, and the sooner they are stopped the better.
And the head case in these terms: a waiter who only needed one plate takes it and leaves. The chef, who was prepared to cook all night, is stopped by the hatch closing rather than by anyone telling them to stop.
Where the analogy stops working. A chef would notice the hatch closing. A process finds out only when it next tries to write — so a program that produces output slowly can sit for a long time after its reader has gone, entirely unaware.
🧪 Exercise A3.1 — Kill a process with a pipeline, on purpose
# The classic. yes would run forever; head stops after one line.
yes | head -1
echo "head exit: ${PIPESTATUS[1]} yes exit: ${PIPESTATUS[0]}"
# 141 = 128 + 13, and signal 13 is SIGPIPE
kill -l 13
# The same thing, made explicit and slower so you can watch it
mkfifo /tmp/sig.pipe
( for i in $(seq 1 100000); do echo "line $i"; done > /tmp/sig.pipe
echo "writer exit: $?" ) &
W=$!
( head -3 < /tmp/sig.pipe ) # reader takes 3 lines and leaves
sleep 1
wait $W 2>/dev/null
echo "writer subshell status: $?"
# And the other direction: a reader with no writer sees clean EOF
( echo "one line" > /tmp/sig.pipe ) &
cat /tmp/sig.pipe
echo "cat exit: $? (0 = clean EOF, not an error)"
# The scripting trap
echo "--- with pipefail ---"
bash -c 'set -euo pipefail; yes | head -1 > /dev/null; echo "reached the next line"'
echo "script exit: $?"
rm -f /tmp/sig.pipe✅ Expected result — click to reveal
y
head exit: 0 yes exit: 141
PIPE
line 1
line 2
line 3
writer subshell status: 141
one line
cat exit: 0 (0 = clean EOF, not an error)
--- with pipefail ---
script exit: 141What to read out of this.
yes exited 141, and head exited 0. 141 − 128 = 13, and kill -l 13 confirms signal 13 is PIPE. The successful command in that pipeline killed the other one, and that is the intended behaviour, not a bug.
${PIPESTATUS[@]} is how you see this at all — plain $? after a pipeline gives you only the last command's status, which is head's 0. Every pipeline you have ever run has been hiding the earlier stages' exit codes unless you asked.
Then the explicit version with the FIFO. head -3 read three lines and exited, closing the read end. The writer loop, which intended to produce a hundred thousand lines, was killed on its next echo. Two things follow. The echo "writer exit: $?" line never printed — the subshell was killed before reaching it, which is what SIGPIPE's default disposition means. And wait reports the subshell's status as 141, the same 128 + 13 you just saw from yes, because the subshell itself died of the signal.
The reverse direction is undramatic: cat on a FIFO whose only writer has finished printed the line and exited 0. Clean EOF. No signal, no error, nothing to handle.
And then the trap. set -euo pipefail turned a completely ordinary yes | head -1 into a script exit of 141. The echo "reached the next line" never ran. This is the failure that appears only in CI, only in scripts written carefully enough to use pipefail, and it looks like a mysterious non-zero exit with no error message anywhere.
Two ways to handle it properly. Either accept the status explicitly — yes | head -1 || [ ${PIPESTATUS[0]} -eq 141 ] — or avoid closing the reader early, for instance by using sed -n '1p;1q'... which has exactly the same behaviour, or by reading the whole stream and discarding the rest. There is no way to make it silently disappear, which is why knowing the number 141 by sight is worth more than any workaround.
🎯 Interview questions — Pipes
Q. What is a pipe, and what happens when you run ls | wc -l?
A pipe is a fixed-size buffer in kernel memory with two file descriptors attached — one write-only, one read-only. No file, no disk. The shell calls pipe() to create it, then forks two children: in one it moves the write end onto file descriptor 1 and runs ls, in the other it moves the read end onto descriptor 0 and runs wc. Neither program knows a pipe exists; both are using stdout and stdin exactly as usual.
The two stages run concurrently, not one after the other, and the buffer is what lets them run at different speeds.
The details that separate candidates: naming the buffer size and what it buys you — 65,536 bytes, and when it fills the writer blocks, which is backpressure and is the reason find / | grep x does not consume unbounded memory. And knowing that the pipe has an inode you can see: ls -l /proc/PID/fd shows pipe:[482913], and matching that inode across processes is how you find which two are connected. That is a real production skill, not trivia.
Q. Why does yes | head -1 terminate rather than run forever?
head reads its one line, prints it, and exits, which closes the only read end of the pipe. The next time yes calls write() there is no reader, so the kernel sends it SIGPIPE, whose default action terminates the process. yes exits with status 141 — 128 + 13, signal 13 being SIGPIPE.
The details that separate candidates: stating the asymmetry and why it exists. Losing all writers gives the reader a clean end-of-file, because a reader with nothing left to read has finished. Losing all readers gives the writer a signal, because output nobody will consume is wasted work. Then the practical follow-on: under set -o pipefail that 141 becomes the pipeline's exit status, so set -euo pipefail plus a routine command | head aborts the script — a very common "works locally, fails in CI" bug. And $? alone never shows it; you need ${PIPESTATUS[@]}.
Q. Several processes write to one pipe. Can their records get mixed together?
Only if the records exceed PIPE_BUF, which is 4096 bytes on Linux. POSIX guarantees that a write of PIPE_BUF bytes or fewer is atomic — it lands as one contiguous run and is never interleaved with another writer's data. Above that size the kernel may split the write, and another writer's bytes can end up in the middle of it.
The details that separate candidates: keeping PIPE_BUF (4096) distinct from the pipe's capacity (65,536) — they are different numbers for different things and get conflated constantly. And recognising the production symptom: many writers sharing one log pipe, occasional records over 4 KiB such as stack traces or large JSON payloads, and one unparseable line every few million that nobody can reproduce. The number of writers is irrelevant; only the record size decides it, so the fix is smaller records or a pipe per writer.
🔗 Part B · Sockets
B1 · A socket is a file descriptor with extra rules
socket() returns a file descriptor, and read() and write() work on it. Everything you learned in Module 03 about descriptors applies unchanged: they are inherited across fork, they count against RLIMIT_NOFILE, and they appear in /proc/PID/fd — as socket:[inode], exactly like a pipe.
What sockets add is addressing (who am I talking to), connection state, and a choice about message boundaries.
Two decisions define any socket. The address family says where the other end can be:
| Family | Where the peer is | Named by |
| AF_UNIX | The same machine | A filesystem path, or an abstract name |
| AF_INET / AF_INET6 | Anywhere reachable by IP | An address and a port |
And the type says what the data looks like:
| Type | Boundaries | Connection | Typical use |
| SOCK_STREAM | None — a byte stream | Yes | TCP; most AF_UNIX sockets |
| SOCK_DGRAM | Preserved — each send is one message | No | UDP; syslog over AF_UNIX |
| SOCK_SEQPACKET | Preserved, and ordered and reliable | Yes | AF_UNIX where records matter |
SOCK_STREAM is a phone call. You are connected to one specific person, everything you say arrives in the order you said it, and nothing is lost. But the other person hears a continuous stream of speech — if you pause between two sentences, nothing marks where one ended. If they need to know where your sentences end, you have to say "over". That is framing, and TCP does not do it for you.
SOCK_DGRAM is a postcard. Each one is a separate item with a beginning and an end, and it arrives whole or not at all. You are not connected to anybody; you just address each card. Cards can be lost, and can arrive out of order.
SOCK_SEQPACKET is a numbered parcel on a booked courier: separate items with clear edges, delivered in order, none lost. It is what you would want most of the time, and it is available on AF_UNIX and almost nowhere else, which is why so few people have used it.
Where the analogy stops working. A phone call has an obvious pause you could interpret as a boundary. A byte stream does not even have that — the reader genuinely cannot tell where one write ended, and no amount of care at the sending end changes it.
🧪 Exercise B1.1 — Sockets are just descriptors
# Every listening socket on this machine, with the process that owns it
sudo ss -tlnp | head -8
# Pick one and look at how it appears in /proc
PID=$(sudo ss -tlnp 2>/dev/null | grep -oP 'pid=\K[0-9]+' | head -1)
echo "--- fds of PID $PID ---"
sudo ls -l /proc/$PID/fd | head -12
# Unix sockets exist too, and there are usually far more of them
sudo ss -xl | head -8
echo "TCP listeners : $(sudo ss -tln | tail -n +2 | wc -l)"
echo "Unix listeners: $(sudo ss -xl | tail -n +2 | wc -l)"
# How many descriptors is a busy process holding, and against what limit?
echo "--- fd usage for PID $PID ---"
echo "open fds: $(sudo ls /proc/$PID/fd | wc -l)"
grep -E '^Max open files' /proc/$PID/limits✅ Expected result — click to reveal
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=623,fd=14))
LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=912,fd=3))
LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=1204,fd=6))
--- fds of PID 623 --- (trimmed)
total 0
lr-x------ 1 root root 64 Aug 21 14:20 3 -> /dev/null
lrwx------ 1 root root 64 Aug 21 14:20 12 -> socket:[19284]
lrwx------ 1 root root 64 Aug 21 14:20 13 -> socket:[19285]
lrwx------ 1 root root 64 Aug 21 14:20 14 -> socket:[19286]
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
u_str LISTEN 0 4096 /run/systemd/private 12841 * 0
u_str LISTEN 0 4096 /run/dbus/system_bus_socket 14028 * 0
u_str LISTEN 0 100 /var/run/docker.sock 21044 * 0
TCP listeners : 3
Unix listeners: 41
--- fd usage for PID 623 ---
open fds: 17
Max open files 1024 524288 filesWhat to read out of this.
The sockets appear in /proc/PID/fd as socket:[19286] — the same shape as pipe:[482913] from Exercise A1.1, with an inode you can match across processes. A socket really is a file descriptor; nothing about the descriptor machinery is special.
One cosmetic thing worth knowing so you are not confused switching between the two: ls quotes its symlink targets ('socket:[19286]') only when writing to a terminal. Pipe it through head, as here, and the quotes vanish. Exercise A1.1's output was not piped, which is why it showed them.
Look at the counts: 3 TCP listeners and 41 Unix listeners. On a normal Linux machine, the overwhelming majority of sockets are local, not network. systemd, D-Bus, the container runtime, the journal, the resolver — almost everything that talks to anything else on the box does it over AF_UNIX, and ss -tln alone shows you almost none of it. If your mental model of "sockets" is TCP, you are missing most of what is happening.
Max open files 1024 524288 is the soft and hard RLIMIT_NOFILE from Module 03. The soft limit is what applies; the hard limit is how far the process could raise it itself. This is the number behind "too many open files", and Section D covers what to do when a service hits it.
Note the Send-Q column on the LISTEN rows: 128 for sshd, 511 for nginx, 4096 for systemd-resolved. On a listening socket that column is not a byte count — Section B3 explains what it really is, and it is one of the most misread fields in ss.
B2 · Unix domain sockets in practice
A Unix domain socket connects two processes on the same machine. It is the default choice for local IPC on Linux, and there are three things worth knowing that people routinely get wrong.
They are genuinely faster than TCP over loopback, typically by somewhere between 1.5× and 2.5×, with the largest gains on small messages and connection-heavy workloads. The reason is that the data never enters the network stack: no headers to build or parse, no routing lookup, no netfilter traversal, no connection state machine. It is not because of checksums — Linux already skips checksums on loopback, so anyone offering that as the explanation is repeating something that stopped being true a long time ago.
They come in two flavours of name. A pathname socket is a file on disk, created by bind(), visible to ls, and — importantly — subject to filesystem permissions: you need write and execute on the containing directory to create one, and write permission on the socket itself to connect. An abstract socket has a name beginning with a NUL byte, shown by ss with a leading @. It has no filesystem presence at all, disappears automatically when the last reference closes, and has no permission checks whatsoever — anything in the same network namespace can connect to it. (Namespaces are Module 12; until then, read "the same network namespace" as "anywhere on this machine".)
They can carry file descriptors. Using SCM_RIGHTS as ancillary data on sendmsg, one process can hand an open descriptor to another. The receiver gets a new descriptor number in its own table referring to the same open file description. Up to 253 at a time; beyond that sendmsg fails with EINVAL.
Talking over TCP loopback is posting a letter to yourself: it goes into an envelope, gets an address written on it, is sorted, and comes back to the building it started in. Everything works, and every one of those steps is wasted effort.
A Unix domain socket is handing the document through the internal door. No envelope, no address, no sorting office.
The two kinds of name are two kinds of door. A pathname socket is a door in a corridor with a lock on it: you can see it, you can control who holds a key, and it stays there until someone removes it. An abstract socket is a door with no lock and no sign, whose location is simply known to those who know it — and anyone in the building who knows the location can open it. Obscurity, not security.
SCM_RIGHTS is the part with no postal equivalent at all: you can pass the key to another room through the door. The receiver does not get a copy of the room's contents; they get their own key to the same room, and either party can now change what is in it.
Where the analogy stops working. A key can be copied and the copies drift apart. A passed descriptor refers to the same open file description, so the file offset is shared — read from one and the other's position moves too.
🧪 Exercise B2.1 — Inspect the Unix sockets your machine is already using
# Every Unix socket. Use -a, not -l: `ss -xl` shows only listening STREAM
# sockets and silently omits every datagram socket on the machine.
sudo ss -xa | head -10
# Abstract sockets are the ones whose name starts with @
echo "--- abstract sockets (no filesystem, no permissions) ---"
sudo ss -xa | awk '$5 ~ /^@/ {print $5}' | sort -u | head
echo "--- pathname sockets, with their actual permissions ---"
for s in $(sudo ss -xa | awk '$5 ~ /^\// {print $5}' | sort -u | head -5); do
ls -l "$s" 2>/dev/null
done
# The permission gate is real for pathname sockets. Try one you should not
# be able to use, as an ordinary user.
command -v nc >/dev/null || echo "(install it: sudo apt-get install -y netcat-openbsd)"
ls -l /run/systemd/private 2>/dev/null
timeout 2 nc -U /run/systemd/private </dev/null 2>&1 | head -2
# Compare loopback TCP with a Unix socket. Send everything down ONE
# connection - opening a socat per message measures process startup,
# not the transport, and the two will come out identical.
command -v socat >/dev/null || sudo apt-get install -y socat
DATA=$(head -c 1048576 /dev/zero | tr '\0' 'x')
socat -u UNIX-LISTEN:/tmp/u.sock /dev/null &
sleep 1
time ( for i in $(seq 1 200); do printf '%s' "$DATA"; done | socat -u - UNIX:/tmp/u.sock )
wait %1 2>/dev/null
socat -u TCP-LISTEN:9999,reuseaddr /dev/null &
sleep 1
time ( for i in $(seq 1 200); do printf '%s' "$DATA"; done | socat -u - TCP:127.0.0.1:9999 )
wait %1 2>/dev/null
rm -f /tmp/u.sock✅ Expected result — click to reveal
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
u_str LISTEN 0 4096 /run/systemd/private 12841 * 0
u_str LISTEN 0 4096 @/tmp/.X11-unix/X0 18420 * 0
u_str LISTEN 0 100 /var/run/docker.sock 21044 * 0
u_dgr ESTAB 0 0 /run/systemd/journal/dev-log 12904 * 0
--- abstract sockets (no filesystem, no permissions) ---
@/tmp/.X11-unix/X0
@/tmp/dbus-8kQr3nWp
--- pathname sockets, with their actual permissions ---
srw-rw---- 1 root root 0 Aug 21 09:14 /run/systemd/private
srw-rw---- 1 root docker 0 Aug 21 09:14 /var/run/docker.sock
srw-rw-rw- 1 root root 0 Aug 21 09:14 /run/systemd/journal/dev-log
srw-rw---- 1 root root 0 Aug 21 09:14 /run/systemd/private
nc: /run/systemd/private: Permission denied
real 0m0.724s # Unix socket
real 0m1.318s # TCP loopbackWhat to read out of this.
Note the survey used ss -xa rather than ss -xl. That matters: -l restricts the output to listening sockets, and a datagram socket like /run/systemd/journal/dev-log — which is how everything on the machine sends to syslog — never appears in it. Current iproute2 also renders bound datagram sockets as ESTAB rather than UNCONN, which surprises people expecting the latter.
The ls -l lines start with s — that is the file type character for a socket, alongside - for regular files, d for directories and p for FIFOs from Module 03. Every one shows size 0: like a FIFO, nothing is ever stored there.
The permissions are doing real work. /run/systemd/private is srw-rw---- owned by root:root, and nc as an ordinary user gets Permission denied. That is the kernel enforcing filesystem permissions on a socket connect, exactly as documented. And /var/run/docker.sock being srw-rw---- root:docker is the entire reason "add the user to the docker group" grants what is effectively root — the group owns a socket that accepts commands to run privileged containers.
Now the abstract ones. @/tmp/.X11-unix/X0 has a leading @, which is ss's rendering of the NUL byte that starts an abstract name. There is no file at /tmp/.X11-unix/X0 to ls, no owner, no mode, and no way to restrict it. Any process in the same network namespace can connect. This is why abstract sockets are convenient for things that should be freely reachable and a poor choice for anything that should not.
The timing: 0.72 s over the Unix socket versus 1.32 s over TCP loopback for the same 200 MB — about 1.8× faster. The gap comes from skipping the whole network stack.
The structure of that measurement matters as much as the result. Everything goes down one connection. An earlier version of this exercise opened a fresh socat per message and the two transports came out within 3% of each other — because process startup was essentially 100% of the cost and the transport was noise. A benchmark that spawns a process per operation measures process spawning, whatever it says on the label.
If both timings are nearly equal, you are timing socat startup rather than the transport. Send many messages down one connection, as above, rather than opening one per message.
B3 · Reading socket state with ss
ss replaced netstat and reads the same kernel data far more cheaply. The flags worth knowing are -t TCP, -u UDP, -x Unix, -l listening, -a all, -n numeric, -p process.
The important thing is not the flags. It is that Recv-Q and Send-Q mean two completely different things depending on the socket's state, and nothing in the output tells you which meaning applies.
| State | Recv-Q | Send-Q |
| LISTEN | Connections completed and waiting to be accepted | The maximum the accept queue may hold — the effective backlog |
| ESTAB | Bytes received but not yet read by the application | Bytes sent but not yet acknowledged by the peer |
This is not a documentation quirk; the kernel genuinely reuses the same two fields for both purposes. The operational payoff is large:
- Recv-Q climbing towards Send-Q on a LISTEN row means the application is not calling accept() fast enough. New connections are being dropped. That is a concurrency problem in the service.
- A large Recv-Q on an ESTAB row means data has arrived and the application has not read it — a slow or stuck reader.
- A large Send-Q on an ESTAB row means the peer is not acknowledging — a network problem, or a peer that has stopped reading.
Three different diagnoses from the same two columns.
For Unix sockets, ss -x shows a Peer Address:Port column where the "port" is an inode number, not a port. That inode is how you find the other end: match it against the Local Address:Port inode of another row. Note that ss -xp prints the socket path only for the listening side, so identifying both processes means correlating inodes rather than reading one line.
A LISTEN socket is a reception desk with a waiting area. Send-Q is how many chairs the waiting area has; Recv-Q is how many people are sitting in them right now. When every chair is full, the next arrival is turned away at the door — they do not queue in the street.
An ESTAB socket is a conversation already in progress. Now Recv-Q is how much the other person has said that you have not yet listened to, and Send-Q is how much you have said that they have not confirmed hearing. Same two numbers on the form, entirely different meaning, and the form does not say which one you are looking at.
The somaxconn detail in these terms: you can order as many chairs as you like, but the building has a fire limit on how many may be in that room. Order five hundred, get sixteen, and nobody tells you.
Where the analogy stops working. A receptionist can see the waiting area filling and call for help. A process gets no notification at all that its accept queue is full — connections are dropped silently, and the only trace is a counter in netstat -s.
🧪 Exercise B3.1 — Fill an accept queue and watch connections be dropped
# The kernel-wide cap on any listen() backlog
sysctl net.core.somaxconn
# A listener with a deliberately tiny backlog that NEVER accepts
python3 -c "
import socket, time
s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('127.0.0.1', 0))
s.listen(2) # backlog of 2, and we never call accept()
open('/tmp/port','w').write(str(s.getsockname()[1]))
time.sleep(25)
" &
sleep 2
PORT=$(cat /tmp/port)
echo "listening on port $PORT"
# Before any clients
ss -tln "sport = :$PORT"
# Now open more connections than the backlog allows, and hold them
for i in $(seq 1 6); do
( exec 3<>/dev/tcp/127.0.0.1/$PORT; sleep 20 ) &
done
sleep 2
echo "--- with 6 clients against a backlog of 2 ---"
ss -tln "sport = :$PORT"
# Count only the ones whose handshake really finished. Plain `ss -tn`
# also lists SYN-SENT, which is precisely the state meaning NOT connected.
echo "clients whose connect() completed:"
ss -tn state established "dport = :$PORT" | tail -n +2 | wc -l
echo "clients still in SYN-SENT (dropped or retrying):"
ss -tn state syn-sent "dport = :$PORT" | tail -n +2 | wc -l
# The kernel counts what it dropped
nstat -az TcpExtListenOverflows TcpExtListenDrops 2>/dev/null || \
netstat -s | grep -iE 'listen queue|overflow'
wait
rm -f /tmp/port✅ Expected result — click to reveal
net.core.somaxconn = 4096
listening on port 41273
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 2 127.0.0.1:41273 0.0.0.0:*
--- with 6 clients against a backlog of 2 ---
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 3 2 127.0.0.1:41273 0.0.0.0:*
clients whose connect() completed:
3
clients still in SYN-SENT (dropped or retrying):
3
#kernel
TcpExtListenOverflows 6 0.0
TcpExtListenDrops 6 0.0What to read out of this.
Before any clients: Recv-Q 0, Send-Q 2. Empty waiting area, two chairs. Send-Q on a LISTEN row is the backlog, and here it matches exactly what listen(2) asked for because 2 is well under somaxconn.
With six clients: Recv-Q reads 3. The queue holds backlog + 1 — a long-standing Linux quirk that catches people comparing the number against their configuration. Three connections are complete and waiting for an accept() that will never come.
Now the split that matters. Three clients completed the handshake; three are still in SYN-SENT. The backlog also sizes the SYN queue, so once it was full the later SYNs were simply dropped and those clients will retry and eventually fail outright.
The first three are the dangerous ones. Their connect() returned success — from the client's point of view this is a connection that was accepted and is now being served. They will sit there waiting for a response and time out much later with something unhelpful. The server never saw them at all: they were never accept()ed, so no application-level log will ever mention them and no server-side latency metric will include them.
TcpExtListenOverflows 6 is the kernel counting exactly that. Note the count exceeds the number of clients — each retransmitted handshake re-increments it, so treat it as a rate rather than a headcount. This counter is the single best evidence for "the service is dropping connections under load", and almost nobody scrapes it.
somaxconn reads 4096 here, the modern default. Anything a program passes to listen() above that is silently reduced — so if your web server config says backlog 65535 and Send-Q on its LISTEN row says 4096, that is why.
The diagnosis chain to remember: Recv-Q at or near Send-Q on a LISTEN row, plus a rising ListenOverflows, means the application is not accepting fast enough. Raising the backlog buys a little more buffering; it does not fix the underlying shortage of workers.
If nstat is not installed, sudo apt-get install -y iproute2 provides it; netstat -s | grep -i overflow shows the same counter with net-tools installed.
🎯 Interview questions — Sockets
Q. When would you use a Unix domain socket instead of TCP on localhost?
Whenever both ends are on the same machine and you control the configuration. The data never enters the network stack — no headers to build or parse, no routing lookup, no netfilter traversal, no connection state machine — so it is typically 1.5× to 2.5× faster, with the biggest gains on small messages and short-lived connections. It also gets you access control for free: a pathname socket obeys filesystem permissions, so srw-rw---- root:docker is a real, enforced restriction, whereas a TCP listener on 127.0.0.1 is reachable by every user on the box.
The details that separate candidates: not giving "TCP computes checksums" as the reason. Linux already skips checksums on loopback, so that explanation has been wrong for years; the real cost avoided is the protocol stack itself. The other detail is the abstract namespace — a socket whose name starts with a NUL byte, shown by ss with a leading @. Abstract sockets have no filesystem presence and no permission checks at all, so "we locked it down with file permissions" is simply false for them, and only a network namespace contains them.
Q. What do Recv-Q and Send-Q mean in ss output?
It depends entirely on the socket's state, and the output does not tell you which meaning applies. On a LISTEN socket, Recv-Q is the number of completed connections waiting to be accept()ed and Send-Q is the maximum the accept queue may hold — the effective backlog. On an ESTAB socket, Recv-Q is bytes received that the application has not read and Send-Q is bytes sent that the peer has not acknowledged.
So the same two columns give three different diagnoses: Recv-Q near Send-Q on a listener means the service is not accepting fast enough; a large Recv-Q on an established socket means a slow reader; a large Send-Q on an established socket means the peer or the network is not keeping up.
The details that separate candidates: knowing that the listen backlog is silently capped at net.core.somaxconn, so Send-Q on a LISTEN row is the only reliable way to see what the backlog actually is rather than what was configured. And naming TcpExtListenOverflows from nstat as the counter that proves connections were dropped — because when an accept queue overflows, the client's connect() has already succeeded and the server sees nothing whatsoever.
Q. What is the difference between SOCK_STREAM and SOCK_DGRAM, and why does it matter to application code?
SOCK_STREAM is a connected, reliable, ordered byte stream with no message boundaries. SOCK_DGRAM is connectionless and preserves message boundaries — each send is one receive — but messages can be lost or reordered.
It matters because a stream socket gives no framing. If you write 100 bytes twice, the reader may see 200 in one read(), or 37 then 163. TCP guarantees the bytes arrive in order and complete; it guarantees nothing about how they are grouped. Every protocol built on TCP therefore has to supply its own framing — a length prefix, a delimiter, or fixed-size records.
The details that separate candidates: naming the failure mode. Code that assumes one write equals one read works perfectly in testing, where messages are small and traffic is light, and starts corrupting data under load when the kernel coalesces or splits differently. It is one of the most common bugs in hand-rolled protocols. Mentioning SOCK_SEQPACKET is a further step up: connection-oriented, reliable, ordered and boundary-preserving — everything most people want — available on AF_UNIX and almost nowhere else, which is why so few engineers have used it.
🧠 Part C · Shared memory and waiting
C1 · Shared memory — the one with no copying
Pipes and sockets both copy: your bytes go from your address space into the kernel and out into the other process. Shared memory does not. The same physical pages are mapped into both address spaces — the mechanism from Module 08 — so once it is set up, passing a megabyte costs nothing at all. There is no system call per message, because there are no messages.
That is also the catch. With no kernel in the loop there is nothing arranging turns, so you must do the synchronisation yourself: everything from Module 06 about races, critical sections and futexes is now your problem. A shared counter incremented by two processes without a lock is wrong in exactly the way that module described.
There are two entirely separate APIs, and mixing them up wastes a lot of time:
| System V | POSIX | |
| Create | shmget() with a numeric key | shm_open() with a name, then mmap() |
| Where it lives | A kernel-internal namespace | A file in /dev/shm, which is a tmpfs |
| List it | ipcs -m | ls -l /dev/shm |
| Remove it | ipcrm | rm, or shm_unlink() |
| Is it a file descriptor? | No — so it cannot be used with epoll | Yes |
POSIX is the modern recommendation, and the "is it a descriptor" row is why: a POSIX object obeys every rule you already know — O_CLOEXEC, ordinary permissions, rm, and inspection with standard tools.
A pipe is passing notes under a door: everything you write is copied, and the copy is what the other person reads.
Shared memory is both of you standing at the same whiteboard. Nothing is copied. You write, they read what you wrote, instantly, because there is only one board. Sending a novel costs the same as sending a word.
And there is the problem. With notes under a door, the door imposes an order — one note at a time, whole. At a whiteboard, you can both write in the same place at once, and what remains is neither of your sentences. Nobody is stopping you, and nothing warns you. If you want turns, the two of you have to agree a rule and follow it, and the whiteboard will not help.
The two APIs are two ways of finding the board. POSIX is a board in a labelled room you can walk to, look at, and lock. System V is a board with a numeric ticket, held in a back office, that you cannot see unless you ask the office for a list — and the office keeps its list separately from the room register, which is why looking in one place tells you nothing about the other.
Where the analogy stops working. Two people at a whiteboard can see each other and naturally take turns. Processes have no such awareness at all: they cannot detect a collision even after the fact.
🧪 Exercise C1.1 — Watch shared memory appear in the machine's totals
# /dev/shm is a tmpfs. Files there are RAM, not disk.
findmnt /dev/shm
df -h /dev/shm
echo "--- before ---"
grep -E '^(Shmem|MemAvailable):' /proc/meminfo
# Create a 64 MiB POSIX shared memory object the crude way.
# shm_open() does exactly this: creates a file in /dev/shm.
dd if=/dev/zero of=/dev/shm/demo bs=1M count=64 status=none
ls -l /dev/shm/
echo "--- after creating 64 MiB ---"
grep -E '^(Shmem|MemAvailable):' /proc/meminfo
# Two processes see the same bytes with no copying and no kernel message
printf 'hello from process A' | dd of=/dev/shm/demo conv=notrunc status=none
( head -c 20 /dev/shm/demo; echo " <- read by a different process" )
# System V is a SEPARATE namespace. ipcs cannot see any of the above.
echo "--- ipcs -m (System V only) ---"
ipcs -m
# POSIX named semaphores live here too, as sem.* entries
ls -l /dev/shm/ | grep 'sem\.' || echo "(no POSIX named semaphores in use)"
rm -f /dev/shm/demo
echo "--- after removing it ---"
grep -E '^(Shmem|MemAvailable):' /proc/meminfo✅ Expected result — click to reveal
TARGET SOURCE FSTYPE OPTIONS
/dev/shm tmpfs tmpfs rw,nosuid,nodev,inode64
Filesystem Size Used Avail Use% Mounted on
tmpfs 1.9G 0 1.9G 0% /dev/shm
--- before ---
MemAvailable: 3794944 kB
Shmem: 12928 kB
total 65536
-rw-r--r-- 1 zaeem zaeem 67108864 Aug 21 15:02 demo
--- after creating 64 MiB ---
MemAvailable: 3728384 kB
Shmem: 78464 kB
hello from process A <- read by a different process
--- ipcs -m (System V only) ---
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
(no POSIX named semaphores in use)
--- after removing it ---
MemAvailable: 3793108 kB
Shmem: 12928 kBWhat to read out of this.
findmnt confirms /dev/shm is tmpfs — a filesystem that exists only in memory. Writing a file there is allocating RAM, and df showing "1.9G available" is showing you half your machine's memory, not disk.
Shmem went from 12928 kB to 78464 kB, an increase of exactly 64 MiB — the file. And MemAvailable fell by roughly the same amount, which is the important half: the kernel knows this memory cannot simply be dropped. Compare that with ordinary page cache, which grows Cached without moving MemAvailable at all, because it can be discarded at any moment. A file in /dev/shm has no file on disk to be discarded back to.
Treat the two numbers differently, though. Shmem is an accounting entry and it is exact. MemAvailable is a heuristic estimate, so it will not always move by the full amount and it does not always spring back afterwards. If you are graphing one of these, graph Shmem.
The read worked, in a different process, with no messages sent. That is the whole appeal: the bytes were never copied anywhere.
Then the part that catches people out. ipcs -m shows an empty table. There is 64 MiB of shared memory on this machine right now and ipcs reports none of it, because ipcs only knows about System V and this is POSIX. Anyone hunting a shared-memory leak with ipcs alone would find nothing and conclude the machine was clean.
After rm, Shmem and MemAvailable both return exactly to where they started. That is the other half of the lesson: /dev/shm objects have kernel persistence. They survive the death of every process that created or used them, and only rm or shm_unlink() frees the memory. A crashed process that had created a large object leaves it behind, permanently, until someone notices.
If your df -h /dev/shm shows a different size, the default is half of RAM; it is set by the size= mount option and containers often set it to 64 MiB, which is a common cause of "no space left on device" from applications that expect more.
C2 · Waiting on many things at once
A server with ten thousand connections has ten thousand descriptors and cannot afford a thread for each. It needs to ask the kernel one question: which of these are ready?
Three answers exist, and the difference between them is why modern servers can hold a hundred thousand connections on one core.
| select | poll | epoll | |
| Descriptor limit | FD_SETSIZE, 1024 — a compile-time constant | None | None |
| Cost per call | O(n) — the whole set is copied in and scanned | O(n) — same | O(ready) |
| Where the interest set lives | Rebuilt and copied every call | Rebuilt and copied every call | Kept in the kernel between calls |
| Portable? | Everywhere | Everywhere | Linux only |
The key idea in epoll is that you register your interest once, with epoll_ctl, and the kernel keeps that set. epoll_wait then returns only the descriptors that are ready. Ten thousand idle connections cost nothing per call, because nothing is copied and nothing is scanned.
epoll has two modes. Level-triggered is the default and behaves like a faster poll: as long as data is available, every call reports it. Edge-triggered (EPOLLET) reports only changes, so you must drain the descriptor completely before returning to epoll_wait — otherwise the remaining data sits there and you are never told about it again. It is faster and it is where most epoll bugs come from.
A porter is responsible for two hundred rooms and must respond when a guest needs something.
select is walking the corridor and knocking on every door in turn to ask whether anyone needs anything. With two hundred rooms and two guests awake, you knock a hundred and ninety-eight times for nothing. And the porter's clipboard has room for exactly a thousand entries, so a bigger hotel simply cannot be managed this way.
poll is the same walk with a bigger clipboard. The limit is gone; the walking is not.
epoll is a bell board behind the desk. Each room is wired up once. The porter sits down, and when a bell rings the board shows which room. Two hundred silent rooms cost nothing, and the work is proportional to how many guests actually ring — not to how many rooms exist.
Level-triggered is a bell that keeps ringing while the guest is still waiting. Edge-triggered is a bell that rings once when they press it: quieter, and if you go back to the desk before finishing with that guest, nothing will remind you.
And the thundering herd: if six porters all watch one board, every ring brings all six to the desk and five walk back. EPOLLEXCLUSIVE is the arrangement where the board wakes fewer of them.
Where the analogy stops working. A porter can hear a guest shouting even without a bell. A process registered with edge-triggered epoll genuinely gets no further notification — the data waits indefinitely and the connection appears hung.
🧪 Exercise C2.1 — Find the event loops on your own machine
# epoll instances appear as descriptors pointing at anon_inode:[eventpoll]
echo "--- processes using epoll, by number of instances ---"
# /proc/PID/cmdline is NUL-SEPARATED, so translate the NULs to spaces.
# `tr -d` would delete them and glue every argument together.
sudo sh -c 'for d in /proc/[0-9]*; do
n=$(ls -l $d/fd 2>/dev/null | grep -c "eventpoll")
[ "${n:-0}" -gt 0 ] && printf "%3d %s\n" "$n" "$(tr "\0" " " < $d/cmdline | head -c 45)"
done' | sort -rn | head -8
# What one of them actually contains: which descriptors it is watching
# NOTE: pgrep -x matches /proc/PID/comm, which is truncated to 15 characters,
# so "systemd-journald" (16) never matches. Use the truncated name, or -f.
PID=$(pgrep -x systemd-journal | head -1)
echo "--- fds of systemd-journald (PID $PID) ---"
sudo ls -l /proc/$PID/fd | grep -E 'eventpoll|socket|inotify' | head -6
# Watch a real event loop at work for five seconds
# -w counts WALL-CLOCK time. Without it, strace -c reports CPU time, and a
# call that sleeps for five seconds contributes almost nothing to it.
# Detach with SIGINT rather than SIGTERM, or no summary is printed at all.
echo "--- syscalls in a live event loop ---"
sudo strace -c -w -f \
-e trace=epoll_wait,epoll_pwait,epoll_ctl,poll,select,ppoll,read,write \
-p $PID 2>/tmp/ev.out &
S=$!
sleep 5
sudo kill -INT $S 2>/dev/null
wait $S 2>/dev/null
tail -10 /tmp/ev.out; rm -f /tmp/ev.out
# select's hard limit is a compile-time constant, unrelated to ulimit
echo "FD_SETSIZE is 1024, fixed at compile time"
echo "your fd limit is $(ulimit -n), which select cannot use above 1024"✅ Expected result — click to reveal
--- processes using epoll, by number of instances ---
4 /lib/systemd/systemd --system --deserialize 31
2 /usr/bin/dockerd -H fd:// --containerd=/run/co
1 /lib/systemd/systemd-journald
1 /usr/sbin/sshd -D
1 /usr/bin/containerd
--- fds of systemd-journald (PID 701) ---
lrwx------ 1 root root 64 Aug 21 15:10 3 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Aug 21 15:10 4 -> socket:[21102]
lrwx------ 1 root root 64 Aug 21 15:10 5 -> anon_inode:inotify
lrwx------ 1 root root 64 Aug 21 15:10 9 -> socket:[21284]
--- syscalls in a live event loop ---
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99.81 4.982104 355864 14 epoll_wait
0.11 0.005482 457 12 read
0.08 0.003911 326 12 write
------ ----------- ----------- --------- --------- ----------------
100.00 4.991497 38 total
FD_SETSIZE is 1024, fixed at compile time
your fd limit is 1024, which select cannot use above 1024What to read out of this.
Almost everything long-lived on the machine has an epoll instance. systemd itself has four — it manages units, sockets, timers and signals through separate event loops. This is not an exotic API used by high-performance servers; it is how ordinary Linux daemons are written.
The descriptor list shows the shape of an event loop. anon_inode:[eventpoll] is the epoll instance itself — a descriptor whose only job is to be waited on. Around it sit the things it watches: sockets, and an anon_inode:inotify for filesystem changes. One process, one blocking call, many sources.
The strace summary is the payoff. In five seconds the process made 14 epoll_wait calls and 24 reads and writes — and not one epoll_ctl. The interest set was registered when the daemon started and has not changed since. That is precisely the property select and poll cannot have: they must copy the whole set into the kernel on every call, so a process watching 10,000 descriptors pays for 10,000 whether or not any are active.
Notice that epoll_wait accounts for 99.8% of the wall-clock time — 4.98 of the 5 seconds. That is the process sleeping, not working. A healthy event loop spends nearly all of its life blocked, and a figure like that against epoll_wait is a sign of health rather than a bottleneck.
The -w flag is doing essential work there. Without it, strace -c reports CPU time, in which a call that slept for five seconds contributes approximately nothing — so the same healthy process would show epoll_wait at a fraction of a percent and you would draw the opposite conclusion. Wall clock and CPU time answer different questions, and -c alone gives you the one you probably did not want.
The last two lines are worth internalising. FD_SETSIZE is 1024 and has nothing to do with ulimit -n. Raise your file-descriptor limit to a million and select still cannot watch descriptor 1025 — the behaviour is undefined and, in practice, memory corruption. That is why select(2)'s own manual page recommends poll or epoll instead.
If strace shows ppoll instead of epoll_wait, that daemon uses poll — plenty do, and for a handful of descriptors it is a perfectly reasonable choice.
🎯 Interview questions — Shared memory and event loops
Q. How does shared memory work, and when would you choose it over a socket?
The kernel maps the same physical pages into two processes' address spaces, so both see one copy of the data. After setup there is no system call and no copying per message — writing a megabyte costs the same as writing a byte. That makes it the fastest IPC there is, and the right choice when two processes on one machine exchange large volumes at high frequency: databases and their client libraries, media pipelines, market-data feeds.
The cost is that the kernel is no longer arranging turns, so you must supply your own synchronisation — locks, semaphores, or atomics. A shared counter incremented by two processes without one is simply wrong.
The details that separate candidates: knowing there are two unrelated APIs and that ipcs only shows System V, so a machine with gigabytes in /dev/shm reports nothing from ipcs -m. Then the operational consequence: POSIX objects have kernel persistence, surviving every process that used them, so a crashed process leaves its segment allocated until someone runs rm. And that memory counts as Shmem, appears inside Cached, and is not reclaimable — Module 09's point, and the reason a /dev/shm leak looks like a memory leak that belongs to no process.
Q. What is the difference between select, poll and epoll?
All three answer "which of these descriptors are ready". select is limited to FD_SETSIZE, which is 1024 and fixed at compile time, and it copies and scans the entire descriptor set on every call. poll removes the limit but keeps the per-call copy and scan, so both are O(n) in the number watched. epoll keeps the interest set in the kernel between calls — you register once with epoll_ctl — and epoll_wait returns only the ready descriptors, so the cost is proportional to how many are active, not how many exist.
That is why a server with 100,000 mostly idle connections is practical with epoll and not with the others.
The details that separate candidates: three. FD_SETSIZE is unrelated to ulimit -n — raising the fd limit does not let select watch descriptor 1025; the behaviour is undefined. Level-triggered is the default; edge-triggered (EPOLLET) reports only changes, so you must drain the descriptor fully or you will never be told about the remainder, which is where most epoll bugs live. And EPOLLEXCLUSIVE, since Linux 4.5, mitigates the thundering herd when several processes watch one listening socket — but the manual says "one or more" will be woken, so it reduces rather than eliminates it.
Q. Two processes share a memory region. How do you keep them from corrupting each other's data?
The same way as threads in Module 06, because the problem is identical once the memory is shared: identify the critical sections and make sure only one party is inside at a time. The usual tools are a POSIX semaphore — a named one lives in /dev/shm as a sem.* entry and is visible to both — a mutex placed inside the shared region and initialised as process-shared, or lock-free atomics for simple counters.
The details that separate candidates: naming the failure precisely. A read-modify-write such as count++ is not one operation; two processes can both read the old value and both write the same new one, losing an increment. It is rare, it is load-dependent, and it leaves no trace — the classic race from Module 06 with no thread involved.
Two further points raise the answer. A lock stored inside the shared region must be explicitly marked process-shared, or it silently only works between threads of one process. And there is a robustness problem sockets do not have: if a process dies holding the lock, nothing releases it and the survivors block for ever — which is why robust mutexes exist, and why some systems prefer a socket's simpler failure semantics even at a cost in speed.
🔍 Part D · Diagnosing IPC problems
D1 · "The process is stuck" — find out what it is waiting for
A hung process that is using no CPU is blocked in a system call. The whole diagnostic question is which one, and there are three cheap ways to find out before you reach for a debugger.
- /proc/PID/wchan — the kernel function it is parked in. One word, no privileges beyond reading the file, no impact on the process.
- /proc/PID/stack — the full kernel stack. Needs root, and is more detail than you usually need.
- strace -p PID — shows the syscall it is inside, and will show it returning if it ever does. Costs the process some performance while attached. (Tracing properly is Module 14; here you only need strace -p and the fact that it reports syscalls as they return.)
The wchan values you will meet most often, and what each means:
| wchan | Waiting for | Look at next |
| pipe_read | Data on a pipe that nobody is writing | The pipe's inode — is there still a writer? |
| wait_for_partner | Blocked in open() on a FIFO — the other side has not opened it at all yet | lsof <fifo-path> — who else has it open? |
| pipe_write | Space in a full pipe | The reader — has it stopped consuming? |
| unix_stream_read_generic | A Unix socket peer | ss -x, match the peer inode |
| inet_csk_accept | An incoming connection — this is healthy | Nothing. A server at rest looks like this |
| futex_wait | A lock held by another thread | Module 06 — the other thread |
| io_schedule | Disk I/O | Module 10 — iostat -x |
| do_wait | A child process to exit | The child — Module 02 |
Diagram source
flowchart TD
A["Process hung,<br>no CPU"] --> B["cat /proc/PID/wchan"]
B --> C{"What does it say?"}
C -->|"pipe_read / pipe_write"| D["Find the pipe inode<br>in /proc/PID/fd"]
D --> E{"Any other process<br>holds that inode?"}
E -->|"No"| F["The other end is gone<br>the process will wait forever"]
E -->|"Yes"| G["Look at that process<br>- it is the real problem"]
C -->|"unix_stream_* / inet_*"| H["ss -xp / ss -tnp<br>find the peer"]
C -->|"futex_wait"| I["Module 06<br>lock contention"]
C -->|"io_schedule"| J["Module 10<br>storage"]
C -->|"inet_csk_accept"| K["Healthy idle server<br>nothing to fix"]Someone in the office has not moved for twenty minutes. They are not asleep and they are not busy; they are waiting for something, and the useful question is what.
Asking them is strace: informative, but you interrupt them to do it.
Looking at where they are standing is wchan, and it is usually enough. At the hatch — waiting for a plate that may never come. At the fax machine — waiting on a message from another office. Outside a locked meeting room — waiting for whoever is inside. At reception, by the door — which is not a problem at all; that is what a receptionist does all day.
The last one matters more than it sounds. A great deal of time is wasted investigating processes that are blocked in exactly the way they are supposed to be. inet_csk_accept on a web server is not a symptom; it is the job.
Where the analogy stops working. A person can tell you what they are waiting for and how long they have waited. wchan gives you the location and nothing else — no duration, no counterpart, no history.
🧪 Exercise D1.1 — Create three different hangs and tell them apart
# Two FIFOs, because one process must not drain the other's buffer.
mkfifo /tmp/hang.r /tmp/hang.w
# Hang 1: reading a FIFO that HAS a writer attached but never sends anything.
# We hold a writer fd open ourselves, so the reader's open() succeeds and it
# blocks in read(). Without this it would block in open() instead - a
# different wait entirely, with wchan `wait_for_partner`.
exec 8> /tmp/hang.r
( cat /tmp/hang.r > /dev/null ) &
H1=$!
# Hang 2: writing to a FIFO whose reader opened it and then stopped reading.
# Same trick in reverse: hold the read end open, never read from it.
exec 9< /tmp/hang.w
KB=$(head -c 1024 /dev/zero | tr '\0' 'x')
( for i in $(seq 1 200); do printf '%s' "$KB"; done > /tmp/hang.w ) &
H2=$!
# Hang 3: waiting for a child that never exits
( sleep 300 & wait ) &
H3=$!
sleep 3
for p in $H1 $H2 $H3; do
for d in $(pgrep -P $p) $p; do
w=$(cat /proc/$d/wchan 2>/dev/null)
s=$(awk '{print $3}' /proc/$d/stat 2>/dev/null)
[ -n "$w" ] && printf 'PID %-7s state=%s wchan=%-24s %s\n' \
"$d" "$s" "$w" "$(tr '\0' ' ' < /proc/$d/cmdline | head -c 30)"
done
done
# A FIFO is found by NAME. Unlike an anonymous pipe it does not appear as
# pipe:[N] in /proc/PID/fd - it appears as its path - so the inode-matching
# procedure from Exercise A1.1 does not apply here.
echo "--- who holds each FIFO? ---"
sudo lsof /tmp/hang.r 2>/dev/null
echo "processes holding /tmp/hang.r: $(sudo lsof -t /tmp/hang.r 2>/dev/null | wc -l)"
kill $H1 $H2 $H3 2>/dev/null
exec 8>&- 9<&-
rm -f /tmp/hang.r /tmp/hang.w✅ Expected result — click to reveal
PID 8412 state=S wchan=pipe_read cat /tmp/hang.r
PID 8419 state=S wchan=pipe_write bash
PID 8431 state=S wchan=hrtimer_nanosleep sleep 300
PID 8430 state=S wchan=do_wait bash
--- who holds each FIFO? ---
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bash 8390 zaeem 8w FIFO 0,14 0t0 950361 /tmp/hang.r
cat 8412 zaeem 3r FIFO 0,14 0t0 950361 /tmp/hang.r
processes holding /tmp/hang.r: 2What to read out of this.
Four processes, all in state S, all using no CPU, all looking identical in top. wchan tells them apart in one word each.
pipe_read — this cat is waiting for data on a FIFO. Whether that is a bug depends entirely on whether a writer exists, which is the next command.
pipe_write — this shell filled the 64 KiB buffer from Section A2 and is now stopped mid-printf. Note the shape of the failure: the process is not broken, it has not crashed, and it will resume the instant somebody reads. It simply never will, because the "reader" here only opened the pipe and never read from it. This is the classic production hang: a log consumer that opened its pipe and then got stuck elsewhere, silently freezing every service writing to it.
do_wait — a parent waiting for a child. Perfectly healthy; every shell running a foreground command looks like this. The interesting process is the child, and it is right above: hrtimer_nanosleep, which is sleep doing its job.
That contrast is the point of the exercise. Two of these four are problems and two are normal, and nothing but wchan distinguishes them. No CPU metric, no memory metric, no log line.
Finally the inode check: 2 processes hold that pipe inode, so a writer does exist and the reader is waiting on data that could still arrive. Had that count been 1 — only the reader itself — the writer would already be gone and the cat would wait for ever. That single number is the difference between "be patient" and "kill it".
If wchan reads 0 for a process, it is not sleeping in the kernel at that instant. Sample it again; a busy process will show varying values or none.
D2 · "Too many open files"
Every pipe end, every socket, and every shared-memory object in this module is a file descriptor, so the limit from Module 03 applies to all of them. The error message is the same in two very different situations, and telling them apart is the whole diagnosis:
| Error | Meaning | Limit involved | Where to look |
| EMFILE | This process has too many descriptors open | RLIMIT_NOFILE, per process | /proc/PID/limits, ls /proc/PID/fd | wc -l |
| ENFILE | The whole machine has too many open | fs.file-max | /proc/sys/fs/file-nr |
EMFILE is overwhelmingly the common one, and it almost always means either a leak or a limit that was never raised for a service that legitimately holds many connections.
Four numbers matter, and they are set in four different places:
- ulimit -n — the soft limit for your shell and its children. This is what applies.
- The hard limit — how far a process may raise its own soft limit unaided.
- LimitNOFILE= in a systemd unit — what a service gets, which has nothing to do with your login shell's ulimit. This is the single most common reason a fix "did not work".
- fs.nr_open — the ceiling above which even root cannot raise the hard limit.
Every door you can open needs a key, and you carry them on a ring. RLIMIT_NOFILE is how many keys your ring holds. Ask the porter for one more when it is full and you are refused — that is EMFILE, and it is about you, not the building.
ENFILE is different: the building has run out of blank keys entirely. Nobody can be issued another, no matter how empty their ring. Rare, and a much bigger problem.
The systemd trap in these terms: there is a staff handbook that says how many keys a person may carry, and a separate contract for each department's automated equipment. Amending the handbook changes nothing about the equipment, and the equipment never reads the handbook because it never joined as staff.
And the leak: someone who takes a key for every room they enter and never hands one back will eventually fill their ring, and the last request fails. The failure has nothing to do with the door they were trying to open at that moment, which is why the error message points at an innocent bystander.
Where the analogy stops working. You would notice a heavy key ring. A process gets no warning at all as it approaches its limit — the first sign is a failure, which is exactly why the ratio is worth graphing.
🧪 Exercise D2.1 — Hit the limit deliberately, then find a leak
# The four numbers, from four places
echo "soft limit (applies now) : $(ulimit -Sn)"
echo "hard limit (self-raise to): $(ulimit -Hn)"
echo "system-wide ceiling : $(cat /proc/sys/fs/nr_open)"
echo "machine-wide open files : $(cat /proc/sys/fs/file-nr)"
# Provoke EMFILE on purpose in a subshell with a tiny limit
echo "--- deliberately hitting EMFILE ---"
# Let the shell CHOOSE the descriptor numbers with {fd}. Naming a specific
# number instead - `exec 15</dev/null` - fails with EBADF rather than EMFILE,
# because the open succeeds and it is the dup2 to fd 15 that is rejected.
bash -c '
ulimit -n 15
echo "limit is now $(ulimit -n)"
n=0
while exec {fd}</dev/null 2>/dev/null; do n=$((n + 1)); done
echo "opened $n extra descriptors before EMFILE"
exec {fd}</dev/null # once more, so the real error message prints
' 2>&1 | tail -3
# Who on this machine is holding the most descriptors, and how close to the limit?
echo "--- top descriptor holders ---"
sudo sh -c 'for d in /proc/[0-9]*; do
n=$(ls $d/fd 2>/dev/null | wc -l)
[ "$n" -gt 20 ] && printf "%6d %-22s soft=%s\n" "$n" \
"$(tr "\0" " " < $d/cmdline | head -c 22)" \
"$(awk "/Max open files/ {print \$4}" $d/limits)"
done' | sort -rn | head -6
# What are those descriptors? Group them - a leak is obvious by shape.
BIG=$(sudo sh -c 'for d in /proc/[0-9]*; do
echo "$(ls $d/fd 2>/dev/null | wc -l) ${d#/proc/}"; done' | sort -rn | head -1 | awk '{print $2}')
echo "--- descriptor breakdown for PID $BIG ---"
# tail -n +2 drops the `total 0` line that `ls -l` always prints first.
sudo ls -l /proc/$BIG/fd 2>/dev/null | tail -n +2 | awk '{print $NF}' | \
sed 's/\[[0-9]*\]/[N]/' | sort | uniq -c | sort -rn | head -8✅ Expected result — click to reveal
soft limit (applies now) : 1024
hard limit (self-raise to): 524288
system-wide ceiling : 1073741816
machine-wide open files : 3488 0 9223372036854775807
--- deliberately hitting EMFILE ---
limit is now 15
opened 5 extra descriptors before EMFILE
bash: /dev/null: Too many open files
--- top descriptor holders ---
184 nginx: worker process soft=65535
97 /lib/systemd/systemd soft=1024
41 /usr/bin/dockerd soft=1048576
28 /usr/bin/containerd soft=1048576
--- descriptor breakdown for PID 1204 ---
142 socket:[N]
21 /var/log/nginx/access.log
8 anon_inode:[eventpoll]
6 pipe:[N]
3 /dev/nullWhat to read out of this.
The four numbers come from four different places and mean four different things. The soft limit of 1024 is what applies right now; the hard limit of 524288 is how far this process could raise itself without privileges — a change systemd made years ago, and the reason many services can fix their own limit at start-up. nr_open at 1073741816 is the absolute ceiling even root cannot exceed.
/proc/sys/fs/file-nr reads 3488 0 9223372036854775807: files currently allocated, files free (always 0 on modern kernels), and the system-wide maximum. That last figure being effectively unlimited is why ENFILE is now rare and why almost every "too many open files" is per-process.
The deliberate failure is worth reading carefully. With a limit of 15, the shell managed only five more descriptors before EMFILE. The limit is a total count, not a quota of new opens — the process already held 0, 1, 2 plus a handful of its own, and those all count. People sizing limits routinely forget the baseline, and then wonder why a service configured for "10,000 connections" falls over at nine and a half thousand.
There is a second trap hidden in how you write the test. exec 15</dev/null does not produce EMFILE — the open() succeeds and it is the dup2() onto descriptor 15 that fails, giving EBADF and the confusing message Bad file descriptor. Genuine EMFILE comes from open() itself when the count is exhausted, which is why the loop lets the shell allocate with {fd}.
Then the fleet view. nginx holds 184 descriptors against a soft limit of 65535, which is a service whose unit file was configured deliberately. systemd holds 97 against 1024. The number that matters is neither of these — it is the ratio, and graphing open / soft_limit per service is the cheapest early warning there is.
The breakdown is where leaks announce themselves. 142 sockets for an nginx worker is normal — that is its connection count. What is not normal is 21 descriptors on the same log file: opening a file once per request and never closing is the textbook leak, and it is instantly visible in this shape of output. A healthy process shows a handful of long-lived files and a large, varying number of sockets; a leaking one shows the same path repeated dozens of times.
If the sudo ls -l breakdown is empty, the largest holder is a kernel thread with no descriptors — skip it and take the next one down.
🎯 Interview questions — Diagnosis
Q. A process is hung and using no CPU. How do you find out what it is waiting for?
cat /proc/PID/wchan first — it names the kernel function the process is parked in, costs nothing, needs no privileges and does not disturb the process. pipe_read and pipe_write mean a pipe, unix_stream_read_generic a Unix socket peer, futex_wait a lock held by another thread, io_schedule disk I/O, do_wait a child. If I need more, /proc/PID/stack as root gives the full kernel stack, and strace -p shows the syscall — at the cost of slowing the process while attached.
Then I follow the wait to its counterpart. For a pipe: read the inode from /proc/PID/fd, and search every process's descriptors for the same inode. If nothing else holds it, the other end is gone and the process will wait for ever.
The details that separate candidates: knowing which wchan values are healthy. inet_csk_accept on a server, or do_wait on a shell, is the process doing exactly its job, and a great deal of investigation time is wasted on processes blocked correctly. The second detail is pipe_write as a production pattern: a stalled log consumer silently freezes every service writing to it, and the producers show no CPU, no errors and nothing in their logs — because the log line they are trying to emit is the one that will not fit.
Q. A service is failing with "too many open files". Walk me through it.
First I check whether it is per-process or machine-wide. EMFILE is the process's own RLIMIT_NOFILE; ENFILE is fs.file-max for the whole machine and is rare on modern kernels, where the system-wide maximum is effectively unlimited. /proc/sys/fs/file-nr distinguishes them in one read.
Then, for the process: ls /proc/PID/fd | wc -l against Max open files in /proc/PID/limits, which shows the limit the process actually has rather than the one someone believes it has. Then I group the descriptors by target — ls -l /proc/PID/fd | awk '{print $NF}' | sort | uniq -c | sort -rn. A leak is obvious by shape: the same file path repeated dozens of times, or sockets to one peer that are never closed.
The details that separate candidates: the systemd trap. /etc/security/limits.conf is applied by PAM at login, so it has no effect on services, which take LimitNOFILE= from their unit. Someone can edit limits.conf, log out and in, see the new ulimit -n, and the daemon still fails — the fix is systemctl edit, daemon-reload and a restart, verified by reading /proc/PID/limits of the running process. The other detail is that the limit counts all descriptors including inherited ones, so "I only open 900 files and the limit is 1024" ignores the baseline the process started with.
🏁 Part E · Practice, docs and self-check
E1 · Production practice
| Symptom in production | What is really happening | What to run | The fix |
| Script exits 141 in CI, works locally | SIGPIPE — a downstream reader closed early, and pipefail propagated it | ${PIPESTATUS[@]}; kill -l 13 | Handle the case explicitly; do not abandon pipefail |
| Service frozen, no CPU, nothing in its logs | Blocked writing to a pipe whose consumer stalled | cat /proc/PID/wchan → pipe_write | Fix the consumer; the producers unblock on their own |
| Shell script hangs forever with no output | Reading a FIFO with no data, or opening one with no writer at all | wchan → pipe_read (reading) or wait_for_partner (opening); then lsof <path> | Start the writer, or open non-blocking |
| Rare unparseable log line nobody can reproduce | Records over PIPE_BUF (4096) interleaved between writers | Measure record sizes; count writers on one pipe | Keep records under 4 KiB, or one pipe per writer |
| Protocol works in test, corrupts under load | Code assuming one write equals one read on a stream socket | Read the framing code | Length prefix, delimiter, or fixed-size records |
| Clients time out; the server logs nothing at all | Accept queue full — connections completed but never accepted | ss -tln Recv-Q vs Send-Q; nstat TcpExtListenOverflows | More workers. Raising the backlog only buffers longer |
| "We set backlog 65535" and nothing changed | Silently capped at net.core.somaxconn | Send-Q on the LISTEN row is the real value | Raise somaxconn too, or accept the cap |
| Local socket secured "with file permissions", but anyone can connect | It is an abstract socket — no filesystem, no permissions | ss -xl | awk '$5 ~ /^@/' | Move to a pathname socket in a restricted directory |
| Memory disappears and belongs to no process | A leaked /dev/shm object, with kernel persistence | ls -l /dev/shm; Shmem in /proc/meminfo | rm it; fix the cleanup path in the application |
| Hunting a shared-memory leak with ipcs finds nothing | ipcs shows System V only; the objects are POSIX | Both ipcs -m and ls -l /dev/shm | Check both, always |
| "Too many open files" after raising limits.conf | PAM applies that at login; services take LimitNOFILE= | /proc/PID/limits of the running process | systemctl edit, daemon-reload, restart |
| Container app fails writing to /dev/shm | Container default /dev/shm is small, not half of RAM | df -h /dev/shm inside the container | Raise the container's shm size |
E2 · Capstone — four IPC tickets
Ticket 1. A nightly job has hung. It uses no CPU, no memory is growing, strace shows nothing happening, and there is no output in its log. It has been like this for four hours. How do you find out what it is waiting for, and how do you decide whether to kill it?
Ticket 2. Users report intermittent connection timeouts to a service. The service's own logs show no errors, no slow requests, and its latency metrics are healthy. CPU and memory on the host are fine. Where do you look?
Ticket 3. A log-processing pipeline occasionally emits a malformed JSON record — perhaps one in five million. It cannot be reproduced. The team has rewritten the JSON serialiser twice. Multiple containers write to a shared pipe consumed by one collector. What is happening?
Ticket 4. A host shows 6 GB of memory used that no process accounts for. The sum of every process's Pss is 900 MB. ipcs -m shows nothing. There have been no OOM kills. Where is the memory?
✅ Ticket 1 — worked answer
Run, in order:
cat /proc/$PID/wchan # one word, no impact
awk '{print $3}' /proc/$PID/stat # state: S, D or R
ls -l /proc/$PID/fd # what is it holding?
sudo cat /proc/$PID/stack # full detail if wchan is ambiguousWhat each answer means. If wchan is pipe_read, it is waiting for data. If pipe_write, it filled a pipe and its consumer stopped. If unix_stream_read_generic or an inet_ function, it is waiting on a peer. If futex_wait, it is Module 06 territory — another thread holds a lock. If io_schedule, it is Module 10 and the state will be D.
The decision of whether to kill it comes from the counterpart, not from the process itself. For a pipe, read the inode from /proc/PID/fd/0 and search every process's descriptors for it:
INODE=$(readlink /proc/$PID/fd/0 | sed 's/[^0-9]//g')
sudo ls -l /proc/[0-9]*/fd 2>/dev/null | grep "pipe:\[$INODE\]" | wc -lIf the count is 1 — only this process — the other end has exited and nothing will ever arrive. Waiting longer is pointless; kill it and fix the job. If the count is 2 or more, a writer still exists, and the real investigation is that process, not this one.
Why strace showed nothing is worth understanding rather than treating as a dead end: strace reports syscalls as they return. A process blocked inside one has already made the call and is producing no events at all. That silence is itself the diagnosis — it confirms a block rather than a loop.
Note the state letter. S is interruptible and kill will work. D is uninterruptible and it will not, which redirects the whole investigation to storage.
✅ Ticket 2 — worked answer
The shape of the symptom is the clue. Clients time out, the server logs nothing, and server-side latency is healthy. Those cannot all be true of requests the application actually saw — so the likeliest answer is that it never saw them. Connections completed the TCP handshake and sat in the accept queue until they were dropped.
Run, in order:
ss -tln 'sport = :443' # Recv-Q vs Send-Q on the LISTEN row
nstat -az TcpExtListenOverflows TcpExtListenDrops
sysctl net.core.somaxconn
ss -tn state established 'sport = :443' | wc -lWhat confirms it. Recv-Q at or near Send-Q on the listening socket, and TcpExtListenOverflows climbing between two readings a minute apart. That counter is unambiguous: it increments only when a completed connection arrives and the queue is full.
Why the metrics all look healthy is the important part to explain. The client's connect() succeeded — the handshake finished — so from the client's side this is a request that was accepted and then never answered. The application was never handed the socket, so it logs nothing, times nothing, and its p99 stays perfect. Every application-level metric is blind to this failure by construction.
The fix. Raising the backlog buys more buffering and delays the symptom; it does not add capacity. The real fix is more workers or faster request handling so accept() keeps up. Also check Send-Q against somaxconn — if the unit asks for 65535 and Send-Q says 4096, the configured value was silently capped and someone believes in a backlog they do not have.
✅ Ticket 3 — worked answer
Stop rewriting the serialiser. The serialiser is fine. This is PIPE_BUF.
Writes of 4096 bytes or fewer to a pipe are atomic — POSIX guarantees they are never interleaved with another writer's data. Above that, the kernel may split the write, and another writer's bytes can land in the middle of it. Several containers writing to one shared pipe, with occasional records above 4 KiB — a stack trace, a large payload, a verbose error — produces exactly this: rare, unreproducible, spliced records.
Confirm it by measuring record size, not by looking at the code:
getconf PIPE_BUF /
awk '{ print length($0) }' /var/log/collector.log | sort -n | tail -5
ls -l /proc/<collector-pid>/fd | grep pipe # how many writers share it?If the largest records exceed 4096 and several processes share one pipe, that is the answer.
Two things that will feel like fixes and are not. Reducing the number of writers: the number of writers does not decide this, the record size does — two writers can corrupt just as three hundred can. And retrying or validating on the consumer side: the data is already wrong by then, and you would be discarding real records.
Real fixes, in order of preference. Give each writer its own pipe, which removes the possibility entirely — this is what a container runtime does when it keeps per-container log streams separate. Or keep every record under 4 KiB, truncating stack traces at the source. Or move to a Unix SOCK_SEQPACKET socket, which preserves message boundaries by design and is exactly the tool this situation calls for.
The general lesson worth stating in the postmortem: one write() per record is necessary but not sufficient. It also has to fit in PIPE_BUF.
✅ Ticket 4 — worked answer
ipcs -m showing nothing proves nothing — it only knows System V. The memory is almost certainly POSIX shared memory in /dev/shm, and it belongs to no process, which is why per-process accounting cannot find it.
Run, in order:
grep -E '^(Shmem|Cached|MemAvailable):' /proc/meminfo
ls -lh /dev/shm/
df -h /dev/shm
findmnt -t tmpfs -o TARGET,SIZE,USED,USE%What confirms it. Shmem at around 6 GB, and /dev/shm containing large files. Note that this memory appears inside Cached, so anyone reasoning "cache is reclaimable, so we have plenty free" is wrong by exactly that 6 GB — MemAvailable already excludes it, which is Module 09's point and the reason to trust that number over your own arithmetic.
Why nothing was OOM-killed fits the story: the memory is genuinely allocated and accounted for, the machine simply has less usable RAM than anyone thinks. It will keep working until something needs the missing 6 GB.
Why it is still there. POSIX shared memory has kernel persistence: an object outlives every process that created or used it, and is freed only by shm_unlink() or rm. A process that crashed, or was killed before its cleanup path ran, leaves its segment allocated for ever. Check the file timestamps against your last incident or deploy — they usually name the moment.
Before deleting anything, confirm nothing is using it:
sudo lsof /dev/shm/<file>If no process holds it, rm frees the memory immediately. If something does, you have a live user rather than a leak — and /dev/shm filling up because a legitimate consumer grew is a capacity question, not a cleanup one. Widen the check with findmnt -t tmpfs too: /run and other tmpfs mounts are RAM by the same mechanism and are a common second offender.
E3 · Documentation reference
| Topic | Where to read it | Why this one |
| Everything about pipes | pipe(7) | Capacity, PIPE_BUF, SIGPIPE, EOF — all in one page |
| Creating one | pipe(2) | And pipe2() for O_CLOEXEC |
| Named pipes | fifo(7) · mkfifo(1) | Read the blocking-on-open rules before scripting one |
| Unix domain sockets | unix(7) | Abstract namespace, permissions, SCM_RIGHTS — all here |
| Sockets in general | socket(2) · socket(7) | Types, options, and what each family supports |
| A pre-connected pair | socketpair(2) | A bidirectional pipe, in effect — often what you actually wanted |
| TCP behaviour and tunables | tcp(7) | Including the backlog and somaxconn interaction |
| Inspecting sockets | ss(8) | It documents every flag but never says what the two queue columns hold — Section B3 does |
| POSIX shared memory | shm_overview(7) · shm_open(3) | Explains /dev/shm and kernel persistence |
| System V IPC | sysvipc(7) · shmget(2) · ipcs(1) | The legacy API you will still meet in older software |
| Semaphores | sem_overview(7) | Named ones appear in /dev/shm as sem.* |
| Message queues | mq_overview(7) | States plainly why POSIX beats System V here |
| Event loops | epoll(7) · epoll_ctl(2) | EPOLLEXCLUSIVE is in epoll_ctl, not epoll(7) |
| The older interfaces | select(2) · poll(2) | select(2) recommends against itself — worth reading why |
| Lightweight notification | eventfd(2) | A counter you can epoll on; how threads wake event loops |
| Moving data without copying | splice(2) | Pipe-to-file and file-to-socket with no userspace copy |
| Anonymous shared memory | memfd_create(2) | A shared region with no name and no cleanup problem |
| Descriptor limits | getrlimit(2) · lsof(8) | EMFILE versus ENFILE |
E4 · Self-assessment
Answer these out loud before moving to Module 12. The section to reread is named after each.
- What is a pipe, physically, and what happens when the shell runs a | b? (A1)
- How do you find which two processes are connected by a given pipe? (A1)
- What is a pipe's capacity, and what is PIPE_BUF? One is a buffer size and one is a guarantee — which is which? (A2)
- What does a blocked pipe writer look like in top, ps and your metrics? (A2)
- Why does yes | head -1 stop, and what exit code does yes get? (A3)
- Why do the two ends of a pipe behave differently when the other side closes? (A3)
- What breaks when set -euo pipefail meets command | head -5? (A3)
- Name the three socket types and say which preserves message boundaries. (B1)
- Why does a TCP-based protocol need its own framing? (B1)
- Give the correct reason Unix sockets beat TCP loopback — and the popular wrong one. (B2)
- What is an abstract socket, and why can you not secure one with chmod? (B2)
- What do Recv-Q and Send-Q mean on a LISTEN row versus an ESTAB row? (B3)
- A client's connect() succeeded but the server logged nothing. How is that possible? (B3, E2 ticket 2)
- Why does shared memory need no system call per message, and what does that cost you? (C1)
- Why does ipcs -m sometimes show nothing on a machine using gigabytes of shared memory? (C1)
- What does epoll do that poll cannot, and why does it matter at 10,000 connections? (C2)
- What is the difference between level-triggered and edge-triggered, and which causes more bugs? (C2)
- A process is hung with no CPU. What is the first thing you read? (D1)
- Which wchan values mean the process is perfectly healthy? (D1)
- Why does editing limits.conf fail to fix "too many open files" for a systemd service? (D2)
E5 · Sources
Manual pages
· pipe(7) · pipe(2) · fifo(7) · mkfifo(1)
· unix(7) · socket(2) · socket(7) · socketpair(2) · tcp(7)
· shm_overview(7) · shm_open(3) · sysvipc(7) · shmget(2) · ipcs(1) · sem_overview(7) · mq_overview(7)
· epoll(7) · epoll_ctl(2) · select(2) · poll(2) · eventfd(2)
· ss(8) · lsof(8) · proc(5) · getrlimit(2) · signal(7)
Kernel
· net/ipv4/tcp_diag.c in the kernel source, for the Recv-Q/Send-Q reuse on LISTEN sockets that no manual page documents
Standards