Module 01 — Kernel, User Space & the System Call Boundary

Updated 21 August 2026

Module 01 · Kernel, User Space & the System Call Boundary

There is a line between your programs and the kernel. Almost every permission error, slow service and broken container you will ever debug is about something crossing that line. This module makes the line visible and teaches you to watch the traffic.

🧠 concept → 🧩 real-world analogy → 🧪 exercise → ✅ expected result (hidden) → 🎯 interview questions (hidden)

🧩 Part A · What an operating system actually is

A1 · The three problems an OS solves

Official docs: The Linux kernel user's and administrator's guide · POSIX.1-2024 — Base Specifications Issue 8

You have used an operating system every day for years, which is exactly why the phrase can stay vague. Let us make it precise.

The easiest way to see what an OS does is to imagine a machine without one.

You have a CPU, some memory, an SSD and a network card. You write a program, and it is the only thing on the machine. Now look at what your program has to do for itself:

To read a file, it needs to know the exact commands your specific SSD understands. Buy a different SSD and your program stops working.
There are no files at all. There are only numbered blocks on a disk. "A file" is an idea somebody has to build.
There is no asking for memory. There is one pool of memory addresses, and you have to divide it up yourself and never make a mistake.
And if you want a second program to run, nothing stops the two of them writing over each other's memory or sending conflicting commands to the same disk.

This is not a made-up example. It is what programming was like before operating systems, and it is still what writing code for a small embedded chip is like today.

Every one of those problems is a real job that someone has to do. An operating system is the software that does all of them, so your program does not have to.

Those jobs group into three problems. Every feature in the remaining fourteen modules is an answer to one of them, so it is worth being able to name all three.

Problem	What happens without it	The OS answer	Taught in
1. Multiplexing	A CPU core runs one instruction stream. The first program to start would own the machine until it chose to stop.	Scheduling and time-slicing	Module 07
2. Abstraction	Every program would ship its own driver for every SSD, NIC and filesystem in existence.	System calls, drivers, the virtual filesystem	Modules 01, 03
3. Isolation	Any program could read your SSH private key, or scribble on another program's memory and take the machine down with it.	Privilege levels, virtual memory, credentials	Modules 08, 13

Say it this way in an interview. "An operating system shares limited hardware between many programs, hides the hardware behind one common interface, and keeps programs from interfering with each other or with the hardware."

Three clauses, thirty seconds — and it hands the interviewer the three follow-up topics they were going to ask about anyway: scheduling, system calls and memory.

Here is the shape of the whole system. Everything else in this track is a zoom-in on one box or one arrow.

Diagram source

flowchart TD
    U["USER SPACE<br>your programs<br>bash, nginx, python, ls"]
    K["KERNEL<br>scheduler - memory manager<br>filesystems - drivers - network"]
    H["HARDWARE<br>CPU - RAM - disk - NIC"]
    U -->|"system calls<br>the only doorway"| K
    K -->|"privileged instructions"| H
    H -->|"interrupts"| K
    K -->|"return values and signals"| U

Two details in that diagram are easy to skim past, and both are load-bearing for the rest of the track:

There is no arrow from user space to hardware. Not a narrow one, not a fast path, not a special case for privileged users. If your program touches a disk, a network card or another process's memory, it did so by asking the kernel.
Interrupts arrive at the kernel, never at your program. Hardware has no idea your program exists. It signals the kernel, and the kernel decides which program, if any, should be woken up as a result.

Real-world analogy — a shared office building

Imagine twenty companies renting floors in one building. Without a building manager, each tenant would have to negotiate its own electricity supply, run its own water pipes, and physically stop rivals wandering into its offices. Nothing would work.

The manager solves exactly the three problems above. Multiplexing: one boardroom, twenty tenants, so there is a booking system. Abstraction: you flip a switch and get light — you never learn which substation feeds the building, and the manager can change supplier without telling you. Isolation: your keycard opens your floor and nothing else, and the plant room opens for nobody.

Where the analogy stops working — and this is the interesting part. A building manager is a person. They are always there, walking the corridors, watching. The kernel is not. It only runs when something calls it. The rest of the time it is not doing anything at all. Section A7 explains how it ever gets called back.

🧪 Exercise A1.1 — How much is your OS actually juggling?

bash

# How many processes exist on this machine right now
ps -e --no-headers | wc -l

# How many CPUs are available to run them
nproc

# Snapshot the run queue. Column 'r' = wanting CPU, column 'b' = blocked
vmstat 1 2 | tail -2

✅ Expected result — click to reveal

plain text

$ ps -e --no-headers | wc -l
243

$ nproc
2

$ vmstat 1 2 | tail -2
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 2913204  61440 561288    0    0     1     3   42   71  1  0 99  0  0

What to read out of it — this is the important part.

243 processes, 2 CPUs. That is about 120 processes per CPU, and yet the machine feels completely idle. If sharing the CPU were the whole story, you would expect it to be crawling.

The vmstat line explains why. The first column, r, is 1. That means exactly one process actually wants the CPU right now.

The other 242 want nothing. They are asleep, waiting for something that has not happened yet — a network packet, a timer, a key press.

This is the most useful idea on the page: on a healthy machine, nearly every process is asleep nearly all of the time. The OS is not heroically juggling 243 things. It has noticed that 242 of them are not moving.

The us sy id wa group on the right splits CPU time four ways: user, system, idle, and waiting for I/O. id 99 means the CPU is doing nothing 99% of the time. Module 07 covers that split properly.

Now imagine this at 500 hosts. Suppose someone sets an alert for "more than 200 processes". It would wake you up 500 times for a completely healthy fleet.

Process count on its own tells you almost nothing. The r column and the CPU split are what tell you whether a machine is in trouble. Alerting on the wrong number is one of the most common reasons on-call rotas become miserable.

A2 · Kernel space and user space — a boundary the CPU enforces

Official docs: intro(2) — introduction to system calls · syscalls(2) — the full syscall list · Kernel admin guide

Here is the mechanism, and it is simpler than the jargon suggests.

Modern CPUs can execute in more than one privilege mode. On x86-64 there are four, numbered 0 to 3 and traditionally called rings. Linux uses only two of them:

Ring 0 — the most privileged. Code here may execute any instruction, address any physical memory, and talk to devices. The kernel runs here.
Ring 3 — the least privileged. Code here may not execute privileged instructions, may not address memory the kernel has not mapped for it, and may not touch devices. Every program you have ever run runs here.

Here is the part that matters. The CPU itself refuses privileged instructions from ring 3. The kernel is not doing the checking. There is no if statement somewhere that the kernel could get wrong or forget. The chip stops the instruction before it does anything.

This surprises people, so let us be clear about it. "Kernel space" is not a part of your RAM stick that you could point at. It is not a separate program running next to yours either.

It is two things together: a CPU mode, and a set of addresses that exist inside every process but can only be reached while the CPU is in that mode.

So the kernel is sitting inside your process's own address space the whole time your program runs. Your program just cannot reach it.

This is why crossing into the kernel is so cheap compared with switching to a different program. Nothing gets loaded or unloaded. The CPU simply changes what it is willing to do.

Diagram source

flowchart LR
    subgraph R3["RING 3 - user mode"]
        A["your program"]
    end
    subgraph R0["RING 0 - kernel mode"]
        B["kernel code"]
        C["device drivers"]
    end
    A -->|"syscall instruction<br>ALLOWED - controlled door"| B
    A -.->|"direct hardware access<br>BLOCKED by the CPU"| C
    B --> C

Real-world analogy — the counter at a bank branch

The lobby is ring 3. The area behind the glass, with the vault and the cash drawers, is ring 0. You may stand at the counter and request a withdrawal; you may not walk around the partition and help yourself.

Now notice what actually stops you. It is not the teller being firm with you. It is not a rule in a handbook. It is glass and a locked door. Send the teller home and you still cannot get in.

That is the point of this section. The CPU blocks the instruction. It does not depend on the kernel remembering to check.

And being an important customer does not help. The branch manager can approve a bigger withdrawal, but even they walk to the vault through the same door under the same rules. This is why sudo gets you past the permission check on /dev/mem and the kernel still says no.

Where the analogy stops working. A bank vault is in a different room. Kernel memory is inside your own process the whole time. The locked door is in your own house, and you cannot turn the handle. That is why going through it is so fast — Section C5 measures exactly how fast.

🧪 Exercise A2.1 — Try to walk through the wall (this is meant to fail)

bash

# /dev/mem is a window onto physical memory.
# First as a normal user:
head -c 16 /dev/mem | xxd

# Now as root, reading an address that IS system RAM (1 MB in).
# Root passes the file permission check. Predict whether the read succeeds.
sudo dd if=/dev/mem bs=1 skip=1048576 count=16 2>&1 | tail -2

✅ Expected result — click to reveal

plain text

$ head -c 16 /dev/mem | xxd
head: cannot open '/dev/mem' for reading: Permission denied

$ sudo dd if=/dev/mem bs=1 skip=1048576 count=16 2>&1 | tail -2
dd: error reading '/dev/mem': Operation not permitted
0+0 records in

What to read out of it. Look carefully — the two errors are different, and the difference is the whole lesson.

As a normal user: Permission denied (EACCES). That is ordinary file permissions. /dev/mem is mode 0640, owner root, group kmem. You failed the same check that stops you reading someone else's file.
As root: Operation not permitted (EPERM). You passed the permission check this time. The kernel refused anyway.

The second refusal comes from a kernel build option called CONFIG_STRICT_DEVMEM, which every mainstream distribution turns on.

Here is the key idea: root is a user-space identity. It decides which files you are allowed to open. It does not change which CPU ring your code runs in. Root code still runs in ring 3, like everything else.

This is a real interview separator. Most candidates say "root can do anything."

A strong candidate says: "Root is a user-space identity. It passes permission checks. It does not change the CPU privilege ring, and the kernel can and does refuse root."

Module 13 goes further, with capabilities — a way of splitting root's powers into about forty separate ones.

One detail worth knowing, because it surprises people. On x86, reads of the first megabyte of physical memory are deliberately still allowed for root, because that region traditionally held BIOS data that some tools need. That is why this exercise reads at the 1 MB mark rather than at zero — read offset 0 and you get real bytes, which looks like the protection is not working.

The refusal appears as soon as you read an address that is genuinely System RAM. Notice also where it appears: the open succeeded and the read failed. The kernel let root open the file and then refused the data.

🧪 Exercise A2.2 — Find code that lives on the other side of the boundary

bash

# Every process has a directory under /proc named after its PID.
# /proc/<pid>/exe is a symlink to the program file it is running.

sudo ls -l /proc/1/exe      # PID 1 - the init system, a user-space program
sudo ls -l /proc/2/exe      # PID 2 - on a normal Linux boot, this is kthreadd

# And see how ps renders the two categories differently:
ps -ef | head -6

✅ Expected result — click to reveal

plain text

$ sudo ls -l /proc/1/exe
lrwxrwxrwx 1 root root 0 Aug 17 09:14 /proc/1/exe -> /usr/lib/systemd/systemd

$ sudo ls -l /proc/2/exe
ls: cannot read symbolic link '/proc/2/exe': No such file or directory
lrwxrwxrwx 1 root root 0 Aug 17 09:14 /proc/2/exe

$ ps -ef | head -6
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 09:14 ?        00:00:02 /sbin/init
root           2       0  0 09:14 ?        00:00:00 [kthreadd]
root           3       2  0 09:14 ?        00:00:00 [rcu_gp]
root           4       2  0 09:14 ?        00:00:00 [rcu_par_gp]
root           5       2  0 09:14 ?        00:00:00 [slub_flushwq]

What to read out of it.

PID 1's exe symlink points at a real file on disk. There is a program you could copy, checksum or run.

PID 2's exe symlink cannot be read at all — ls prints an error and then a listing with no arrow. That is not a bug, and it is not a permissions problem, because you ran it as root.

kthreadd has no program file because it is a kernel thread. It is code built into the kernel itself. It gets scheduled like a process so the kernel can manage it, but it never runs in ring 3 and it has no user-space memory of its own. There is no file to point at.

ps tells you the same thing at a glance: a name in square brackets is a kernel thread. You will see dozens — kworker, ksoftirqd, kswapd, jbd2.

Now you know why strace is no use on them, why they have no working directory, and why you do not kill them.

If you ran this inside a Docker container, PID 1 was your own application and PID 2 probably did not exist. That is not this exercise failing — it is a genuinely different view of the machine, and it is the trap you will meet deliberately in Exercise B4.2.

A3 · How you observe the kernel — /proc and /sys

Official docs: proc(5) — the process filesystem · The /proc filesystem — kernel docs · sysfs(5)

Everything from here on depends on being able to see what the kernel is doing, so this section has to come before the interesting parts.

But there is a problem, and Section A2 created it. You are in ring 3. The kernel is in ring 0. You cannot read its variables and you cannot attach a debugger to it.

So how does anyone see what a kernel is doing?

The kernel publishes. It puts its internal state into two special directories that look and behave like ordinary folders. That way you read kernel state with the same open, read and close that you already use for files. No new tool, no new interface.

Path	What it exposes	Rough rule of thumb
/proc	One numbered directory per process, plus system-wide state: CPU, memory, interrupts, mounts, kernel tunables under /proc/sys	Processes and kernel internals. Older, less consistent, human-readable
/sys	The device and driver model: buses, devices, drivers, kernel modules, cgroup and power settings	Hardware and drivers. Newer, strictly one value per file

These are not files, and believing they are will mislead you. Nothing under /proc or /sys exists on your disk. There is no block on the SSD holding /proc/interrupts.

When you open one of these paths, the kernel runs some code and builds the answer right then. The "file" is a live view, made on demand.

That explains two odd things. They all report a size of zero, because the kernel does not know how long the answer will be until it has made it. And if you read one twice a second apart, you get different content, even though nothing wrote to it.

It also means reading them costs something. Reading /proc/PID/smaps on a big process makes the kernel walk through all of that process's memory, which takes real time. A monitoring agent reading all of /proc every second on a busy machine is doing genuine work. This is a well-known cause of "the monitoring is what is making the box slow".

Real-world analogy — the departures board at a railway station

It hangs on the wall and looks like a sign, so people call it a sign. But nothing is printed on it. There is no piece of paper anywhere in that station saying "the 14:32 is delayed by 6 minutes". The board asks a system for the answer and shows it, every time you look up.

That is exactly what /proc is. /proc/uptime looks like a file in a folder. There is no such file anywhere on your disk. The kernel works out the answer when you ask. Look twice and you get two different answers, with nothing having been written in between.

The cost works the same way in both cases. One departures board is free. A thousand screens all asking every second is real work for the system behind them. That is why a monitoring agent reading all of /proc every second can end up as the busiest thing on the machine.

Where the analogy breaks. A departures board only shows you things. Parts of /proc/sys are writable — it is as if writing on the board actually changed the timetable.

🧪 Exercise A3.1 — Prove these are not files

bash

# Every one reports zero bytes - yet clearly has content
ls -l /proc/uptime /proc/meminfo /proc/interrupts
wc -c /proc/uptime

# Read the same 'file' twice, one second apart
cat /proc/uptime; sleep 1; cat /proc/uptime

# What kind of filesystem is this? Note the type, and that there is no device
findmnt /proc
findmnt /sys

# Confirm it is nowhere on your disk
df -h /proc

✅ Expected result — click to reveal

plain text

$ ls -l /proc/uptime /proc/meminfo /proc/interrupts
-r--r--r-- 1 root root 0 Aug 17 11:41 /proc/interrupts
-r--r--r-- 1 root root 0 Aug 17 11:41 /proc/meminfo
-r--r--r-- 1 root root 0 Aug 17 11:41 /proc/uptime

$ wc -c /proc/uptime
16 /proc/uptime

$ cat /proc/uptime; sleep 1; cat /proc/uptime
2554.87 5013.22
2555.90 5015.28

$ findmnt /proc
TARGET SOURCE FSTYPE OPTIONS
/proc  proc   proc   rw,nosuid,nodev,noexec,relatime

$ findmnt /sys
TARGET SOURCE FSTYPE OPTIONS
/sys   sysfs  sysfs  rw,nosuid,nodev,noexec,relatime

$ df -h /proc
Filesystem      Size  Used Avail Use% Mounted on
proc               0     0     0    - /proc

What to read out of it.

ls -l says 0 bytes. wc -c says 16 bytes. Both are correct.

ls asks "how big is this?" before reading, and the kernel has no answer to give. wc just reads until the content runs out, then counts what it got.

A size of zero on something that clearly has content is the clearest sign there is that you are looking at a made-on-demand view rather than a file.

The two cat outputs differ — 2554.87 then 2555.90. Nothing wrote to that file in between. The kernel computed a fresh answer each time.

findmnt names what is going on. The filesystem type is proc and sysfs, and the source is proc, not a disk like /dev/sda2. And df -h reports a size of 0 with nothing used, because there is no space involved at all.

Now imagine this at 500 hosts. Almost every monitoring agent you will ever use — node_exporter, Datadog, Telegraf, the kubelet — is really just a /proc reader on a timer.

Once you know that each read runs kernel code rather than fetching stored bytes, it is obvious why how often you collect, and how much you collect, actually costs something. An agent collecting per-process metrics on a host with 30,000 processes can become the busiest thing on that host.

🧪 Exercise A3.2 — The /proc/self trick, and the shape of a process directory

bash

# /proc/self always means "whichever process is asking"
cat /proc/self/comm
ls /proc/self | grep -E '^(cgroup|cmdline|comm|cwd|environ|exe|fd|limits|maps|mountinfo|ns|oom_score|root|stat|statm|status)$'

# Prove it resolves differently for each reader
cat /proc/self/comm     # answered for 'cat'
bash -c 'cat /proc/self/comm'
readlink /proc/self     # a different PID every single time

# Now the same directory for a process you name explicitly
sleep 60 &
ls -l /proc/$!/cwd /proc/$!/exe /proc/$!/root
kill $!

✅ Expected result — click to reveal

plain text

$ cat /proc/self/comm
cat

$ ls /proc/self | grep -E '^(cgroup|cmdline|comm|cwd|...)$'
cgroup
cmdline
comm
cwd
environ
exe
fd
limits
maps
mountinfo
ns
oom_score
root
stat
statm
status

$ bash -c 'cat /proc/self/comm'
cat

$ readlink /proc/self
5218
$ readlink /proc/self
5219

$ ls -l /proc/5233/cwd /proc/5233/exe /proc/5233/root
lrwxrwxrwx 1 zaeem zaeem 0 Aug 17 11:44 /proc/5233/cwd -> /home/zaeem
lrwxrwxrwx 1 zaeem zaeem 0 Aug 17 11:44 /proc/5233/exe -> /usr/bin/sleep
lrwxrwxrwx 1 zaeem zaeem 0 Aug 17 11:44 /proc/5233/root -> /

What to read out of it.

cat /proc/self/comm printed cat, not bash.

/proc/self does not point at one fixed place. The kernel works out who is asking and points it at them. cat was the one doing the reading, so cat saw itself.

This is the most useful shortcut in /proc, because a script can use it without needing to look up its own PID first.

readlink /proc/self returning a different number each time proves the point mechanically: each invocation is a new process, so the link resolves differently.

Now look at the list of names in that directory. Together they are a map of what a process actually is, and you will meet every one of them in this track:

fd — its open files
maps — its memory layout
status and stat — its state and counters
limits — the ceilings it is allowed to use
cwd, exe, root — where it is, what it runs, what it sees as /

Those last three are live links the kernel keeps up to date. root -> / looks pointless right now. It stops looking pointless when you meet a process whose / is somewhere else entirely, which is exactly what happens in Exercise B4.2.

Interview-grade detail. /proc/PID/exe is a live symlink to the running program, and it survives the file being deleted or replaced. That is how you recover a binary that has been removed from disk while still running (cp /proc/PID/exe /tmp/recovered), and how you check whether a running service is still executing the old code after a package upgrade — a deleted exe target shows as (deleted). That one fact turns "did the patch actually take effect?" from a guess into a command.

A4 · Program, process, and the three resources the OS controls

Official docs: proc(5) — the process filesystem · proc_pid_status(5) · POSIX.1-2024 Base Definitions

We have been using the word process since Exercise A1.1 in the everyday sense. Now we need it properly, because the next three sections talk about the kernel waking processes up and taking the CPU away from them.

Start with the difference that catches people out in interviews:

Term	What it is	How to think about it
Program	A file on disk. Machine instructions and initial data, sitting there doing nothing.	A recipe printed on paper. Passive. One copy.
Process	A running instance of a program: its own address space, its open files, its credentials, its current position in the instructions.	Someone actually cooking from that recipe. Active. Ten people can cook from one recipe at once.

So /usr/bin/bash is one program. If five people are logged in, there are five bash processes. Five separate lots of memory, five separate positions in the same instructions, all sharing one file on disk.

One file, many running copies. That is why the OS has to keep track of processes at all.

To run a process, the kernel gives it three things — and it must be able to take all three back. Every remaining module in this track is about one of these three rows.

Resource	What the process is given	How the kernel keeps control of it	Taught in
CPU time	The illusion of owning a CPU core continuously	A hardware timer interrupt forcibly returns control to the kernel, which may hand the core to someone else	Section A7, Module 07
Memory	The illusion of owning a large, private, contiguous address space starting at zero	The CPU's memory management unit translates every address through tables only the kernel can write	Module 08
I/O and devices	Simple uniform verbs: open, read, write, close	All device access goes through kernel drivers; the process never addresses hardware	Sections A5–A6, Modules 03, 10

The single idea behind all three rows: each one is a lie the kernel tells, backed by hardware that makes the lie unbreakable.

Your process believes it has a CPU to itself. It does not — a timer yanks it away hundreds of times a second.

Your process believes it has a private address space starting at address zero. It does not — the MMU is rewriting every address it issues.

Your process believes it is writing to a disk. It is not — it is handing bytes to the kernel, which may not touch the disk for another thirty seconds.

Saying it this bluntly is useful, because it changes how you see the whole subject.

Operating systems are mostly about which illusions are kept up, what hardware keeps them up, and what happens when one of them slips. And a slipping illusion is exactly what a performance problem is:

A slow disk is the I/O illusion slipping.
Heavy page faulting is the memory illusion slipping.
Processes queuing for CPU is the CPU illusion slipping.

Real-world analogy — a recipe, and a restaurant kitchen at dinner service

A recipe card is a program. It is paper. It does nothing on its own. There is one copy.

A cook actually making that dish right now is a process. Six cooks can work from the same recipe card at the same time, each on a different step, each with their own pans. Six processes, one program.

To cook, each one needs three things from the kitchen, and this is the three-resource table made physical:

A station at the stove — CPU time. There are four burners and nine cooks, so the head chef moves people on and off.
A workbench — memory. Yours, with your ingredients on it, and nobody else reaches across it.
A runner to the walk-in fridge — I/O. You do not go into the cold store yourself. You ask, and someone brings it.

The head chef can take back all three, and that is what makes the kitchen work rather than descend into a brawl.

Where the analogy stops working — and this is the lesson of the box above. A real cook can look around the kitchen. They can see that nine people are sharing four burners.

A process cannot see any of that. It is told it has a burner to itself, a workbench with no size limit, and a fridge that answers instantly. All three are illusions the kernel keeps up. A performance problem is just one of them slipping.

🧪 Exercise A4.1 — See one program become many processes, each holding all three resources

bash

# One program file on disk
ls -l /usr/bin/sleep

# Start three processes from that one program
sleep 300 & sleep 300 & sleep 300 &

# Three processes, one program
ps -eo pid,ppid,comm,etime | grep -w sleep

# Now inspect the three resources for one of them
PID=$(pgrep -n sleep)
echo "--- CPU: how much it has been given, and its state"
ps -o pid,stat,time,pri,ni -p $PID
echo "--- MEMORY: its private address space"
grep -E '^(VmSize|VmRSS|Threads)' /proc/$PID/status
echo "--- I/O: the file descriptors it holds"
ls -l /proc/$PID/fd

kill %1 %2 %3

✅ Expected result — click to reveal

plain text

$ ls -l /usr/bin/sleep
-rwxr-xr-x 1 root root 39256 Mar 31 08:41 /usr/bin/sleep

$ ps -eo pid,ppid,comm,etime | grep -w sleep
   4127    3901 sleep       00:04
   4128    3901 sleep       00:04
   4129    3901 sleep       00:04

--- CPU: how much it has been given, and its state
    PID STAT     TIME PRI  NI
   4129 S    00:00:00  19   0

--- MEMORY: its private address space
VmSize:     2748 kB
VmRSS:      1408 kB
Threads:    1

--- I/O: the file descriptors it holds
total 0
lrwx------ 1 zaeem zaeem 64 Aug 17 11:02 0 -> /dev/pts/0
lrwx------ 1 zaeem zaeem 64 Aug 17 11:02 1 -> /dev/pts/0
lrwx------ 1 zaeem zaeem 64 Aug 17 11:02 2 -> /dev/pts/0

What to read out of it.

One program, three processes. ls -l shows one file of 39 KB. ps shows three different PIDs — 4127, 4128 and 4129 — all running that same file, all started by your shell (3901).

Three separate lives from one file. That is the program-versus-process difference, shown rather than defined.

Now the three resources, one at a time:

CPU. STAT is S — sleeping. TIME is 00:00:00: in four seconds of existence this process has consumed zero measurable CPU time. It was given the right to CPU time and has used almost none, which is the normal condition for nearly every process on a healthy machine — exactly what Exercise A1.1 showed from the other direction.
Memory. VmSize 2748 kB is the size of the address space the process believes it has. VmRSS 1408 kB is how much physical RAM is actually backing it. They are different numbers, and the gap is the memory illusion made visible. Module 08 is built entirely on this distinction; for now notice only that the kernel promised more than it delivered and nothing broke.
I/O. Three descriptors — 0, 1, 2 — all pointing at /dev/pts/0, your terminal. The process has no idea what a terminal is. It has three small integers, and the kernel knows what they mean. Module 03 takes this apart.

Now imagine this at 500 hosts. You patch one program file. Every process started after that gets the fix.

But every process already running carries on using the old code it loaded into memory. It does not notice the file changed.

That is why needrestart and dnf needs-restarting exist. It is why an OpenSSL patch does nothing until services restart. And it is why "we patched it last Tuesday" and "we are still vulnerable" are very often both true at the same time.

A5 · How the kernel actually talks to hardware

Official docs: Driver implementer's API guide · lspci(8) · proc(5)

Section A2 said user space cannot reach hardware. That raises the obvious question this section answers: how does the kernel reach it?

The part of the kernel that talks to hardware is called a device driver.

A driver knows the language of one specific piece of hardware. It offers the rest of the kernel a simple, standard way to ask for things, and it does the translating.

Here is what that looks like in practice. The filesystem code says "read block 4096". It has no idea what kind of disk it is talking to. The driver turns that request into the exact commands that particular disk understands.

Swap the SSD for a SATA disk and only the driver changes. Nothing above it has to be rewritten.

A driver has exactly three ways to reach hardware. There is no fourth.

Mechanism	How it works	What it is good for
Memory-mapped I/O	The device's control registers are wired to appear at physical memory addresses. The driver reads and writes them like ordinary memory — but those accesses reach the device, not RAM.	Everything modern. Fast, and needs no special CPU instructions.
Port I/O	A separate address space reached only by the privileged in and out instructions. x86-specific and largely historical.	Legacy devices: PS/2 keyboard, serial ports, the old interrupt controller.
Interrupts and DMA	The device initiates, rather than the CPU. Covered in full in Section A6.	Anything where waiting for the device would waste the CPU — which is everything real.

Memory-mapped I/O is strange, so read this twice.

A driver writes the number 1 to address 0xfebc0010. No memory is involved. The motherboard sends that address to a network card instead, and the card starts sending data.

The address looks like a memory address. The CPU instruction is an ordinary store instruction. And the result is completely different.

This is why /dev/mem is locked down so hard — you saw it refuse even root in Exercise A2.1. Writing to the wrong address there is not corrupting data. It is sending commands to hardware.

It is also why the kernel simply never puts those addresses into any user process's memory map. Your program cannot reach them even by accident.

Diagram source

flowchart TD
    APP["Your process - ring 3<br>calls write on a socket"]
    SYS["Kernel: socket layer<br>then TCP and IP"]
    DRV["Device driver for THIS card<br>ring 0 - speaks its language"]
    MMIO["MMIO registers<br>physical addresses wired<br>to the device"]
    NIC["Network card"]
    APP -->|"system call"| SYS
    SYS -->|"generic: send this buffer"| DRV
    DRV -->|"device-specific register writes"| MMIO
    MMIO --> NIC
    NIC -.->|"interrupt when done"| DRV

Read that diagram for what it makes replaceable. Everything above the driver is generic — the socket layer has never heard of Intel or Broadcom. Everything below it is specific. That single seam is why Linux runs on a laptop, a router and a mainframe with the same networking code.

Real-world analogy — the interpreter, and the switch wired to reception

For the driver — a translator.

You speak only English. You need to deal with people who speak Japanese, Arabic and Greek. You do not learn three languages. You say what you want, once, in English, and a translator turns it into whatever the other person understands.

Need to talk to someone new? You change the translator. You do not change how you speak.

That is the driver. The kernel says "send this data", once, the same way every time. Each driver turns it into the commands its own card understands.

For memory-mapped I/O — the switch wired to reception.

A hotel room has a row of light switches on the wall. They all look identical. Same plate, same click.

But one of them is not wired to a light. It is wired to reception, and flipping it sends a porter to your room. Same action, same-looking switch, completely different result — and you cannot tell by looking.

That is what happens when a driver writes to 0xfebc0010. An ordinary write, an ordinary-looking address, and a network card starts transmitting.

Where the analogy stops working. A guest could walk over and look at the switch closely. Your program cannot. The kernel never puts those addresses on your wall at all.

🧪 Exercise A5.1 — Look at the real hardware map of your machine

bash

# Physical address ranges claimed by devices for memory-mapped I/O
sudo head -20 /proc/iomem

# The legacy port I/O space, for contrast - note how small and old it looks
sudo head -12 /proc/ioports

# Which driver is bound to which device
lspci -k 2>/dev/null | head -20 || echo "install it: sudo apt install -y pciutils"

✅ Expected result — click to reveal

plain text

$ sudo head -20 /proc/iomem
00000000-00000fff : Reserved
00001000-0009fbff : System RAM
000f0000-000fffff : Reserved
00100000-7ffdbfff : System RAM
  01000000-01e00fff : Kernel code
  02000000-025fffff : Kernel rodata
  02600000-029d1fff : Kernel data
feb80000-febbffff : 0000:00:03.0
  feb80000-febbffff : virtio-pci-modern
febc0000-febc0fff : 0000:00:04.0
  febc0000-febc0fff : virtio-pci-modern
fec00000-fec003ff : IOAPIC 0
fed00000-fed003ff : HPET 0

$ sudo head -12 /proc/ioports
0000-0cf7 : PCI Bus 0000:00
  0000-001f : dma1
  0020-0021 : pic1
  0040-0043 : timer0
  0060-0060 : keyboard
  0064-0064 : keyboard
  0070-0071 : rtc0
  00f0-00ff : fpu

$ lspci -k | head -20
00:03.0 Ethernet controller: Red Hat, Inc. Virtio 1.0 network device
	Kernel driver in use: virtio-pci
00:04.0 SCSI storage controller: Red Hat, Inc. Virtio 1.0 block device
	Kernel driver in use: virtio-pci

What to read out of it.

/proc/iomem is the physical address map of the whole machine, and it is not all RAM. 00100000-7ffdbfff : System RAM is actual memory. But febc0000-febc0fff : virtio-pci-modern is a 4 KB window that is not memory at all — it is a device's control registers, wired into the address space. That is memory-mapped I/O, visible as a range you can point at.

Notice Kernel code, Kernel rodata and Kernel data nested inside System RAM. There is the kernel, occupying specific physical addresses, exactly as Section A2 described it.

/proc/ioports is basically a museum. timer0 at 0040, keyboard at 0060, pic1 at 0020 — these numbers were fixed on the original IBM PC in 1981 and are still reserved today.

That is what "legacy" really means: not old code, but an address nobody is ever allowed to reuse.

lspci -k finishes the picture by showing which driver is handling which device. This is the first command to run when a device is present but not working.

If the Kernel driver in use line is missing, the machine found the hardware and no driver took it on. That is a completely different problem from a broken device, and dmesg will usually explain it in one line.

Now imagine this at 500 hosts. lspci -k is how you find the eleven machines that got a different NIC revision in a later hardware batch and are silently running a different driver — exactly the shape of a bug that appears on 2% of hosts, resists reproduction, and costs a week. Hardware inventory is not paperwork, it is a debugging input.

A6 · Interrupts, polling and DMA — how input and output actually move

Official docs: proc_interrupts(5) · Dynamic DMA mapping guide · Driver implementer's API guide

Here is the problem this section solves. A disk read takes on the order of 100 microseconds on a fast SSD. A CPU executes roughly 300,000 instructions in that time. So what should the CPU do while it waits?

There are only two possible answers, and the second is why computers are usable at all.

Polling — ask repeatedly

The CPU loops, reading a status register over and over until the device says it is done.

Simple, and it burns an entire core doing nothing. 300,000 wasted instructions per disk read, and no other process can run on that core meanwhile.

Still correct in two narrow cases: devices so fast the interrupt would cost more than the wait, and very simple embedded systems.

Interrupts — be told

The driver starts the operation and the kernel puts the process to sleep. The CPU runs someone else entirely.

When the device finishes it raises an electrical signal. The CPU stops mid-stream, enters the kernel, and the kernel wakes the sleeping process.

Nothing is wasted. This is why 243 processes on 2 CPUs felt completely idle in Exercise A1.1.

So an interrupt is hardware forcing the CPU into the kernel. A system call is a program choosing to go into the kernel. Same destination, opposite trigger.

Remember that pair, because those are the only two ways the kernel ever starts running.

That solves half the problem. The device can now tell the CPU when it is finished. But there is a second half: who actually moves the data?

If the CPU had to copy every arriving network packet by hand, a fast network card would use up the whole machine doing nothing but copying.

It does not have to. DMA — Direct Memory Access — lets the device write into memory by itself.

The driver tells the device where to put the data. The device moves it while the CPU gets on with something else. Only when it is finished does it raise an interrupt to say so.

So the CPU is involved at the start and at the end. Never in the middle.

There is an uncomfortable side to DMA, and it is worth knowing.

A device doing DMA writes straight to physical memory. It goes around the protection that keeps processes out of each other's memory. So a faulty or malicious device could in principle write anywhere in RAM.

That is the whole reason the IOMMU exists. It sits between devices and memory and limits what each device is allowed to touch.

This is also why plugging an untrusted Thunderbolt device into a laptop is a real attack and not a theoretical one, and why hypervisors require IOMMU support before letting a virtual machine use a physical device directly.

Now the complete path. This is the answer to "how do input and output devices share information", start to finish. Trace it slowly — several later modules are a zoom-in on one step of it.

Diagram source

sequenceDiagram
    participant P as Process in ring 3
    participant K as Kernel
    participant D as Driver
    participant HW as Device
    P->>K: read syscall - give me data
    K->>D: request the transfer
    D->>HW: program the device via MMIO<br>here is the DMA target address
    K->>P: no data yet - process is put to SLEEP
    Note over K: CPU handed to a different process<br>nothing is wasted
    HW->>HW: fetch the data
    HW-->>K: DMA - device writes straight into RAM
    HW->>K: raise interrupt - transfer complete
    Note over K: TOP HALF: acknowledge fast,<br>do almost nothing
    K->>K: BOTTOM HALF: softirq does the real work
    K->>P: mark the process RUNNABLE again
    Note over P: scheduler resumes it<br>and the read finally returns

Two parts of that diagram deserve naming, because both show up in interviews and in real incidents:

The process sleeps. It does not spin and it does not poll. It is removed from the run queue entirely and consumes no CPU. That is what a blocked process is, and it is why vmstat's r column read 1 while 243 processes existed.
Top half and bottom half. Interrupt handlers run with interrupts disabled on that CPU, so they must be extremely short — miss another interrupt and you lose data. The handler does the bare minimum and defers the real processing to a softirq, which runs with interrupts enabled again. When you see ksoftirqd burning CPU on a busy host, that is the bottom half falling behind — almost always network receive.

Real-world analogy — waiting for a delivery

You are working from home and a parcel is due. Three arrangements, and they map one-to-one onto the three mechanisms:

Polling — you stand at the window all morning watching for the van. You will not miss it. You will also get no work done at all, and the parcel arrives no sooner. That is a CPU spinning on a status register.
Interrupts — you get on with your work and the courier rings the doorbell. You stop mid-sentence, deal with it, and go back. Nothing was wasted. This is why 243 processes on 2 CPUs sat completely idle in Exercise A1.1.
DMA — the courier has the code to your parcel box, loads the contents in himself, and then rings the bell to say it is done. You never carried a box. The CPU is involved at the start and the end, never for the moving of bytes.

Top half and bottom half is the last piece, and it is the most human bit of the whole analogy. When the bell rings you open the door, take the parcel, say thank you and close the door — you do not stand on the doorstep unpacking it, because a second courier could arrive and you would miss them. You unpack afterwards. That is exactly why an interrupt handler acknowledges the device and defers the real work to a softirq: it runs with interrupts disabled, so anything slow means a missed doorbell and lost data.

Where the analogy breaks, and it matters. A courier with a key to your parcel box can only reach the box. A device doing DMA writes to physical memory and bypasses the MMU entirely — it has a key to the whole house. That is the uncomfortable fact the IOMMU exists to fix.

🧪 Exercise A6.1 — Watch interrupts and DMA happen in real time

bash

# Snapshot the interrupt counters
grep -E 'CPU0|LOC|virtio|eth|nvme|ahci' /proc/interrupts

# Generate real device work
ping -c 50 -i 0.02 127.0.0.1 >/dev/null 2>&1
curl -s -o /dev/null https://kernel.org 2>/dev/null || echo "(no external network - fine)"

# Snapshot again and compare which counters moved
grep -E 'CPU0|LOC|virtio|eth|nvme|ahci' /proc/interrupts

# Interrupts per second, live, alongside context switches
vmstat 1 3

✅ Expected result — click to reveal

plain text

$ grep -E 'CPU0|LOC|virtio|nvme' /proc/interrupts
            CPU0       CPU1
  10:      14822          0   PCI-MSI 65536-edge      virtio0-input.0
  11:       2109          0   PCI-MSI 65537-edge      virtio0-output.0
  12:      88431      91204   PCI-MSI 98304-edge      virtio1-req.0
 LOC:     512044     498211   Local timer interrupts
 RES:       4021       3990   Rescheduling interrupts

### after generating traffic
  10:      15104          0   PCI-MSI 65536-edge      virtio0-input.0
  11:       2361          0   PCI-MSI 65537-edge      virtio0-output.0
  12:      88431      91204   PCI-MSI 98304-edge      virtio1-req.0
 LOC:     512340     498507   Local timer interrupts

$ vmstat 1 3
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 2901120  61440 563100    0    0     0     0  118  204  0  1 99  0  0
 1  0      0 2900864  61440 563100    0    0     0    24  892 1640  2  3 95  0  0
 0  0      0 2900864  61440 563100    0    0     0     0  131  219  0  0 99  0  0

What to read out of it.

The network counters moved and the disk counter did not. virtio0-input.0 went from 14,822 to 15,104 — 282 interrupts for 50 pings plus an HTTPS fetch. virtio1-req.0, the block device, is unchanged at 88,431: nothing touched the disk, because everything needed was already in memory. You have just measured which hardware a workload uses, with no application instrumentation whatsoever.

Notice 282 interrupts, not one per packet. Modern NICs coalesce: the card waits a few microseconds to see whether more packets arrive and raises one interrupt for the batch. That is a deliberate latency-for-throughput trade, tunable with ethtool -C, and it is why interrupt count and packet count are never equal.

Now look at the distribution across CPUs. virtio0-input.0 shows 14,822 on CPU0 and 0 on CPU1 — every network interrupt landing on one core. On a busy host that single line is a capacity ceiling you can see coming, and the fix has names: irqbalance, explicit affinity via /proc/irq/N/smp_affinity, or multi-queue with RSS.

LOC — local timer interrupts — is the largest counter by far, and climbing steadily whether or not anything is happening. Around half a million on each CPU already. That counter is the subject of the next section, and it is the mechanism that makes multitasking possible at all.

In vmstat, in is interrupts per second and cs is context switches per second. They rose together during the traffic (118 → 892 interrupts, 204 → 1,640 switches) and fell back afterwards. Interrupt arrives, process wakes, scheduler switches to it — you are watching the sequence diagram above, in aggregate, live.

Now imagine this at 500 hosts. Reading /proc/interrupts twice, ten seconds apart, is one of the fastest diagnostics that exists for "this box is inexplicably busy" — a runaway counter names the guilty device immediately. Two patterns you will genuinely meet: a failing disk generating enormous interrupt counts as it retries, and all network interrupts pinned to CPU0 so one core saturates at 100% while the other 31 idle. That second host is out of capacity at 3% average CPU utilisation, and completely invisible to any dashboard showing a machine-wide average.

A7 · How the OS keeps control of the CPU — the timer interrupt

Official docs: Timers — kernel documentation · proc_interrupts(5) · The kernel's command-line parameters

This is the question most people never think to ask, and it is the foundation everything in Module 07 is built on.

Section A1 said the OS multiplexes the CPU across many programs. Section A4 said the kernel can take CPU time back. But look at what that actually requires. Once the CPU is executing your process's instructions in ring 3, the kernel is not running. It is not watching from somewhere. It is not a background daemon — recall the Part A interview answer: the kernel is not a process. It is inert code that executes only when something forces the CPU into ring 0.

So if your program contains an infinite loop and never makes a system call, what takes the CPU back?

The answer is that software cannot do it. Hardware must.

The kernel sets up a hardware timer to fire an interrupt at regular intervals. When it fires, the CPU is pushed out of ring 3 and into the kernel. It does not matter what your program was doing, and your program is not asked.

That is the LOC counter you watched climbing in Exercise A6.1 — on every CPU, forever.

This one mechanism is what makes preemptive multitasking possible. Without it, a single runaway loop would freeze the machine for good and the only way out would be the power button.

Cooperative multitasking

A process keeps the CPU until it voluntarily yields, usually by making a system call.

One infinite loop hangs the entire machine. One misbehaving application takes everything down with it.

Not hypothetical history: this is how Windows 3.x and classic Mac OS worked, and exactly why those systems froze so memorably.

Preemptive multitasking

A hardware timer interrupt returns control to the kernel on a fixed schedule, whatever the process is doing.

The kernel then decides whether to resume the same process or hand the CPU to another.

An infinite loop wastes CPU but cannot hang the machine — you can still log in and kill it. Every modern OS works this way.

Diagram source

sequenceDiagram
    participant HW as Timer hardware
    participant CPU as CPU
    participant K as Kernel
    participant P as Your process
    P->>CPU: running your code in ring 3
    HW->>CPU: timer interrupt fires
    CPU->>K: FORCED into ring 0 - no cooperation needed
    K->>K: update the clock, charge CPU time to this process
    K->>K: has this process used its share?
    alt share used up
        K->>K: pick another process, switch context
        Note over P: paused - state saved, resumes later
    else share remaining
        K->>P: return to ring 3, carry on
    end

Two clarifications that head off a very common misunderstanding.

The timer does not switch to another program. It only pulls the CPU into the kernel, and the kernel then decides what to do. Most of the time it resumes the very same program.

That decision is scheduling policy, and it belongs to Module 07. The timer is the mechanism, and that is what belongs here.

Modern kernels do not tick all the time. The old design fired at a fixed rate — 100, 250 or 1000 times a second — even on a CPU doing nothing at all.

Today's kernels are mostly tickless. If a CPU is idle, or is running only one program, the tick is switched off so the CPU can go into a deep sleep state and use less power.

That is why LOC climbs more slowly on an idle machine than the configured rate would suggest. At datacentre scale it saves real money.

Real-world analogy — the invigilator's bell versus the meeting that never ends

You have sat in both of these rooms.

Cooperative is a meeting where whoever is speaking carries on until they decide to stop.

Most people are polite, so usually it works. Then one person will not stop, and the whole meeting is stuck. Nobody can do anything, because the only way to get a turn is for the speaker to give it up. That is Windows 3.x. That is one runaway loop freezing a machine.

Preemptive is a timed exam. The bell goes at a fixed time and pens go down in the middle of a sentence. Nobody asks whether you are ready.

Now here is the part that makes the whole section click. The bell cannot be controlled by a student. If it were, the student who refuses to stop would simply never ring it. The bell has to be outside the room, on its own clock.

That is exactly why preemption cannot be done in software. While your program runs, the kernel is not running, so it cannot interrupt anything. Something outside, with its own clock, has to force the door open. That something is the hardware timer.

Where the analogy stops working. The exam bell ends the exam. The timer interrupt does not end anything — it just pulls the kernel in to have a look, and most of the time it hands the CPU straight back to the same program.

Ringing the bell is the mechanism. Deciding who writes next is the policy, and that is Module 07.

🧪 Exercise A7.1 — Prove the kernel can take the CPU back from a program that never asks

bash

# Your configured tick rate
grep -E 'CONFIG_HZ=|CONFIG_NO_HZ' /boot/config-$(uname -r) 2>/dev/null | head -6

# Baseline timer interrupt count
grep LOC /proc/interrupts

# A pure CPU loop that makes ZERO system calls. It will never yield voluntarily.
bash -c 'while :; do :; done' &
SPIN=$!

# The machine is still completely responsive. Prove it.
sleep 3
uptime
ps -o pid,stat,pcpu,comm -p $SPIN

# Timer interrupts kept firing the whole time
grep LOC /proc/interrupts

# How many times did the kernel take the CPU away by force?
grep -E 'voluntary' /proc/$SPIN/status

kill $SPIN

✅ Expected result — click to reveal

plain text

$ grep -E 'CONFIG_HZ=|CONFIG_NO_HZ' /boot/config-6.8.0-45-generic | head -6
CONFIG_NO_HZ_COMMON=y
CONFIG_NO_HZ_IDLE=y
CONFIG_NO_HZ=y
CONFIG_HZ=250

$ grep LOC /proc/interrupts
 LOC:     512044     498211   Local timer interrupts

$ uptime
 11:34:07 up 42 min,  1 user,  load average: 0.71, 0.23, 0.09

$ ps -o pid,stat,pcpu,comm -p 4712
    PID STAT %CPU COMMAND
   4712 R    99.4 bash

$ grep LOC /proc/interrupts
 LOC:     512796     498726   Local timer interrupts

$ grep -E 'voluntary' /proc/4712/status
voluntary_ctxt_switches:        1
nonvoluntary_ctxt_switches:     743

What to read out of it — the last two lines are the whole point of Part A.

That loop made zero system calls. It never asked for anything, never yielded, never cooperated. And yet:

voluntary_ctxt_switches: 1 — it gave up the CPU willingly exactly once, at startup.
nonvoluntary_ctxt_switches: 743 — the kernel took the CPU away by force 743 times in three seconds.

That second number is the timer interrupt doing its job, counted. Nothing in that program consented to any of it. Meanwhile uptime returned instantly and your shell stayed responsive, on a 2-CPU box with a process pinned at 99.4%.

LOC rose by 752 on CPU0 over three seconds — about 250 per second, matching CONFIG_HZ=250 exactly. That is the CPU running the spinner, and it cannot go tickless, because the tick is only suppressed on idle CPUs and this one is anything but.

CPU1 rose by only 515 over the same window, because it was idle for part of it and could skip ticks. Two CPUs, same kernel, different counts — that difference is tickless operation, visible as a number.

STAT is R — runnable — not S. Compare with Exercise A4.1, where sleep sat in S consuming no CPU. Same ps column, opposite meaning, and you can now explain both: S means blocked and off the run queue awaiting an interrupt; R means it wants CPU and the timer is the only thing rationing it.

Interview-grade detail. The voluntary/involuntary split is one of the most useful and least-known diagnostics on Linux, and both numbers are free in /proc/PID/status:

High voluntary switches → the process blocks constantly. It is waiting on I/O, locks or the network. Adding CPU will not help it.

High involuntary switches → the process wants CPU and keeps being preempted. It is competing for a scarce resource. That is a capacity or contention problem.

Those two send you in completely opposite directions, and a dashboard showing only "CPU %" cannot tell them apart. Module 07 builds the full diagnosis on this foundation.

A8 · Monolithic, microkernel, hybrid — and where Linux actually sits

Official docs: Kernel admin guide — core subsystems · The kernel's command-line parameters

The previous section drew one box labelled "kernel". The obvious next question is how much you put inside that box — and that is the only thing kernel architectures disagree about.

Design	What runs in ring 0	Trade-off you are buying	Examples
Monolithic	Scheduler, memory manager, filesystems, network stack, and all device drivers	Fast — a filesystem calling the disk driver is a function call. Risky — a bad driver can panic the machine.	Linux, BSD
Microkernel	Only the minimum: scheduling, memory, message passing. Drivers and filesystems become ordinary user-space processes.	Robust — a crashed driver restarts like any process. Slower — the same work becomes messages across the boundary.	QNX, seL4, MINIX 3
Hybrid	Microkernel structure, but performance-critical parts pulled back into ring 0.	A pragmatic middle. The label is contested and mostly marketing.	Windows NT, XNU (macOS)

The trap in "Linux is monolithic". It is true, and on its own it is an incomplete answer that sounds memorised.

Linux is monolithic and modular. Drivers can be built as separate .ko files and loaded into ring 0 while the machine is running, instead of being built into the kernel itself.

But modular means when the code is loaded, not how much power it has. A loaded module has exactly the same power as code that was built in. That is why a third-party module can crash your kernel, and why loading one usually voids vendor support.

Saying that second half is what makes the answer sound like experience rather than revision.

Real-world analogy — one big kitchen versus a food hall

Monolithic is one big restaurant kitchen. Grill, pastry, sauces and washing up all in the same room.

Passing a pan from the grill to the counter takes a second. Nothing is faster than that.

But a fire at the grill fills the whole room with smoke, and the entire kitchen stops. One station's disaster is everybody's disaster.

Microkernel is a food court. Separate stalls, own walls, own staff, own extraction.

A stall can catch fire, close, clean up and reopen while everyone else carries on selling. But if you need chillies from the noodle stall, you have to walk there and back. What took a second now takes a trip.

That is the whole argument, in one line: how fast you can pass things around, against how far a disaster spreads. Linux picked the single kitchen. That is why a bad network driver crashes your whole machine instead of just restarting itself.

And the modular part: a loadable kernel module is a chef you take on halfway through service.

They arrived later, so people think of them as separate. They are not. They are standing at your stove, with your knives, in the same room. If they start a fire, it is the same fire.

That is why loadable modules are about when code is loaded, never about how much power it has.

🧪 Exercise A8.1 — Watch code being added to ring 0 at runtime

bash

# List the driver/feature modules currently loaded into the kernel
lsmod | head -8

# How many are loaded in total
lsmod | wc -l

# Inspect one. 'filename' shows it really is a file on disk that got loaded in.
modinfo xfs 2>/dev/null | head -6 || modinfo ext4 | head -6

✅ Expected result — click to reveal

plain text

$ lsmod | head -8
Module                  Size  Used by
nf_tables             270336  0
binfmt_misc            24576  1
nls_iso8859_1          12288  1
virtio_net             57344  0
virtio_blk             20480  3
crct10dif_pclmul       12288  1
xfs                  2166784  1

$ lsmod | wc -l
72

$ modinfo xfs | head -6
filename:       /lib/modules/6.8.0-45-generic/kernel/fs/xfs/xfs.ko.zst
license:        GPL
description:    SGI XFS with ACLs, security attributes, scrub, no debug enabled
author:         Silicon Graphics, Inc.
srcversion:     3D9C1F0A7B4E2C6D8A5F0B1E4C7D2A93
depends:        exportfs,libcrc32c

What to read out of it.

filename is the point of this exercise. xfs.ko.zst is a compressed file sitting on your disk. It was read at boot and loaded into ring 0.

That is two megabytes of filesystem code, running with full hardware power, loaded from a path that anyone with root could have replaced.

The Used by column is a count of who depends on the module. virtio_blk shows 3 because three disks are using it. You cannot unload a module while that number is above zero — the kernel will not pull the floor out from under running code.

virtio_net and virtio_blk tell you something extra for free. Those drivers only exist to talk to a hypervisor, so this machine is a virtual machine. Exercise B1.1 confirms it another way.

Now imagine this at 500 hosts. Fleet-wide, lsmod output is a security surface: any module loaded is arbitrary ring-0 code. This is why hardened environments set kernel.modules_disabled=1 after boot, why Secure Boot requires signed modules, and why "just install the vendor's kernel module" is a change-control conversation and not a dnf install.

🎯 Interview questions — What an OS is

Q. What is the difference between kernel space and user space?

User space is where applications run, at CPU privilege ring 3, with no ability to execute privileged instructions or reach memory the kernel has not mapped for them. Kernel space is ring 0, with full access to memory, privileged instructions and devices. The only legitimate way across is a system call.

Go deeper than the published answer, in this order:

The boundary is enforced by the CPU in hardware, not by kernel diligence. A privileged instruction from ring 3 faults before it takes effect.
Kernel space is not a separate place in RAM — it is mapped into every process's address space and simply unreachable from ring 3. That is why entering it is far cheaper than switching processes.
The blast radius differs: a user-space crash kills one process, a kernel-space fault is a panic or an oops that can take the machine with it.

The detail that separates candidates: mention that root is a user-space identity, not a privilege ring. Most candidates conflate the two. Root code still executes in ring 3 and the kernel still refuses it things — /dev/mem under CONFIG_STRICT_DEVMEM is the cleanest example.

Q. What is the Linux kernel and what does it actually do?

The kernel is the privileged core that manages CPU, memory, devices and filesystems, and mediates every request programs make of the hardware. Its main jobs are process and thread scheduling, memory management, the filesystem and storage stack, the network stack, and device drivers.

Deeper framing that lands better: describe it as a resource arbiter and an abstraction layer, then give one concrete example of each. Arbiter: two processes both want the CPU, the scheduler decides which one runs and for how long. Abstraction: open() behaves identically whether the target is on XFS, NFS, tmpfs or a USB stick, because the virtual filesystem layer sits underneath.

The detail that separates candidates: note that the kernel is not a process. It has no PID, it does not get scheduled as a unit, and it does not run continuously. It executes on borrowed time — inside your process's context when you make a system call, or in interrupt context when hardware demands attention. Candidates who picture the kernel as a background daemon reason incorrectly about almost everything downstream.

Q. Monolithic vs microkernel — which is Linux, and what is the trade-off?

Linux is monolithic: scheduler, memory manager, filesystems, network stack and device drivers all execute in ring 0 in a single address space. A microkernel keeps only scheduling, memory and IPC privileged, and pushes drivers and filesystems out into user-space processes.

The trade-off is performance against fault isolation. In a monolithic kernel a filesystem asking the block layer for a sector is a function call. In a microkernel it is a message across the privilege boundary, with the associated cost. In exchange, a microkernel can restart a crashed disk driver as if it were any other service, whereas in Linux that same crash is a kernel panic.

Add the modular nuance without being asked: Linux is monolithic and modular. .ko modules load driver code into ring 0 at runtime. That is a packaging property, not a privilege property — a loaded module is exactly as dangerous as compiled-in code.

The detail that separates candidates: connect it to something you have operated. "This is why a bad NIC driver panics a host rather than restarting cleanly, and why we treat out-of-tree modules as a change-control item rather than a package install." Architecture questions are checking whether you can reason from design to operational consequence — not whether you memorised two definitions.

Q. Why can't an application talk to hardware directly? Wouldn't that be faster?

It would be faster for that one application and catastrophic for everything else, and the reasons are worth giving in order:

Arbitration. Two programs writing to the same disk sectors with no coordination corrupt the filesystem. Someone has to serialise access, and that someone must be more privileged than both.
Isolation. Direct device access implies direct memory access. A device programmed to DMA into arbitrary physical memory can read any process's secrets, so hardware access is equivalent to total access.
Portability. The abstraction is the product. write() works the same on NVMe, a SATA disk, tmpfs, a pipe and a socket, which is why your program does not need rewriting per device.

The nuance that shows real depth: the kernel does hand out controlled direct access when the overhead genuinely matters — memory-mapped I/O, io_uring, DPDK and SPDK for userspace networking and storage, and the vDSO for cheap time reads. The rule is not "never touch hardware", it is "never touch hardware unmediated". The kernel sets up the mapping once, under its own supervision, and then steps out of the data path.

The detail that separates candidates: frame the exceptions as "never unmediated" rather than "never". Saying "the kernel does hand out controlled direct access when the overhead justifies it — it sets the mapping up once, under its own supervision, then steps out of the data path" shows you understand the boundary as an engineering trade-off with a measurable cost, not as a rule you memorised.

Q. What is an interrupt, and what role does it play in an operating system?

An interrupt is a signal that forces the CPU to stop what it is executing and enter the kernel at a predefined handler. Hardware interrupts come from devices — a disk finishing a read, a packet arriving, a timer expiring. Software interrupts and exceptions come from the CPU itself — a page fault, a divide by zero, a system call on older designs.

The framing that shows understanding: an interrupt is the hardware-initiated, involuntary way into ring 0. A system call is the software-initiated, voluntary way. Those are the only two ways the kernel ever begins running — the kernel is not a process and does not run continuously, so if neither of those has happened, it is not executing at all.

Two roles matter most operationally:

I/O completion. Without interrupts the CPU would have to poll devices, wasting entire cores waiting. Interrupts are what let a blocked process consume zero CPU.
The timer interrupt. This is the one that makes preemptive multitasking possible, and the next question is about it.

The detail that separates candidates: describe the top half / bottom half split. An interrupt handler runs with interrupts disabled on that CPU, so it must be extremely short or you lose the next interrupt. It acknowledges the device and defers real work to a softirq that runs with interrupts re-enabled. Then give the observable: ksoftirqd pinned at high CPU on a busy host is the bottom half failing to keep up, nearly always network receive — and cat /proc/interrupts twice, ten seconds apart, names the responsible device in one step.

Q. What is DMA, and why does it exist? How does it relate to polling and interrupts?

The three are answers to two different questions, and candidates who conflate them give muddled answers.

Question one: how does the CPU learn the device is ready? Either it asks repeatedly (polling, which burns a core doing nothing) or it is told (interrupts, which let the CPU run something else meanwhile).

Question two: who moves the bytes? If the CPU copies them register by register, a fast NIC or SSD saturates the processor. DMA — Direct Memory Access — lets the device write directly into RAM by itself. The driver supplies a physical target address, the device transfers while the CPU does other work entirely, and only then raises an interrupt to report completion. The CPU participates at the start and the end, never in the middle.

So the modern path is DMA plus interrupts: the device moves the data and then says it is done.

The details that separate candidates:

DMA bypasses the MMU, because it targets physical memory. A buggy or malicious device could write anywhere in RAM. That is precisely why the IOMMU exists, why Thunderbolt DMA attacks are real rather than theoretical, and why hypervisors require IOMMU support before permitting device passthrough. This point alone usually ends the question favourably.
Polling is not obsolete. At very high packet rates the interrupt itself becomes the bottleneck, so Linux's NAPI switches from interrupts to polling under load, and DPDK and SPDK poll exclusively. "Interrupts are better" is the shallow answer; "interrupts are better until the interrupt rate itself is the cost" is the operational one.

Q. What is the difference between preemptive and non-preemptive (cooperative) multitasking?

Under cooperative multitasking a process keeps the CPU until it voluntarily gives it up. Under preemptive multitasking the kernel can take the CPU away at any moment, whether the process cooperates or not.

The mechanism is the answer, and most candidates never reach it. Preemption is not something the kernel can simply decide to do, because while a process runs in ring 3 the kernel is not executing at all. It cannot interrupt anything, because it is not running. Preemption therefore requires hardware: the kernel programs a timer to fire an interrupt at intervals, and that interrupt forces the CPU out of ring 3 and into the kernel regardless of what the process was doing. Only then can the scheduler make a decision.

The consequence is the thing everyone has experienced: under cooperative multitasking one infinite loop freezes the whole machine — which is exactly why Windows 3.x and classic Mac OS hung so memorably. Under preemptive multitasking that same loop wastes CPU but you can still log in and kill it.

The details that separate candidates:

Separate mechanism from policy. The timer interrupt is the mechanism; deciding who runs next is the policy. Interviewers often ask this question hoping to get to the scheduler, and drawing the line yourself shows you know they are different layers.
Quote the observable. /proc/PID/status carries voluntary_ctxt_switches and nonvoluntary_ctxt_switches. High voluntary means the process blocks on I/O or locks — more CPU will not help it. High involuntary means it wants CPU and keeps being preempted — that is contention or a capacity shortfall. Those two point in opposite directions, and a dashboard showing only "CPU %" cannot distinguish them.
Modern kernels are tickless. The tick is suppressed on idle CPUs and on cores running exactly one runnable task, so the timer no longer fires blindly at a fixed rate. That saves real power at datacentre scale, and it is why timer interrupt counts do not match CONFIG_HZ on an idle machine.

Q. How does an operating system manage devices? What is a device driver?

A device driver is kernel code that knows one specific piece of hardware's language and presents it to the rest of the kernel through a generic interface. The filesystem layer says "read sector 4096"; the driver turns that into the exact register writes that controller expects. That seam is why the same networking and filesystem code runs on a laptop, a router and a mainframe — swap the hardware and only the driver changes.

A driver has exactly three mechanisms: memory-mapped I/O (device registers wired to appear at physical memory addresses, accessed with ordinary load and store instructions), port I/O (a separate address space via privileged instructions — x86-specific and largely legacy), and interrupts plus DMA for device-initiated work.

The details that separate candidates — give at least one observable:

MMIO is genuinely surprising and worth stating plainly. Writing a value to a physical address can start a network transmission with no RAM involved at all. That is why writing to the wrong physical address is issuing hardware commands rather than corrupting data, and why /dev/mem is locked down even against root on any modern kernel.
Name the diagnostic. lspci -k shows which driver is bound to which device. When Kernel driver in use is absent, the hardware was detected and no driver claimed it — a completely different failure from a broken device, and one dmesg will usually explain in a line. Fleet-wide, this is how you find the handful of hosts that received a different hardware revision and are silently running a different driver, which is the classic shape of a bug that hits 2% of machines and resists reproduction for a week.
Connect it to the architecture question. In a monolithic kernel every driver runs in ring 0, so a bad driver panics the host rather than restarting cleanly. That is the operational price of the design, and it is why out-of-tree modules are a change-control item rather than a package install.

🛠️ Part B · Building your lab

B1 · Why a disposable virtual machine, and not your laptop

Official docs: systemd-detect-virt(1) · Multipass documentation · Kernel admin guide

Most tutorials skip this step and it costs people days. Here is the honest reasoning.

You are going to break things on purpose. Several exercises in this track fill memory until the kernel kills something, saturate a disk, and stop processes in ways that make a terminal unresponsive. You want a machine you can delete.
Almost everything interesting needs root. Reading another user's /proc entries, tracing processes you do not own, and writing to /proc/sys all require it. Handing root to a learning exercise on your daily machine is a bad habit to build.
Your laptop is too noisy to learn on. A desktop OS runs hundreds of background processes. When you are trying to see the effect of one thing, a browser waking up every few seconds hides it. A fresh server VM idles at around 100 processes and near-zero CPU, which makes cause and effect visible.
macOS and Windows are not Linux. They are genuinely different kernels. /proc does not exist on macOS at all, so roughly half of this track has nowhere to run.
Snapshots make mistakes free. Break the boot process in Module 05, roll back, try again.

A VM you already pay for counts. A 1 GB cloud instance is a perfectly good lab and costs a few dollars a month. The only requirements for this track are: a mainstream Linux distribution, a kernel from the last few years, root access, and the willingness to destroy it.

Real-world analogy — the driving school car in an empty car park

Nobody learns to drive in their own car, in rush hour, on a hill. They learn in a driving school car, in an empty car park, where stalling costs nothing.

Your lab VM is that empty car park. You are going to fill memory until the kernel starts killing things, hammer a disk, and freeze terminals on purpose. Doing that on your laptop is like learning to reverse in traffic. You can, but the mistakes cost more than they teach.

The empty part matters as much as the car. Your laptop is running hundreds of background programs. So when you change one thing and measure it, a browser waking up hides your result. A fresh server VM sits at almost nothing, and that is what lets you see cause and effect at all.

Where the analogy stops working. A driving school car is safer than a normal car. Your VM is not safer. It is cheaper. It will break just as badly as a real machine — the difference is you delete it and make a new one in ninety seconds.

🧪 Exercise B1.1 — Find out what you are running on right now

bash

# Are we on bare metal, in a VM, or in a container?
systemd-detect-virt

# Ask the two questions separately - a container can run inside a VM
systemd-detect-virt --vm
systemd-detect-virt --container

# The distribution and kernel, in one place
hostnamectl | head -12

✅ Expected result — click to reveal

plain text

$ systemd-detect-virt
kvm

$ systemd-detect-virt --vm
kvm

$ systemd-detect-virt --container
none

$ hostnamectl | head -12
 Static hostname: oslab
       Icon name: computer-vm
         Chassis: vm 🖴
      Machine ID: 4f1c2b9a6e7d43f0b8c5a2d1e9f7034b
         Virtualization: kvm
Operating System: Ubuntu 24.04.3 LTS
          Kernel: Linux 6.8.0-45-generic
    Architecture: x86-64

What to read out of it.

kvm means a hardware-virtualised guest — a real, separate kernel. That matches what you already deduced in Exercise A8.1 when you saw virtio_net and virtio_blk in lsmod: paravirtualised drivers only exist to talk to a hypervisor. Two independent signals agreeing is how you build confidence in a diagnosis.

--container returning none and exiting non-zero is the useful half. systemd-detect-virt answers two separate questions, and a container running inside a VM answers yes to both. Treating them as one question is a common mistake.

Other values you will meet: none (bare metal), docker, lxc, podman, wsl, amazon (Nitro), microsoft (Hyper-V and Azure), vmware, oracle (VirtualBox).

Now imagine this at 500 hosts. systemd-detect-virt is one of the cheapest and most reliable facts to collect fleet-wide. It tells you instantly which hosts are bare metal (different failure modes, different patching cadence) and which are containers pretending to be hosts in your inventory — the second category is a classic source of monitoring that has silently been measuring the wrong machine for a year.

B2 · Installing the lab

Official docs: Multipass — install · Kernel admin guide

Pick one of these. Multipass is recommended because it is a single command on all three host operating systems and it does not ask you to click through an installer wizard.

bash

# ---- Option 1: Multipass (recommended) --------------------------------
# On macOS:
brew install --cask multipass
# On Windows: download the installer from the Multipass docs linked above
# On Linux:
sudo snap install multipass

# ---- Option 2: You already have a cloud VM ---------------------------
# Nothing to install. Skip to B3.

# ---- Option 3: VirtualBox / VMware / UTM -----------------------------
# Install the hypervisor, then an Ubuntu Server 24.04 or Rocky 9 ISO.
# Choose the SERVER image, not the desktop image - you want the quiet machine.

Do not use Docker as your lab for this track. It will appear to work for the first few exercises and then quietly teach you wrong things — a container shares the host's kernel, so /proc reports facts about a machine you are not on. Exercise B4.2 demonstrates this deliberately so you can recognise it. Containers are excellent; they are just the subject of Module 12, not the classroom for Module 01.

Real-world analogy — the flight simulator wired to the wrong aircraft

A flight simulator is a great way to learn to fly. Now imagine one where the dials were wired to a different aeroplane. Your fuel gauge shows that other plane's fuel. Your altimeter shows its height.

Every control still works. Nothing feels wrong. And everything you learn about reading the dials is wrong.

That is what using Docker as your classroom would do here. The commands all run. The output looks confident. But nproc, free and uname are showing you the host's numbers, not the container's. You would spend two weeks building careful habits on figures that describe a machine you are not on.

So: use a VM. A real cockpit, wired to itself.

Where the analogy stops working — and this is a point in favour of containers. A miswired simulator is simply broken. A container is not broken. It is doing exactly what it was designed to do.

The mismatch is a real property of how containers work, not a bug. That is why it gets a whole exercise (B4.2) instead of a footnote, and why it is the subject of Module 12.

🧪 Exercise B2.1 — Create the lab VM

bash

# Create a 2-CPU, 2 GB, 10 GB VM called "oslab"
multipass launch 24.04 --name oslab --cpus 2 --memory 2G --disk 10G

# Confirm it is running
multipass list

✅ Expected result — click to reveal

plain text

$ multipass launch 24.04 --name oslab --cpus 2 --memory 2G --disk 10G
Launched: oslab

$ multipass list
Name       State       IPv4            Image
oslab      Running     10.191.44.118   Ubuntu 24.04 LTS

What to read out of it. The first launch downloads a cloud image and takes a few minutes; later ones are near-instant because the image is cached.

If it fails, the two overwhelmingly common causes are:

launch failed: Downloaded image hash does not match — a corrupted cache. Fix with multipass purge and retry.
On Windows or macOS, an error about the hypervisor not being available means virtualisation is disabled in firmware, or another hypervisor (Hyper-V, VirtualBox) holds the virtualisation extensions. Only one can own them at a time.

Note the memory: 2 GB is deliberate. In Module 09 you will trigger the out-of-memory killer, and doing that on a 64 GB machine takes an uncomfortably long time.

B3 · Connecting and verifying it works

Official docs: uname(1) · os-release(5) · The kernel's command-line parameters

bash

multipass shell oslab      # if you used Multipass
ssh user@your-vm-ip        # if you used a cloud VM

Before doing anything else, learn to read the machine's identity card. You will type uname -a thousands of times in your career and most people never learn what the fields mean.

Real-world analogy — the engine versus the trim level

"What car is that?" has two answers, and people mix them up all the time.

There is the engine — how big it is, how many cylinders, which generation. And there is the trim — the badge on the back, the seats, the screen, the warranty.

The same engine goes into a basic model and a luxury one. The same trim can be sold with three different engines. They are separate facts.

uname -r is the engine. /etc/os-release is the trim.

The kernel decides what the machine can do — which filesystems exist, which container features work, which tracing tools run. The distribution decides what it is like to work with — the package manager, the defaults, the support contract.

So Ubuntu 22.04 can be running a 6.11 kernel. Rocky 9 ships a 5.14 kernel that has been patched for years.

That last bit is the one that costs people money. A manufacturer can replace a faulty part and keep the same engine number. So the number on the badge does not tell you what is actually inside.

This is why you cannot decide whether a kernel is affected by a security bug just by comparing version numbers. Vendors fix things without changing the number. You check the vendor's security notices, not the badge.

Where the analogy stops working. You cannot swap a car engine over lunch. You can boot a different kernel on the same distribution with one reboot, so these two facts drift apart far more often than they would in a car park.

🧪 Exercise B3.1 — Decode your kernel's identity

bash

uname -a

# The same information, one field at a time, so you can see which is which
uname -s   # kernel name
uname -n   # network node hostname
uname -r   # kernel RELEASE  <- the one that matters most
uname -v   # kernel VERSION (build info, not what you think)
uname -m   # machine hardware architecture

# The distribution is a completely separate question from the kernel
cat /etc/os-release

✅ Expected result — click to reveal

plain text

$ uname -a
Linux oslab 6.8.0-45-generic #45-Ubuntu SMP PREEMPT_DYNAMIC Wed Jul 15 19:22:11 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux

$ uname -r
6.8.0-45-generic

$ uname -v
#45-Ubuntu SMP PREEMPT_DYNAMIC Wed Jul 15 19:22:11 UTC 2026

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.3 LTS (Noble Numbat)"
ID=ubuntu
ID_LIKE=debian

What to read out of it. Take 6.8.0-45-generic apart:

6 major, 8 minor — the upstream kernel line. This decides which features exist at all: cgroup v2 behaviour, io_uring maturity, eBPF capabilities, which scheduler you have.
0 — upstream patch level.
-45 — the distribution's build number. Ubuntu and Red Hat backport fixes into old kernel lines, so a "6.8.0" kernel from a vendor may contain patches that upstream only shipped in 6.11. This is why matching a CVE to a kernel by version string alone is unreliable.
-generic — the flavour. You may also see -aws, -azure, -cloud, -lowlatency, -rt.

Two traps here, both of which catch experienced people.

uname -v is not the kernel version. It is a build description. The version you want is uname -r. The two letters are named the opposite way round from what anyone would expect, and this catches experienced people out.

The kernel and the distribution are separate facts. Here /etc/os-release says Ubuntu 24.04 and uname -r says 6.8.0-45. Neither one implies the other.

When someone asks "what OS is this box?", they almost always need both answers. Giving only one is how upgrade plans go wrong.

One more free fact. The three repeated x86_64 fields are machine hardware name (-m), processor type (-p) and hardware platform (-i) — the operating system (-o) is the GNU/Linux right at the end, not one of the three. If you are on Apple Silicon or Graviton you will see aarch64 there instead, and that single field explains most "the binary won't run" tickets you will ever receive.

B4 · Installing the observation toolset

Official docs: strace(1) · Yama LSM — ptrace_scope · proc(5)

Four packages carry most of this track. Install them now so no later exercise stalls on a missing binary.

bash

# Debian / Ubuntu
sudo apt update
sudo apt install -y strace ltrace procps sysstat man-db manpages-dev

# RHEL / Rocky / AlmaLinux / Fedora
sudo dnf install -y strace ltrace procps-ng sysstat man-db man-pages

Package	What it gives you	Used from
strace	Shows every system call a process makes — the boundary from Part A, made visible	Part C onward
ltrace	Shows library calls instead, so you can see the wrapper layer separately	Section C2
procps / procps-ng	ps, top, vmstat, free, pmap — the /proc readers	Everywhere
sysstat	pidstat, iostat, mpstat, sar — per-process and historical statistics	Modules 07, 10, 14
man-db, manpages-dev	The section 2 and 3 manual pages: the offline copy of every doc link in this track	Section D3

Real-world analogy — the stethoscope, and who is allowed to use one

strace is a stethoscope. It lets you hear what is going on inside something you cannot open up.

The rest of the toolset is the rest of the doctor's bag. ps and top take the pulse. pidstat tracks it over time. vmstat watches the whole ward.

But you cannot point a stethoscope at any stranger you like. A hospital has several rules about who may examine whom, and failing any one of them stops you:

Are you staff?
Is this your patient?
Has this ward been restricted?
Does this unit have its own extra rules?

"But I am a doctor" answers the first question and none of the others.

That is exactly the situation in Exercise B4.1. Four separate gates: do you own the process, what does Yama allow, do you have the capability inside a container, and does SELinux or AppArmor permit it. Someone who answers "you need root" has named one gate out of four.

Where the analogy stops working — and this is the most important warning in the module. A stethoscope is passive. Listening costs the patient nothing.

strace stops the process at every single system call. It is more like interrupting a surgeon after every stitch to ask how it is going. On a busy production process that can make things ten times slower. Module 14 teaches tools that watch without stopping anything.

🧪 Exercise B4.1 — Verify the toolset, and meet your first permission wall

bash

strace -V | head -1
ltrace -V | head -1
pidstat -V

# Trace a trivial command you own
strace -c ls /tmp >/dev/null

# Now trace a process owned by root
strace -p 1

✅ Expected result — click to reveal

plain text

$ strace -V | head -1
strace -- version 6.8

$ strace -c ls /tmp >/dev/null
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  0.00    0.000000           0        23           mmap
  0.00    0.000000           0        12           close
  0.00    0.000000           0         9         6 openat
  0.00    0.000000           0         4           write
  ...
------ ----------- ----------- --------- --------- ----------------
100.00    0.000132                    98        11 total

$ strace -p 1
strace: attach: ptrace(PTRACE_SEIZE, 1): Operation not permitted

What to read out of it. The last line is the point of the exercise, and it is not a broken install.

Tracing a process uses something called ptrace. It lets the tracer read the target's memory and its CPU registers. That is complete control over the other process, so the kernel guards it carefully.

Two separate gates have to open before it works:

Ownership or capability. PID 1 is root's. You are not root, so you are refused. sudo strace -p 1 passes this gate.
The Yama LSM. Even as the right user, kernel.yama.ptrace_scope can forbid it. Check with sysctl kernel.yama.ptrace_scope. 0 is classic permissions, 1 (the Ubuntu and Debian default) allows tracing only your own descendants, 2 requires CAP_SYS_PTRACE, and 3 disables tracing entirely and cannot be undone without a reboot.

The errors column in the summary is worth a second look too: openat was called 9 times and failed 6 times, and ls worked perfectly anyway. Failed system calls are completely normal — that is the dynamic linker searching directories for a library until it finds one. Section C4 comes back to this, because "I see errors in strace, that must be the bug" is one of the most reliable ways to waste an afternoon.

Interview-grade detail. When asked why strace fails in production, most candidates say "you need root". The stronger answer names both gates and adds the container case: in Docker and Kubernetes, ptrace also needs the SYS_PTRACE capability, which is dropped by default. That is why kubectl exec plus strace fails even as root inside the container, and why debug sidecars exist.

Real-world analogy — the serviced office and the building directory

Your company rents a small office inside a big building. Your contract is clear: two desks and one parking space.

But the board in the lobby — the one everybody actually reads — still describes the whole building: 400 desks, 200 parking spaces, a gym.

Now your manager plans next week from that lobby board. He invites forty people and promises everyone parking.

On the day, thirty-eight are turned away at the barrier. Nobody can explain it, because every number he used was correct — about a building he does not have.

That is what happens when a program sizes itself from the host's numbers instead of its own limits. A Java process picking its memory from total RAM. A Go program picking its worker count from CPU count. Nginx with worker_processes auto.

The limits are set and enforced in one place. The numbers everyone reads come from somewhere else. Nothing connects the two.

Where the analogy stops working — and it is a point in the container's favour. The lobby board is just careless. /proc is not careless. It is honestly reporting the kernel it belongs to, and there is only one kernel here.

A container is not a small building at all. It is a few desks in the same building with a screen around them. That is why uname -r is the same inside and out, and why thinking of containers as "small VMs" eventually catches everyone out.

🧪 Exercise B4.2 — The container trap (this one is meant to mislead you)

Run this only if you have Docker available. It is here to inoculate you against a genuinely dangerous illusion. If you do not have Docker, read the expected result — the lesson survives without running it.

bash

# On the HOST, note the truth:
nproc
free -h | head -2
uname -r

# Now enter a container limited to half a CPU and 256 MB, and ask the same questions:
docker run --rm -it --cpus 0.5 --memory 256m ubuntu:24.04 bash

#   ...inside the container:
nproc
free -h | head -2
uname -r
ps -ef
exit

✅ Expected result — click to reveal

plain text

### ON THE HOST
$ nproc
2
$ free -h | head -2
               total        used        free      shared  buff/cache   available
Mem:           1.9Gi       412Mi       789Mi       1.0Mi       745Mi       1.4Gi
$ uname -r
6.8.0-45-generic

### INSIDE THE CONTAINER (--cpus 0.5 --memory 256m)
root@3f9c1a2b8d44:/# nproc
2
root@3f9c1a2b8d44:/# free -h | head -2
               total        used        free      shared  buff/cache   available
Mem:           1.9Gi       412Mi       789Mi       1.0Mi       745Mi       1.4Gi
root@3f9c1a2b8d44:/# uname -r
6.8.0-45-generic
root@3f9c1a2b8d44:/# ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 10:22 pts/0    00:00:00 bash
root          10       1  0 10:23 pts/0    00:00:00 ps -ef

What to read out of it — read this slowly, it is the most important paragraph in Part B.

You limited the container to half a CPU and 256 MB. Inside it, nproc says 2 CPUs and free says 1.9 GB. Both are the host's numbers. The container was told a limit and then handed a view of the machine that does not mention it.

uname -r is identical inside and out, and that is the explanation: there is only one kernel here. A container is not a small machine. It is a set of ordinary processes on the host kernel, which the kernel has been instructed to show a restricted view of the system to. The restriction covers which processes are visible, which filesystem is visible, and which network is visible — ps -ef proves it, showing 2 processes instead of the host's 243. But it does not cover the machine-capacity numbers that nproc and free read out of /proc.

The production consequence, and it is expensive. Any runtime that sizes itself from nproc or from total memory will size itself for the host. A JVM picking a heap from "total RAM", a Go program setting GOMAXPROCS from CPU count, an Nginx worker_processes auto, a Python multiprocessing pool — all of them will provision for 2 CPUs and 1.9 GB inside a box allowed 0.5 CPU and 256 MB.

The result is a pod that is throttled into uselessness or killed for exceeding memory, with application logs that show nothing wrong. Modern JVMs and Go runtimes have learned to read the limits directly instead; a great deal of older software has not.

Now imagine this at 500 hosts. This is why fleet capacity dashboards built from in-container free output overstate available memory by an order of magnitude, and why the fix is to read the limits from where they are actually enforced. Module 12 shows you exactly where that is.

This is also, concretely, why Module 01 asked you to build a VM. Half the observations in this track would have been lies.

🎯 Interview questions — Environment and isolation

Q. What is the difference between a virtual machine and a container?

A VM runs its own kernel on virtualised hardware provided by a hypervisor. A container is a set of ordinary processes running on the host's kernel, which the kernel has been configured to give a restricted view of the system — restricted process list, restricted filesystem, restricted network — plus enforced limits on the resources they may consume.

The consequences worth stating, because they are what the question is really testing:

Boot time and footprint. A VM boots a kernel: seconds, and hundreds of megabytes of overhead. A container starts a process: milliseconds, and near-zero overhead.
Isolation strength. A VM's boundary is the hypervisor, a much smaller and more scrutinised interface. A container's boundary is the kernel's entire system-call surface, which is enormous. A kernel vulnerability is a container escape; it is usually not a VM escape.
Kernel choice. Containers on one host must all run the same kernel and the same kernel version. You cannot run a Windows container on a Linux host, and you cannot test a kernel upgrade in a container.

The detail that separates candidates: give the observable proof rather than the definition. "Run uname -r inside a container and on its host — identical, because there is one kernel. Then run nproc in a container limited to half a CPU: it still reports the host's CPU count, because the limits are enforced in one place and reported in another." That answer says you have debugged this, not read about it.

Q. How do you check what kernel and what distribution a server is running, and why does the distinction matter?

uname -r for the kernel release; cat /etc/os-release for the distribution. hostnamectl prints both plus the virtualisation type in one shot.

They matter separately because they are genuinely independent. The kernel decides which features exist — cgroup v2 semantics, io_uring, eBPF capabilities, filesystem support. The distribution decides your userland: libc version, package manager, init configuration, default security module. Ubuntu 22.04 can run a 6.11 kernel; Rocky 9 ships a 5.14 kernel with years of backports.

The detail that separates candidates: explain why version strings alone cannot settle a CVE question. Enterprise distributions backport security fixes without changing the upstream version number, so a "5.14.0" RHEL kernel may contain patches upstream shipped in 6.6. The correct method is the vendor's errata — rpm -q --changelog kernel or the distribution's security tracker — not comparing numbers to an upstream changelog. Candidates who confidently compare version numbers here are showing you exactly how they would mis-triage a vulnerability scan.

Bonus: uname -v is the build string, not the version. Mixing up -r and -v in a script is a real and common bug.

Q. strace fails on a production process with "Operation not permitted", even as root. Why?

Because ptrace — the mechanism strace uses — is gated in more than one place, and root only opens the first gate.

Work through them in this order out loud:

Ownership. You may trace processes running as you. sudo handles this.
Yama LSM. kernel.yama.ptrace_scope is 1 by default on Debian and Ubuntu, permitting tracing only of your own descendants. 2 requires CAP_SYS_PTRACE; 3 disables ptrace entirely and cannot be reversed without a reboot.
Capabilities in containers. SYS_PTRACE is dropped from Docker's default capability set. Root inside the container is still refused. This is the case that confuses people most, because everything looks like root.
Mandatory access control. SELinux or AppArmor can deny the operation independently of all of the above. Check dmesg or the audit log — the denial will not appear in strace's own error message.

The detail that separates candidates: volunteer the operational caution. strace works by stopping the target at every system call, which can slow a busy process by an order of magnitude or more, and if strace is killed at the wrong moment the target can be left stopped. On a production host you reach for something else first — perf, or eBPF-based tooling, which sample without halting the process. Knowing when not to use the tool is worth more than knowing its flags, and Module 14 is where that toolkit gets built properly.

🚪 Part C · The system call boundary

C1 · What a system call actually is

Official docs: syscalls(2) — the complete list · intro(2) — introduction to system calls · POSIX.1-2024 System Interfaces

Part A established that user space cannot reach hardware. So how does ls read a directory?

It asks. A system call is a request from a ring-3 program for the kernel to do something for it. It is the only way across.

There are only about 450 of them on a current x86-64 kernel. That is a surprisingly short list for everything a computer can do.

The sequence is short, and worth knowing exactly, because half the interview questions in this part are really checking whether you know it.

Diagram source

sequenceDiagram
    participant App as App in ring 3
    participant CPU as CPU
    participant Krn as Kernel in ring 0
    App->>App: put syscall number in a register
    App->>CPU: execute the syscall instruction
    CPU->>Krn: switch to ring 0, jump to the kernel entry point
    Krn->>Krn: look the number up in the syscall table
    Krn->>Krn: validate arguments, do the work
    Krn->>CPU: execute sysret
    CPU->>App: back in ring 3, result waiting in a register

Three things about that sequence are worth saying out loud:

The program does not choose where it lands. It cannot jump to an arbitrary kernel address. It executes one instruction and the CPU decides the destination, from a table the kernel set up at boot. That is what makes the doorway a doorway rather than a hole.
The number, not the name, is the interface. write is 1 on x86-64 and 64 on ARM64. The names are a convenience for humans; the stable contract is the number, which is why Linux essentially never reuses one.
Nothing has been unloaded. The same CPU, executing in the same process's context, at a different privilege level. Remember the callout in A2: this is why the transition is far cheaper than switching between two programs.

Real-world analogy — the pharmacy counter

You cannot walk behind the counter and take what you want off the shelves. You hand a slip through the window. Someone qualified goes and gets it and hands you back a result.

One window, one procedure, and no exceptions for confident customers.

Every part of the mechanism is in there:

You do not choose where the pharmacist goes. You hand in the slip; they decide which shelf. Your program runs one instruction and the CPU decides where it lands, from a list the kernel wrote. That is what makes it a proper door rather than a hole in the wall.
The code on the slip is what counts, not the name. Prescriptions use product codes because names are ambiguous and change. In the same way, write is number 1 on one type of CPU and number 64 on another. The number is the real agreement, which is why Linux almost never reuses one.
The list of procedures is short. A pharmacy handles a huge range of needs with a handful of standard steps. Around 450 system calls cover everything a computer does.

Where the analogy stops working. A pharmacy is a separate building you walk to. The kernel is already inside your own process. The window is in your own wall, which is why the trip takes nanoseconds rather than being a journey at all.

🧪 Exercise C1.1 — Watch a single system call happen

bash

# Show only the write() calls that 'echo' makes
strace -e trace=write echo hello

✅ Expected result — click to reveal

plain text

$ strace -e trace=write echo hello
write(1, "hello\n", 6)                  = 6
hello
+++ exited with 0 +++

What to read out of it. That one line is the entire boundary, and every field is meaningful:

write — the system call name, resolved by strace from the number.
1 — the first argument: the file descriptor. 1 is standard output. Module 03 takes descriptors apart properly.
"hello\n" — the second argument: the buffer. strace has reached into the process's memory and printed it for you, which is exactly the power that made ptrace a privileged operation in Exercise B4.1.
6 — the third argument: how many bytes to write. Five letters plus the newline.
= 6 — the return value: six bytes were written. Section C4 is entirely about this number.

Notice the ordering in the output: the write(...) line appears before hello. strace reports the call as it is made; the terminal shows the text once the kernel has delivered it. Getting used to that ordering saves confusion later when you are reading thousands of lines.

Notice also what is absent. There is no system call for "print a string", no call to format text, no call to look up what hello means. All of that happened in user space. Only the final act of moving bytes out of the process needed the kernel.

C2 · The library wrapper and the raw system call

Official docs: intro(2) — return values and errno · syscalls(2) · errno(3)

Programs almost never execute the syscall instruction themselves. They call a function in the C library — glibc on most distributions, musl on Alpine — and that function does it for them. This layering causes a lot of confusion, so let us be exact about who does what.

Layer	Where it runs	What it is responsible for
Your code	Ring 3	Calls a normal-looking function
libc wrapper	Ring 3	Arranges arguments, executes the syscall instruction, translates the kernel's answer into a return value plus errno
Kernel	Ring 0	Does the work, returns a single number

The counter-intuitive part, and it is the source of a lot of muddled thinking. The kernel does not know what errno is. It has never heard of it.

The kernel returns one number. On failure that number is negative — the negated error code. The libc wrapper sees the negative value, stores its absolute value into the errno variable, and returns -1 to your program. errno is a user-space variable that libc maintains, documented in section 3 of the manual, not section 2.

This is why the same underlying kernel behaviour is described one way in man 2 open and another way in your program, and why a program that makes a raw system call without libc gets a negative number and no errno at all.

Some functions look like system calls and are not. printf, malloc and fopen are pure library functions that may make system calls underneath — printf buffers your text in user space and only calls write when the buffer fills or is flushed. That buffering is the reason output sometimes appears out of order when a program crashes: the text was still sitting in user space and never crossed the boundary.

Real-world analogy — the travel agent and the airline

You tell a travel agent "get me to Tokyo on Friday." The agent fills in the airline's forms, in the airline's format, and deals with the airline's systems. You never see any of that.

Now watch what happens when it fails.

The airline's system sends back something short and numeric — a rejection code. The agent looks it up and writes you a note saying "sorry, that fare is sold out."

Two separate things happened there: the airline's code, and the agent's note.

That is the most misunderstood thing about errno. The kernel has never heard of errno. It returns one number, and a negative one means failure. The library function reads that number, writes the reason into errno, and hands your program -1.

errno is the agent's note. It lives in user space, not in the kernel. That is why write is documented in section 2 and errno in section 3 — two different organisations, two different sets of paperwork.

The same layering explains printf. "Book me to Tokyo" is not one transaction with the airline; the agent does a lot of work before contacting anyone. printf builds your text entirely in user space and only crosses into the kernel once, with write, when it has enough to send.

That is exactly why a crashing program's last few lines of output sometimes never appear. They were still sitting on the agent's desk, never sent.

🧪 Exercise C2.1 — See the two layers separately

bash

# Layer 1: the system calls the process makes
strace -e trace=write echo hello

# Layer 2: the LIBRARY calls it makes. Note we trace a dynamically linked binary.
ltrace -e '*' /bin/echo hello 2>&1 | head -10

# Which functions does a binary actually import from libc?
ldd /bin/echo

✅ Expected result — click to reveal

plain text

$ strace -e trace=write echo hello
write(1, "hello\n", 6)                  = 6
hello

$ ltrace -e '*' /bin/echo hello 2>&1 | head -10
echo->getenv("POSIXLY_CORRECT")                   = nil
echo->__printf_chk(1, 0x5c4f2a1b3d40, 0x7ffd1c2e, 6) = 6
echo->exit(0 <no return ...>

$ ldd /bin/echo
        linux-vdso.so.1 (0x00007ffd1c3f9000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2a3c000000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f2a3c2f1000)

What to read out of it. The two traces show completely different things about the same command, and that is the lesson.

ltrace shows __printf_chk — a library function. strace shows write — a system call. The program called printf; libc formatted the string in user space, and only then crossed the boundary once with write. One library call, one system call, and they do not have the same name.

ldd explains why any of this is visible at all: /bin/echo is dynamically linked against libc.so.6, so those calls go through a shared library that ltrace can intercept. A statically linked binary — most Go binaries, anything built with -static — shows almost nothing under ltrace, because there is no library boundary left to watch. strace still works perfectly on those, because the syscall boundary cannot be optimised away.

And there in ldd's first line is linux-vdso.so.1, a library with no path on disk. Hold that thought until Section C5.

If ltrace prints nothing or errors, you are likely on a distribution where it is unmaintained, or tracing a static binary. That is not a problem — ltrace is a teaching aid here and appears nowhere else in this track. strace is the tool that matters.

C3 · Reading strace output fluently

Official docs: strace(1) — options and filters · syscalls(2) · Yama LSM — ptrace_scope

Every strace line has the same shape, and once you can read it at a glance you can debug problems that logs will never show you.

plain text

openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
└─┬──┘ └───────────┬──────────────────────────────┘  └┬┘
  │                │                                  └── return value
  │                └───── arguments, decoded by strace
  └──────────────────── the system call name

The flags you will actually reach for:

Flag	What it does	When you want it
-e trace=NAME	Only show the named calls. Accepts groups: %file, %network, %process, %memory	Almost always — raw output is unreadable
-c	Print a summary table instead of the calls	"Where is this process spending its time?"
-f	Follow children created by fork and clone	Anything that spawns workers — otherwise you trace an idle parent
-p PID	Attach to a running process	Something is hung right now
-o FILE	Write the trace to a file	Always, for anything non-trivial — trace output competes with program output
-T / -tt	Time spent in each call / wall-clock timestamps	"Which call is slow?"
-s N	Print N bytes of string arguments instead of the default 32	Reading truncated paths or payloads

Real-world analogy — the itemised phone bill

Someone hands you a 3,000-line phone bill and asks why it is so big. You do not read it line by line. You sort it, and you look for one of exactly two patterns:

One number called 4,000 times, a few seconds each. Nobody is having a long conversation. Something is dialling in a loop. That is ls -la making 2,843 statx calls. It is doing too many small things, and the fix is to do fewer of them.
Three calls, forty minutes each. Almost no calls, and nearly all the cost. Somebody is on hold. That is getent hosts spending 95% of its time in two calls. It is waiting, and the fix is somewhere else entirely — the network, the disk, the other service.

Those two bills lead to completely different conversations. And here is the catch: both of them show up as "95% of the total" in a summary.

That is why the number of calls and the time per call matter more than the percentage. Anyone who only reads the percentage column gets both cases wrong.

Where the analogy stops working. A phone bill is produced for you afterwards, at no cost. strace produces its bill by stopping the process at every call, so the act of measuring changes what you are measuring.

So always ask for the smallest bill you can: use -e trace= to filter, -o to write it to a file, and keep it short on a production machine.

🧪 Exercise C3.1 — Find out where a command's time actually goes

bash

# Summary mode. strace -c sorts by time by default; add -S calls to sort by count.
strace -c -o /tmp/trace-ls.txt ls -la /usr/bin
head -15 /tmp/trace-ls.txt

# Now the same for something that touches the network
strace -c -f -o /tmp/trace-dns.txt getent hosts kernel.org
head -15 /tmp/trace-dns.txt

✅ Expected result — click to reveal

plain text

$ head -15 /tmp/trace-ls.txt
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 71.42    0.003891           1      2843           statx
 12.03    0.000655           4       144           getdents64
  6.17    0.000336           2       142           write
  4.28    0.000233          14        16           mmap
  2.11    0.000115          12         9         6 openat
  1.44    0.000078           7        10           close
  ...
------ ----------- ----------- --------- --------- ----------------
100.00    0.005448                   3182        11 total

$ head -15 /tmp/trace-dns.txt
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 94.88    0.041022       20511         2           recvfrom
  2.31    0.000998         998         1           connect
  1.02    0.000441          14        31           mmap
  ...

What to read out of it — the two traces tell opposite stories.

For ls -la, the summary is dominated by 2,843 statx calls. ls -la needs metadata for every entry, and it asks the kernel separately for each one. The directory had roughly 1,400 files, so that is two statx per file. That single number explains something you have certainly experienced: ls is instant on a normal directory and unbearable on a network filesystem with 50,000 files. It is not "slow" — it is making tens of thousands of round trips.

For getent hosts, the picture is completely different: 2 calls consuming 95% of the time, at 20 milliseconds each. Low count, high usecs/call — that is the signature of waiting, not working. Those recvfrom calls are the DNS response arriving from the network.

This is the reading skill worth practising. Two different failure shapes, and you diagnose them from opposite columns:

Many calls, low usecs/call → the program is chatty. The fix is batching, caching, or a better algorithm.

Few calls, high usecs/call → the program is blocked on something external. The fix is elsewhere: the network, the disk, the lock, the other service.

Candidates who only ever look at the % time column miss this entirely, because both shapes can show 95% in the same place.

The errors column shows 6 failed openat calls in the ls run, and ls still worked. That is the next section.

C4 · How system calls report failure — return values and errno

Official docs: errno(3) — the full error list · intro(2) — return conventions · POSIX.1-2024 Base Definitions

There is one convention, and it holds across almost every system call:

Success returns zero or a useful non-negative number — bytes transferred, a file descriptor, a process ID.
Failure returns -1, and libc sets errno to say why.

The error names are worth learning because they map directly onto the errors users report to you.

errno	Message you see	What it actually means, and the usual cause
EACCES	Permission denied	You failed a permission check. File mode, ownership, or a directory on the path that you cannot traverse.
EPERM	Operation not permitted	The operation itself is forbidden to you regardless of the file's mode — usually a missing capability. Not the same as EACCES.
ENOENT	No such file or directory	Nothing at that path. Also what you get for a dangling symlink, and for a missing interpreter on a script's shebang line.
ENOSPC	No space left on device	Out of blocks or out of inodes. df -h and df -i answer different questions.
EAGAIN	Resource temporarily unavailable	On a socket, no data yet — normal. On fork, you have hit a process or thread limit.
EMFILE	Too many open files	This process hit its file-descriptor limit. Per-process.
ENFILE	Too many open files in system	The whole machine hit its limit. Very different blast radius, one letter apart.
ECONNREFUSED	Connection refused	The host answered and nothing is listening on that port. Distinct from a timeout, which means no answer at all.

Real-world analogy — the rejected application form

You send off an application and it comes back rejected.

"Rejected" on its own is useless. The reason code is the whole content of the letter, because each reason sends you somewhere completely different:

"You are not eligible for this" — you failed a rule about who you are. Arguing with the clerk will not help. That is EACCES, a failed permission check.
"This is closed to everyone" — you were eligible and it was still refused. That is EPERM. Both letters say "denied" to the person reading them, and they send you to different buildings.
"No such office at that address" — that is ENOENT.
"The filing room is full" — that is ENOSPC. And note there are two ways to be full: no shelf space left, or no folder numbers left. df -h and df -i answer those two different questions.

There is a second lesson here, and it is the one people get wrong.

One rejection does not mean something went wrong. A clerk who checks four filing rooms before finding your file produces three "not here" replies and then succeeds. That is exactly what happens when a program looks for a library in several folders before finding it — a screen full of ENOENT in a completely healthy program.

What actually tells you something is the last rejection before someone gave up, or a rejection from an address you did not expect. That is why you read a trace backwards from the end.

🧪 Exercise C4.1 — Watch a permission failure at the boundary (meant to fail)

bash

# Read a file you are not allowed to read
strace -e trace=openat cat /etc/shadow

# Now a file that does not exist at all - predict the difference first
strace -e trace=openat cat /etc/nope

# And the numeric error codes behind the names
errno EACCES 2>/dev/null || grep -rn "define EACCES" /usr/include/asm-generic/errno-base.h

✅ Expected result — click to reveal

plain text

$ strace -e trace=openat cat /etc/shadow
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/shadow", O_RDONLY)  = -1 EACCES (Permission denied)
cat: /etc/shadow: Permission denied
+++ exited with 1 +++

$ strace -e trace=openat cat /etc/nope
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/nope", O_RDONLY)    = -1 ENOENT (No such file or directory)
cat: /etc/nope: No such file or directory
+++ exited with 1 +++

$ grep -rn "define EACCES" /usr/include/asm-generic/errno-base.h
17:#define	EACCES		13	/* Permission denied */

What to read out of it. Follow the chain of custody for the error, because this is the thing most people never see:

The kernel refused and returned -13.
libc turned that into a return value of -1 and set errno to 13.
strace printed both the symbolic name and the message: -1 EACCES (Permission denied).
cat checked the return value, looked up the message, and printed cat: /etc/shadow: Permission denied.

The message the user reports to you is the fourth link in that chain. Being able to walk back to link one is the difference between guessing and diagnosing.

Notice also that both failures produced an identically-shaped openat line and a completely different errno. When a colleague says "it says permission denied", the actual question is which errno — EACCES sends you to file modes and directory traversal, EPERM sends you to capabilities and security modules, and they look the same to a user.

Now imagine this at 500 hosts. A deployment fails on 3 of 500 hosts with "permission denied" in the application log. The log gives you link four. One strace -f -e trace=%file -o /tmp/t.txt <command> on one failing host gives you link one, and typically the exact path that differs. This is the single highest-leverage use of strace in an operations career: not performance work, but finding the file the application will not tell you it could not open.

🧪 Exercise C4.2 — Failed system calls are normal (this is why you must not panic)

bash

# Count how many syscalls fail during a completely successful command
strace -c -o /tmp/py.txt python3 -c "print('ok')" 2>/dev/null || \
  strace -c -o /tmp/py.txt bash -c "echo ok"
grep -E "errors|total|openat|stat" /tmp/py.txt

# Look at what those failures actually were
strace -e trace=openat bash -c "echo ok" 2>&1 | grep ENOENT | head -5

✅ Expected result — click to reveal

plain text

$ grep -E "errors|total|openat|stat" /tmp/py.txt
% time     seconds  usecs/call     calls    errors syscall
  8.12    0.000411           6        62        18 openat
  3.94    0.000199           4        44         9 newfstatat
100.00    0.005063                    714        31 total

$ strace -e trace=openat bash -c "echo ok" 2>&1 | grep ENOENT | head -5
openat(AT_FDCWD, "/usr/local/lib/tls/x86_64/libtinfo.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT
openat(AT_FDCWD, "/usr/local/lib/tls/libtinfo.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT
openat(AT_FDCWD, "/usr/local/lib/x86_64/libtinfo.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT
openat(AT_FDCWD, "/usr/local/lib/libtinfo.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/tls/libtinfo.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT

What to read out of it. A completely successful command produced 31 failed system calls, and that is entirely healthy.

Look at the pattern in the ENOENT lines: the dynamic linker is walking its search path, trying libtinfo.so.6 in one directory after another until it finds it. Every miss is an ENOENT. Python doing an import behaves identically, walking sys.path. Searching by failing is how path resolution works everywhere in Unix.

The trap this exercise exists to prevent. You will one day strace a broken process, see a screen of ENOENT, and conclude you have found the bug. You almost certainly have not. Errors in strace output are the default state.

What is actually diagnostic:

The last failure before the process gave up or exited.
A failure whose path is one you did not expect — a config file in the wrong place, a socket that should exist.
A failure the program did not recover from, which you spot because the next line is exit_group or write(2, "error...") to stderr rather than another attempt.

Reading a trace backwards from the end is usually faster than reading it forwards.

C5 · What crossing costs — mode switch, context switch, and the vDSO

Official docs: vdso(7) — the virtual dynamic shared object · syscalls(2) · intro(2)

Two terms get used interchangeably in blog posts and they are not the same thing. Interviewers ask about this specifically because the confusion is so widespread.

Mode switch — what a syscall does

Ring 3 to ring 0 and back, within the same process.

Save a few registers, change privilege level, jump to the kernel entry point. Nothing is unloaded; the kernel was already mapped into this address space.

Roughly tens to a few hundred nanoseconds on modern hardware.

The scheduler is not involved. Your process never stopped being the running process.

Context switch — a different thing entirely

The scheduler takes the CPU away from one process and gives it to another.

Save the full register set, switch the page tables, flush or tag the address-translation cache, and start again with cold CPU caches.

Roughly a few microseconds of direct cost, and often far more in lost cache warmth.

An order of magnitude or two more expensive than a mode switch.

They are related but not causally linked. A system call causes a mode switch. It causes a context switch only if it blocks — read on an empty socket, sleep, waiting on a lock. A getpid never blocks, so it is a pure mode switch. A read from page cache usually is too. That distinction is exactly what Module 07 uses to explain voluntary versus involuntary context switches, and what Module 02 uses to explain process states.

Real-world analogy — your own stockroom versus a change of tenant

You run a small shop.

A mode switch is stepping through the door into your own stockroom. You are still the shopkeeper. Your stock is still on the shelves. The till is where you left it. You are back at the counter in seconds, because nothing moved.

That is ring 3 to ring 0. Same process, same memory, kernel already there.

A context switch is your lease ending and a different business moving into the unit. Your stock is boxed up and carried out. Theirs is carried in. The sign above the door changes.

When you eventually come back, you have to unpack everything and remember where it all went. The removal van is the obvious cost. The unpacking is the bigger one — and that unpacking is the CPU rebuilding caches it lost.

That is the honest reason one takes tens of nanoseconds and the other takes microseconds or more.

It also explains why one does not always cause the other. A system call is always a trip to the stockroom. It only becomes a change of tenant if you have to wait in there — and if you are going to be waiting, the landlord may as well let someone else trade meanwhile.

And the cost of the trips themselves, which is what Exercise C5.1 measures: posting two thousand letters one envelope at a time, versus posting one parcel. Identical contents. The cost was never the weight. It was the two thousand walks to the post office.

That is why moving 2 MB one byte at a time takes five times longer than moving 2 GB in big blocks, with no disk involved in either.

🧪 Exercise C5.1 — Measure what the boundary costs

bash

# Same data, two block sizes. Count the system calls in each.
strace -c -o /tmp/small.txt dd if=/dev/zero of=/dev/null bs=1    count=100000
strace -c -o /tmp/large.txt dd if=/dev/zero of=/dev/null bs=100K count=1
grep -E "read|write|total" /tmp/small.txt /tmp/large.txt

# Now the wall-clock cost, without strace, at a realistic scale.
# First: 2 MB moved one byte at a time.
time dd if=/dev/zero of=/dev/null bs=1 count=2000000
# Then: 2 GB moved a megabyte at a time. That is 1000x more data.
time dd if=/dev/zero of=/dev/null bs=1M count=2000

✅ Expected result — click to reveal

plain text

$ grep -E "read|write|total" /tmp/small.txt /tmp/large.txt
/tmp/small.txt: 47.31    0.041882           0    100000           read
/tmp/small.txt: 46.02    0.040741           0    100000           write
/tmp/small.txt:100.00    0.088521                200042           total
/tmp/large.txt:  2.11    0.000009           4         2           read
/tmp/large.txt:  1.94    0.000008           4         2           write
/tmp/large.txt:100.00    0.000427                    46           total

$ time dd if=/dev/zero of=/dev/null bs=1 count=2000000
2000000+0 records in
2000000+0 records out
2000000 bytes (2.0 MB, 1.9 MiB) copied, 1.71581 s, 1.2 MB/s

real    0m1.718s
user    0m0.402s
sys     0m1.309s

$ time dd if=/dev/zero of=/dev/null bs=1M count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 0.341044 s, 6.1 GB/s

real    0m0.343s
user    0m0.001s
sys     0m0.335s

What to read out of it — this is the headline result of Part C.

Moving 2 MB took 1.72 seconds. Moving 2 GB — a thousand times more data — took 0.34 seconds. The small transfer was about 5,000 times slower per byte.

No disk was involved. /dev/zero and /dev/null are pure kernel constructs; not one byte moved through hardware. The entire 1.7 seconds was four million boundary crossings.

Now read the time breakdown, because it names the culprit precisely:

Small blocks: sys 1.309s against user 0.402s. The overwhelming majority of the time was spent in the kernel, and almost none of it doing anything useful.
Large blocks: sys 0.335s, user 0.001s, and it moved a thousand times the data.

A high sys time with nothing to show for it is the fingerprint of syscall overhead. That is a diagnosis you can now make from time alone.

Now imagine this at 500 hosts. This is not a contrived benchmark; it is one of the most common real performance bugs there is. An application reading a file a line at a time without buffering, a log shipper doing one write per event, a health check opening a new connection every second — all of them are this exercise in production clothing.

It is also why io_uring exists, why Nginx batches writes, why every language's I/O library buffers by default, and why "just add a buffer" is sometimes a hundredfold speedup for a one-line change. When you see high sys time and low throughput, count the syscalls before you touch anything else.

The obvious question after that exercise: if crossing is expensive, what about calls that happen constantly and do almost nothing — like asking the time? A logging library might read the clock a million times a second.

The answer is the vDSO, the linux-vdso.so.1 with no path on disk that you saw in Exercise C2.1. It is a small shared library the kernel maps into every process's address space at startup. It contains real, executable ring-3 code for a handful of read-only operations, and the kernel keeps a shared page of data — the current time, most importantly — up to date behind it. So a program asking the time runs an ordinary function call and never crosses the boundary at all.

Real-world analogy — the clock on the waiting-room wall

In a busy doctor's waiting room, everyone wants to know the time. If every person queued at reception to ask, the receptionist would do nothing else all day — hundreds of interruptions for a question with the same answer every time.

So the surgery hangs a clock on the waiting room wall. The receptionist keeps it right. Everyone else just looks up. No queue, no interruption, and the answer is just as accurate.

That is exactly what the vDSO is. The kernel puts a small piece of real, runnable code into every process, plus a page of data it keeps up to date. So a program asking the time makes an ordinary function call and never enters the kernel at all.

In Exercise C5.2, twenty thousand clock reads produce zero clock system calls.

Notice what it is not. It is not a copy your program made, and it is not a trick. It is real code, and it is readable and runnable but not writable — the surgery hangs the clock, and patients cannot move the hands.

Why this matters in production. The clock only helps if the surgery can keep it right.

On some virtual machines and older hardware, the system falls back to a poor timing source. When that happens the shortcut is switched off and everyone starts queuing at reception again.

On a service that writes a lot of timestamps, that shows up as a big, unexplained rise in system CPU time after a migration, with no code change to blame it on.

🧪 Exercise C5.2 — Find the system call that isn't there

bash

# Confirm the vDSO is mapped into this shell's address space
grep vdso /proc/self/maps

# Read the clock 20,000 times, then count the clock system calls.
# EPOCHREALTIME is a bash builtin that reads the current time.
strace -c -o /tmp/vdso.txt bash -c 'for i in $(seq 1 20000); do x=$EPOCHREALTIME; done'
grep -E "clock_gettime|gettimeofday|total" /tmp/vdso.txt

# For contrast, 20,000 calls that CANNOT be served from user space:
strace -c -o /tmp/nov.txt bash -c 'for i in $(seq 1 20000); do x=$(</proc/uptime); done'
grep -E "openat|read|total" /tmp/nov.txt

✅ Expected result — click to reveal

plain text

$ grep vdso /proc/self/maps
7ffd1c3f9000-7ffd1c3fb000 r-xp 00000000 00:00 0     [vdso]

$ grep -E "clock_gettime|gettimeofday|total" /tmp/vdso.txt
100.00    0.001204                    421        18 total

$ grep -E "openat|read|total" /tmp/nov.txt
 31.02    0.038114           1     20009        4 openat
 28.55    0.035082           1     40018          read
100.00    0.122871                   80663        6 total

What to read out of it. In the first summary there is no clock_gettime line at all. The shell read the clock twenty thousand times and made zero clock system calls. The total syscall count for the whole loop is 421 — process startup and nothing else.

The second loop asked the kernel for /proc/uptime the same number of times and produced 20,009 openat and 40,018 read calls: 80,663 syscalls in total, and about a hundred times the time spent. Same number of iterations, same kind of question, two completely different costs — because one answer could be published into user space and the other could not.

The r-xp on the vDSO mapping is worth noticing: readable and executable, not writable. It really is code, running in ring 3, in your process. The kernel wrote it there and cannot let you modify it.

Interview-grade detail. Being able to name the vDSO shows you understand that the syscall boundary has a measurable cost the kernel actively works around, rather than being a formality.

The operational payoff is concrete: gettimeofday and clock_gettime are served from the vDSO only when the system clocksource supports it. If a host falls back to the hpet or acpi_pm clocksource — which happens on some virtualised and older hardware — the vDSO path is disabled and every clock read becomes a real system call. On a logging-heavy service that shows up as a large, unexplained jump in sys time after a migration, with no code change. Check it with cat /sys/devices/system/clocksource/clocksource0/current_clocksource; you want to see tsc or kvm-clock, not hpet.

🎯 Interview questions — The system call boundary

Q. What is a system call? Walk me through what happens when a program calls one.

A system call is the mechanism by which a ring-3 program requests a service from the kernel — the only legitimate way across the privilege boundary. There are roughly 450 on a current x86-64 kernel.

Walk the sequence, in order:

The program calls a libc wrapper such as write().
The wrapper places the syscall number in a register and the arguments in others, then executes the syscall instruction.
The CPU switches to ring 0 and jumps to a fixed kernel entry point — the program does not choose the destination.
The kernel looks the number up in the syscall table, validates the arguments, and does the work.
It returns a single value, and sysret returns the CPU to ring 3.
libc inspects that value; if it is negative it stores the absolute value in errno and returns -1.

The details that separate candidates, in rough order of impact:

The kernel does not know about errno. It returns one number. errno is a user-space variable libc maintains — which is why errno is documented in section 3 and write in section 2.
The number, not the name, is the ABI. write is 1 on x86-64 and 64 on ARM64. This is why syscall numbers are essentially never reused.
A syscall is a mode switch, not a context switch — and being able to say why is the next question.

Q. What is the difference between a mode switch and a context switch?

A mode switch changes the CPU's privilege level between ring 3 and ring 0 within the same process. A context switch changes which process or thread the CPU is executing.

Why the cost differs, mechanically — this is the part that demonstrates real understanding:

A mode switch saves a small register set and changes privilege level. The address space does not change, because the kernel is already mapped into every process. Cost: tens to a few hundred nanoseconds.

A context switch saves the full register set, switches page tables, and invalidates or re-tags the TLB, so the new process starts with cold address-translation and data caches. Cost: a few microseconds directly, frequently much more in lost cache warmth — and the indirect cost is usually the larger one.

They are not causally linked. Every system call causes a mode switch. It causes a context switch only if it blocks. getpid() never blocks. A read() served from page cache usually does not. A read() that has to wait for the network always does.

The detail that separates candidates: mention KPTI, the Meltdown mitigation. It removed the assumption that the kernel is cheaply mapped into every address space, so post-2018 kernels do a page-table switch on syscall entry too — measurably raising syscall cost on affected CPUs. Syscall-heavy workloads lost real throughput to it. Bringing that up shows you know this is live engineering with production consequences, not a textbook diagram.

Q. A process is using 90% CPU but it is all system time. How do you investigate?

High sys time means the CPU is executing kernel code on this process's behalf. Something is either making an enormous number of system calls, or making expensive ones.

The order to work in:

Confirm the split. pidstat -u 1 separates %usr from %system per process, so you are certain which one is high before you spend effort.
Count the calls. strace -c -f -p PID for a few seconds. You are reading two columns: a very high calls count means chattiness; a high usecs/call on a low count means blocking.
Read the shape. Millions of small read/write calls means unbuffered I/O. Heavy futex traffic means lock contention. Heavy mmap/munmap or brk means memory-allocation churn. Heavy epoll_wait returning immediately means a busy-poll loop.
Confirm without stopping the process. perf top -p PID shows which kernel functions are actually on-CPU, and unlike strace it samples instead of halting the target.

The details that separate candidates:

Say out loud that strace is dangerous here. It stops the target at every system call and can slow a busy process by 10x or more. On a production host you time-box it, or you use perf or eBPF instead. Reaching for strace on a hot production process with no caveat is a genuine red flag to an experienced interviewer.
Give the classic cause and its fix: an application doing one write per log line with no buffering. Two million syscalls to move two megabytes — exactly the shape from Exercise C5.1.
Do not forget the non-application answers. High sys time across many processes points at the host, not the app: an interrupt storm, a failing NIC, a filesystem in trouble, or heavy page reclaim. mpstat -P ALL 1 will show whether it is one CPU or all of them, which splits those cases immediately.

Q. What is the vDSO and why does it exist?

The vDSO — virtual dynamic shared object — is a small shared library the kernel maps into every process's address space at startup. It holds real ring-3 code for a few read-only operations, most importantly reading the clock, backed by a shared page the kernel keeps current.

It exists because some "system calls" are called extremely often and do almost no work. clock_gettime on a logging-heavy service can be called millions of times a second. Paying a mode switch each time is pure overhead, so the kernel publishes the answer into user space instead and lets the process read it with an ordinary function call.

Typically served this way: clock_gettime, gettimeofday, time, getcpu. You can see it in any process with grep vdso /proc/self/maps, or in ldd output as linux-vdso.so.1 with no path on disk.

The detail that separates candidates: the failure mode. The vDSO fast path works only if the clocksource supports it. A host that falls back to hpet or acpi_pm — which does happen on some virtualised and older hardware — loses it, and every clock read becomes a genuine system call. The symptom is a large unexplained rise in sys time on a timestamp-heavy service after a migration, with no code change to blame. cat /sys/devices/system/clocksource/clocksource0/current_clocksource settles it in one command: you want tsc or kvm-clock.

📚 Part D · Reading the official documentation

D1 · The manual, and why the section number matters

Official docs: man(1) · man-pages(7) — section conventions · intro(2)

Every documentation link in this module has an offline twin already installed on your machine. Learning to reach it is worth more than any bookmark, because it works on a production host with no internet access at three in the morning.

The manual is divided into numbered sections, and the numbering is not decorative. It is the same distinction Part C spent its time on: section 2 is the kernel, section 3 is the library.

Section	Contains	What you go there for
1	Commands a user can run from a shell	man 1 ls, man 1 strace
2	System calls — the kernel interface	man 2 write, man 2 openat. Return values and the kernel's contract
3	Library functions — everything else in libc	man 3 printf, man 3 errno. The wrapper layer from Section C2
4	Device files under /dev	man 4 null, man 4 random
5	File formats and configuration files	man 5 proc, man 5 os-release, man 5 fstab
7	Overviews, conventions, protocols	man 7 vdso, man 7 credentials. The best writing in the whole manual
8	System administration commands	man 8 mount, man 8 lspci

Why this is not pedantry. write exists in section 2 as a system call and printf exists in section 3 as a library function, and that split is exactly the boundary you spent Part C learning. When you type man write you get whichever section the system finds first, which may not be the one you meant.

The habit worth building: when you want the kernel's contract, say the section. man 2 write tells you what the kernel guarantees — including the detail that a successful write may write fewer bytes than you asked for, which is one of the most common sources of subtle bugs in networking code and is documented in exactly one place.

Real-world analogy — asking a library for "Mercury"

Walk into a library and ask for a book on Mercury.

The librarian's first question is: the planet, the metal, or the god?

Three completely different books, in three different parts of the building. Only one of them is what you meant. The classification number is not red tape — it is the part of your request that carries your actual meaning.

Typing man write is asking for "Mercury" and taking whichever book is nearest the door. There are four to choose from:

write(1) — a command that sends a message to another logged-in user. Completely unrelated.
write(2) — the system call. What you meant.
write(1posix) — the POSIX standard's version of the section-1 command.
write(3posix) — the POSIX standard's description of the function, rather than Linux's.

So man 2 write is not pedantry, it is naming the shelf. And the split is exactly the one Part C spent its time on: section 2 is the kernel, section 3 is the library — the airline and the travel agent, filed separately because they are separately responsible.

The payoff is concrete. Only man 2 write tells you that a successful write may transfer fewer bytes than you asked for. That single sentence is the root of a classic family of socket bugs, it is not in most tutorials, and it is sitting on every one of your servers right now with no internet required.

🧪 Exercise D1.1 — Read the same name in two sections and see the boundary

bash

# Section 2 - the system call. Read the RETURN VALUE and ERRORS sections.
man 2 write | head -40

# How many manual pages exist for the name 'write'?
whatis write

# Now something only the manual will tell you. Find the sentence in
# man 2 write about partial writes:
man 2 write | grep -A6 "may transfer fewer"

# And the errno list for the same call
man 2 write | sed -n '/^ERRORS/,/^CONFORMING\|^STANDARDS/p' | head -25

✅ Expected result — click to reveal

plain text

$ whatis write
write (1)            - send a message to another user
write (2)            - write to a file descriptor
write (1posix)       - write to another user
write (3posix)       - write on a file

$ man 2 write | grep -A6 "may transfer fewer"
       Note that a successful write() may transfer fewer than count bytes.
       Such partial writes can occur for various reasons; for example,
       because there was insufficient space on the disk device to write all
       of the requested bytes, or because a blocked write() to a socket,
       pipe, or similar was interrupted by a signal handler after it had
       transferred some, but before it had transferred all of the requested
       bytes.

$ man 2 write | sed -n '/^ERRORS/,/^CONFORMING\|^STANDARDS/p' | head -25
ERRORS
       EAGAIN The file descriptor fd refers to a file other than a socket
              and has been marked nonblocking (O_NONBLOCK) ...
       EBADF  fd is not a valid file descriptor or is not open for writing.
       EDQUOT The user's quota of disk blocks ... has been exhausted.
       EFAULT buf is outside your accessible address space.
       EFBIG  An attempt was made to write a file that exceeds the ...
       EINTR  The call was interrupted by a signal before any data was ...
       EIO    A low-level I/O error occurred while modifying the inode.
       ENOSPC The device containing the file referred to by fd has no room.
       EPERM  The operation was prevented by a file seal.
       EPIPE  fd is connected to a pipe or socket whose reading end is
              closed. ...

What to read out of it.

whatis write shows four different pages for one name. Section 1 is a command that messages another logged-in user — completely unrelated. Section 2 is the system call you actually meant. Typing man write would have shown you the wrong one.

The partial-write paragraph is the reason this exercise exists. write(fd, buf, 1000) returning 800 is success, not failure. It returned a non-negative number, so by the convention in Section C4 nothing went wrong — and if your code assumes all 1000 bytes went out, you have silently lost 200 of them. This is one of the classic bugs in socket programming, and it is documented plainly in one place: the manual page.

The ERRORS list is the same thing from the other direction. Every errno from the Section C4 table appears here with the specific meaning it carries for this call. EPERM on write means a file seal, which is nothing like EPERM on ptrace. Generic errno tables are a starting point; the per-call ERRORS section is the actual answer.

Now imagine this at 500 hosts. Half the "mysterious" production bugs in low-level code are documented behaviours nobody read: partial writes, EINTR on interrupted calls, EAGAIN on non-blocking descriptors. All three are in man 2 write, offline, on every one of those 500 hosts, right now.

D2 · Finding the page when you do not know its name

Official docs: apropos(1) · man(1) — the -k and -f options · man-pages(7)

The manual is only useful if you can find things in it without already knowing the answer. Three commands cover it:

bash

apropos KEYWORD      # search page names and one-line descriptions (same as man -k)
whatis NAME          # the one-line description for an exact name (same as man -f)
man -K "phrase"      # slow full-text search of every page's body - a last resort

If apropos says "nothing appropriate", the database is missing, not the manual. Run sudo mandb to build it. On minimal container images the manual pages are stripped out entirely to save space — which is another reason your lab is a VM and not a container.

Real-world analogy — the shelf number versus the librarian

man 2 write is walking straight to a shelf because you already know the number. That only works when you know what you are looking for.

apropos is asking the librarian: "what do you have on process credentials?" You do not need the title, the author or the number — just the subject. She searches the catalogue's descriptions and hands you three books you did not know existed, which is the entire point.

And apropos -s 7 . is asking to see the whole reference section. Section 7 pages are not reference tables, they are the essays: sched(7), signal(7), namespaces(7), capabilities(7), cgroups(7), credentials(7). Written by the people who built the thing, in continuous prose, explaining rationale rather than parameters — which is precisely what tutorials leave out. That list is a syllabus for this whole track, and it is already installed.

Where the analogy breaks, usefully. A librarian knows her stock whether or not anyone has indexed it. apropos reads a database that must be built, so "nothing appropriate" usually means the index is missing, not the book — run sudo mandb. On minimal container images the manual pages are stripped out entirely to save space, which is one more reason your lab is a VM.

🧪 Exercise D2.1 — Find pages you did not know existed

bash

# Everything about process credentials
apropos -s 2,3,7 credential

# Which section-7 overviews exist at all? These are the best pages in the manual.
apropos -s 7 . | head -30

# Search within a single section
apropos -s 2 "memory"

# If this returns nothing, build the index first
sudo mandb 2>/dev/null | tail -2

✅ Expected result — click to reveal

plain text

$ apropos -s 2,3,7 credential
credentials (7)      - process identifiers

$ apropos -s 7 . | head -30
aio (7)              - POSIX asynchronous I/O overview
capabilities (7)     - overview of Linux capabilities
cgroups (7)          - Linux control groups
credentials (7)      - process identifiers
epoll (7)            - I/O event notification facility
inode (7)            - file inode information
mount_namespaces (7) - overview of mount namespaces
namespaces (7)       - overview of Linux namespaces
pid_namespaces (7)   - overview of PID namespaces
pipe (7)             - overview of pipes and FIFOs
sched (7)            - overview of CPU scheduling
signal (7)           - overview of signals
socket (7)           - Linux socket interface
vdso (7)             - overview of the vDSO

What to read out of it. That second list is, without exaggeration, a syllabus for this entire track — written by the people who built the thing, installed on your machine, free.

sched(7) is Module 07. signal(7) is Module 04. credentials(7) is Module 02. namespaces(7) and cgroups(7) are Module 12. capabilities(7) is Module 13. pipe(7) is Module 11. inode(7) is Module 03.

Section 7 pages are overviews, not reference tables. They explain mechanisms and rationale in continuous prose, which is precisely what most online tutorials leave out. When you finish a module in this track, reading the matching section-7 page is the highest-value follow-up available, and it costs nothing.

Interview-grade habit. Being able to say "I checked man 7 signal and the disposition is inherited across exec except for handled signals, which are reset to default" is a categorically different answer from "I think it's inherited". Interviewers notice when a candidate's knowledge has a source, especially when the source is the one shipped on the machine.

D3 · Reading kernel documentation effectively

Official docs: Kernel admin guide index · The /proc filesystem · Kernel command-line parameters

Kernel documentation intimidates people because they open it at the wrong door. There are really only three doors, and knowing which to use eliminates most of the difficulty.

Door	Who it is written for	Use it when
man-pages (sections 2, 3, 5, 7)	Application developers and operators	Almost always. This is the userspace-facing contract
docs.kernel.org/admin-guide	System administrators	Tunables, boot parameters, cgroups, subsystem behaviour
docs.kernel.org — everything else	Kernel developers	Rarely. Only when you need internals, and expect C

Three habits that make official docs faster than searching the web.

Check the version. docs.kernel.org defaults to the latest kernel; your host is probably older. Behaviour genuinely changes between versions, so match the docs to uname -r before trusting a detail.

Read man 7 X before man 2 X. The section-7 overview gives you the model; the section-2 page gives you the parameters. Reading them the other way round is how documentation becomes intimidating.

In any man page, read the sections in this order: DESCRIPTION → RETURN VALUE → ERRORS → NOTES → BUGS. NOTES and BUGS carry the Linux-specific behaviour and the genuine surprises, and almost nobody reads that far.

Real-world analogy — three manuals for the same car

Every car has three levels of documentation. Picking the wrong one is the main reason people think documentation is impossible to read:

The owner's handbook — how to check the oil, what the warning lights mean. Written for the person driving. That is man-pages, and it is the right door almost every time.
The workshop manual — torque settings, service intervals, diagnostic procedures. Written for a mechanic. That is docs.kernel.org/admin-guide: tunables, boot parameters, subsystem behaviour.
The engineering drawings — the casting tolerances of the cylinder head. Written for the people who designed it. That is the rest of docs.kernel.org, and expect C.

Someone who opens the engineering drawings to find out how to check the oil concludes, reasonably, that car documentation is written for nobody. They opened the wrong book.

Two habits follow directly:

Match the manual to the car in front of you. docs.kernel.org shows the newest kernel; yours is probably older, and behaviour genuinely changes. Check uname -r first — nobody services a 2019 engine from a 2026 manual.

Read man 7 X before man 2 X. The overview gives you the model, the reference gives you the parameters. Doing it the other way round is reading a parts list before you know what the part does. And in any page, the order that pays is DESCRIPTION → RETURN VALUE → ERRORS → NOTES → BUGS — the Linux-specific surprises live at the bottom, where almost nobody reads.

🧪 Exercise D3.1 — Confirm the docs match the machine in front of you

bash

# The kernel this documentation must match
uname -r

# Kernel tunables are documented AND live-readable. Same fact, two places.
sysctl kernel.pid_max
cat /proc/sys/kernel/pid_max

# Every tunable your kernel exposes, counted
sysctl -a 2>/dev/null | wc -l

# The offline copy of the kernel's own build configuration
grep -c . /boot/config-$(uname -r)

✅ Expected result — click to reveal

plain text

$ uname -r
6.8.0-45-generic

$ sysctl kernel.pid_max
kernel.pid_max = 4194304

$ cat /proc/sys/kernel/pid_max
4194304

$ sysctl -a 2>/dev/null | wc -l
1287

$ grep -c . /boot/config-6.8.0-45-generic
11842

What to read out of it. sysctl kernel.pid_max and cat /proc/sys/kernel/pid_max return the identical value because they are the same thing. sysctl is a thin convenience wrapper over /proc/sys, translating dots into slashes. Knowing that means you never need sysctl to be installed; on a stripped-down image cat and echo do the same job.

1,287 tunables. Every one of them is a decision someone made about how your machine behaves, and the overwhelming majority are documented under docs.kernel.org/admin-guide/sysctl/. When a blog post tells you to set a mysterious vm. or net.ipv4. value, that directory is where you find out what it actually does before you apply it to 500 hosts.

The 11,842-line /boot/config-* file is the complete build configuration of the exact kernel you are running — which features exist, which are modules, which were compiled out. When documentation describes a feature and your kernel does not have it, this file settles the argument in one grep, as you already did for CONFIG_HZ in Exercise A7.1.

🏁 Part E · Practice, capstone, reference and review

E1 · Production practice

Everything in this module maps onto something you will actually do. This table is the translation.

Situation	What you now reach for	What it tells you
Host is busy but no process looks guilty	vmstat 1, then /proc/interrupts twice ten seconds apart	High in with low us means device or interrupt work, not application work
Process at 100% CPU, all system time	pidstat -u 1 to confirm the split, then a time-boxed strace -c -f -p PID	Many calls means chattiness; few calls with high usecs/call means blocking
Application fails with "permission denied" and unhelpful logs	strace -f -e trace=%file -o /tmp/t.txt CMD, then read from the end	The exact path and the exact errno. EACCES and EPERM send you to different places
Service still vulnerable after patching	ls -l /proc/PID/exe — look for (deleted)	The process is still running the old code from memory. It needs a restart
Container sized wrongly, or OOM-killed with no app error	Compare nproc and free -h inside the container against the host	The runtime sized itself from host figures, not from its limits
Hardware present but not working	lspci -k, then dmesg	Missing Kernel driver in use means detected but unclaimed — not a dead device
One core saturated while the rest idle	/proc/interrupts per-CPU columns, mpstat -P ALL 1	Interrupts pinned to one CPU. A capacity ceiling invisible to any average
Process is slow and you must not stop it	grep ctxt /proc/PID/status	High voluntary means blocked on I/O or locks; high involuntary means CPU contention
sys time jumped after a migration, no code change	cat /sys/devices/system/clocksource/clocksource0/current_clocksource	hpet instead of tsc disables the vDSO fast path for clock reads
You need documentation on an air-gapped host	apropos -s 7 ., then man 7 TOPIC	The authoritative overview is already installed

E2 · Capstone exercise

🧪 CAPSTONE — Profile an unfamiliar command end to end, using only this module

No new tools. The goal is to produce a written diagnosis, in your own words, using only what Parts A to D taught.

bash

# The subject: a command that does real work across several subsystems.
# Substitute anything you like that reads many files.

# 1. Predict FIRST, in writing, before running anything:
#    - Will this be dominated by syscall count, or by blocking?
#    - Which hardware will it touch? Which will it not?
#    - Will it be preempted often, or will it block often?

# 2. Baseline the machine
grep -E 'CPU0|LOC|virtio|nvme|ahci' /proc/interrupts > /tmp/irq-before.txt
vmstat 1 3

# 3. Measure the command
time find /usr -type f -name "*.so*" > /tmp/found.txt
strace -c -f -o /tmp/cap.txt find /usr -type f -name "*.so*" > /dev/null

# 4. Collect the evidence
head -15 /tmp/cap.txt
grep -E 'CPU0|LOC|virtio|nvme|ahci' /proc/interrupts > /tmp/irq-after.txt
diff /tmp/irq-before.txt /tmp/irq-after.txt

# 5. And for a process that blocks instead, for contrast
strace -c -f -o /tmp/net.txt getent hosts kernel.org
head -8 /tmp/net.txt

✅ What a good answer looks like — click to reveal

You are not being marked on numbers. You are being marked on whether your diagnosis follows from your evidence. A complete answer states all six of these:

Which syscalls dominate, and what that means. find is dominated by newfstatat and getdents64 in the tens of thousands. That is the chatty shape from Section C3: many calls, low usecs/call. It is doing work, not waiting.
The user/system split from time. sys should substantially exceed user. find does almost no computation — it asks the kernel about files. Compare this against Exercise C5.1, where high sys with nothing achieved was the fingerprint of pointless syscall overhead. Here the syscalls are the actual work, which is why the same signal means something different.
Which interrupts moved and which did not. Block-device interrupts rise only if the data was not already cached. Run it twice: the second run is far faster and moves far fewer disk interrupts, because the page cache served it. Network interrupts should not move at all. You have just proved which hardware a workload uses, with no instrumentation.
The contrast case. getent hosts shows the opposite shape: a handful of calls, most of the time in one or two of them. Low count, high usecs/call — blocked on the network. Adding CPU would do nothing for it, and adding CPU would genuinely help find.
The errors are not the bug. Your trace will contain ENOENT and probably EACCES on directories you cannot enter. find completed anyway. Section C4: failed system calls are the default state, and only the last one before a give-up is diagnostic.
The preemption evidence. If you also captured /proc/PID/status: on a cold page cache find blocks on the disk constantly and shows a high voluntary switch count. Run it a second time, with everything cached, and it stops blocking — it becomes syscall-bound and the switches shift towards involuntary. Compare either with the spinner in Exercise A7.1, which was almost entirely involuntary from the start. Same two counters, opposite diagnoses, opposite remedies.

Why this is the capstone. This is the actual shape of production diagnosis: form a hypothesis, choose a measurement that can disprove it, read the shape rather than the absolute numbers, and use a contrast case to show your interpretation is not an artefact. Every later module gives you more instruments; none of them replaces this loop.

E3 · Official documentation reference

Every link below was checked before it was published here. Each has an offline twin on your machine, given in the third column.

Topic	Official page	Offline equivalent
System calls, all of them	syscalls(2)	man 2 syscalls
Syscall conventions, errno	intro(2) · errno(3)	man 2 intro · man 3 errno
The /proc filesystem	proc(5) · kernel docs	man 5 proc
Per-process status fields	proc_pid_status(5)	man 5 proc_pid_status
sysfs and the device model	sysfs(5)	man 5 sysfs
Interrupt counters	proc_interrupts(5)	man 5 proc_interrupts
DMA	Dynamic DMA mapping guide	—
Device drivers	Driver implementer's API guide · lspci(8)	man 8 lspci
Timers and tickless	Timers — kernel docs	/boot/config-$(uname -r)
The vDSO	vdso(7)	man 7 vdso
Tracing and ptrace policy	strace(1) · Yama LSM	man 1 strace
Kernel administration	Admin guide · Boot parameters	sysctl -a
Identity and virtualisation	uname(1) · os-release(5) · systemd-detect-virt(1)	man 5 os-release
Using the manual itself	man(1) · man-pages(7) · apropos(1)	man 7 man-pages
The standard itself	POSIX.1-2024 — Base Specifications Issue 8	man 7 standards
Lab environment	Multipass documentation	multipass help

Your offline documentation toolkit, in one place. man N NAME for a specific page. apropos KEYWORD to search names and descriptions. whatis NAME for the one-line summary. man -K "phrase" for slow full-text search. sudo mandb when apropos finds nothing. sysctl -a for every kernel tunable, and /boot/config-$(uname -r) for what your kernel was actually built with. None of these needs a network connection.

E4 · Self-assessment

Answer these out loud, without opening anything. If an answer comes out as a definition rather than a mechanism, that is the section to re-read.

Your program executes an infinite loop and makes no system calls. Explain, mechanically, how the kernel ever runs again — and why software alone cannot achieve this.
A colleague says "root can do anything on Linux." Give a specific counter-example from this module and explain what root actually is.
write(fd, buf, 1000) returns 800. Did it succeed or fail? What must correct code do next, and where is this documented?
Explain the difference between a mode switch and a context switch, including why one is roughly an order of magnitude more expensive.
A container is limited to 0.5 CPU and 256 MB. Inside it, nproc reports 2 and free -h reports 1.9 GB. Explain why, and name one concrete production failure this causes.
strace -c on a process shows 4 million write calls and a sys time far exceeding user time. What is your diagnosis, and what is the fix?
ls -l /proc/uptime reports 0 bytes but cat returns content. Explain what /proc actually is, and give one operational consequence.
Trace the complete path of a disk read from the read() system call to the data arriving in your process, naming: what puts the process to sleep, what moves the bytes, what wakes it, and why the interrupt handler is split in two.
A process shows voluntary_ctxt_switches: 12 and nonvoluntary_ctxt_switches: 48000. What kind of problem is this, and would adding CPU help? What would the opposite ratio mean?
You need to know whether a system call can return fewer bytes than requested, on an air-gapped production host with no internet. What exactly do you type?

E5 · Sources

Interview questions in this module were drawn from published 2026 question sets and then deepened with operational detail beyond the published answers.

Technical content is sourced from the official documentation listed in E3.

Next: Module 02 — Processes, fork/exec & Process States. It picks up exactly where Section A4 left off, and turns the three-resource sketch into the real process lifecycle.

Spotted a mistake or want something added? Send me a note.

🧱Module 01 — Kernel, User Space & the System Call Boundary

🧩 Part A · What an operating system actually is

A1 · The three problems an OS solves

A2 · Kernel space and user space — a boundary the CPU enforces

A3 · How you observe the kernel — /proc and /sys

A4 · Program, process, and the three resources the OS controls

A5 · How the kernel actually talks to hardware

A6 · Interrupts, polling and DMA — how input and output actually move

A7 · How the OS keeps control of the CPU — the timer interrupt

A8 · Monolithic, microkernel, hybrid — and where Linux actually sits

🎯 Interview questions — What an OS is

🛠️ Part B · Building your lab

B1 · Why a disposable virtual machine, and not your laptop

B2 · Installing the lab

B3 · Connecting and verifying it works

B4 · Installing the observation toolset

🎯 Interview questions — Environment and isolation

🚪 Part C · The system call boundary

C1 · What a system call actually is

C2 · The library wrapper and the raw system call

C3 · Reading strace output fluently

C4 · How system calls report failure — return values and errno

C5 · What crossing costs — mode switch, context switch, and the vDSO

🎯 Interview questions — The system call boundary

📚 Part D · Reading the official documentation

D1 · The manual, and why the section number matters

D2 · Finding the page when you do not know its name

D3 · Reading kernel documentation effectively

🏁 Part E · Practice, capstone, reference and review

E1 · Production practice

E2 · Capstone exercise

E3 · Official documentation reference

E4 · Self-assessment

E5 · Sources

Module 01 — Kernel, User Space & the System Call Boundary