Module 05 — Boot & Init: Firmware to systemd

Updated 22 August 2026

Module 05 · Boot and init — from firmware to systemd

This is the last module in the Foundation tier, and it is here on purpose. Booting looks like a beginner topic, but explaining it honestly needs PID 1, mounts, and signal handling — all of which you now have.

🧠 concept → 🧩 real-world analogy → 🧪 exercise → ✅ expected result (hidden) → 🎯 interview questions (hidden)

Before you start, you should already know:

From Module 01 — kernel versus user space, device drivers, and how to read /proc and /sys.

From Module 02 — what PID 1 is and why it is special.

From Module 03 — mounts, mount points, and how one filesystem hides another.

From Module 04 — signals, TERM versus KILL, and graceful shutdown.

Everything in this module is read-only or reversible. Nothing here will stop your lab booting — but do it on the lab VM anyway, not on a machine you care about.

🔌 Part A · Before the kernel runs

A1 · Firmware — the first code that runs

Official docs: bootup(7) — the whole boot sequence · Kernel admin guide · The kernel's command-line parameters

When you press the power button, there is no operating system. There is no filesystem, no driver, and nothing in memory. The CPU has to start executing something, and that something has to be in a fixed place it can find without help.

That is the firmware: code stored on a chip on the motherboard. Its job is small and specific:

Check the hardware is present and working.
Find something bootable.
Load the first part of it into memory and jump to it.

Then it is finished. Firmware does not stay resident and does not manage anything.

There are two kinds, and which one you have changes almost everything about the next step.

	BIOS (legacy)	UEFI (modern)
How it finds the boot code	Reads the first 512 bytes of a disk — the master boot record — and runs it.	Reads a real filesystem (a FAT partition, the ESP) and runs a named .efi file.
How much room the boot code has	446 bytes. Genuinely.	As much as it likes. It is an ordinary file.
Boot entries	None. There is one MBR.	A list stored in NVRAM on the motherboard, editable with efibootmgr.
Signature checking	No.	Yes — Secure Boot.

That 446-byte figure is the reason GRUB exists in the shape it does. You cannot fit a bootloader in 446 bytes, so on BIOS systems it is split into stages: a tiny first stage whose only job is to find and load a bigger second stage.

Real-world analogy — a relay race

Booting is a relay. Four runners, and each one's only real job is to hand the baton to the next.

Firmware runs the first leg. It knows almost nothing — no files, no network — but it knows where to find the next runner.
The bootloader runs the second leg. It can read a disk and knows how to start a kernel.
The kernel runs the third. Now there are drivers and memory management — and unlike the first two runners, the kernel never leaves the track; it stays underneath everything for as long as the machine is on. What it hands over is control of userspace, not the machine.
The initramfs init runs a short fourth leg: it loads the drivers it carries, mounts the real root, and hands over by execing the real init.
systemd runs the last leg, and it stays too.

The useful part of the picture is what happens at each handover: the first two runners stop. Firmware is not sitting underneath your running system supervising things. It handed over and finished. This is why "the BIOS is causing my performance problem" is almost always wrong — by the time Linux is up, the firmware has not executed an instruction in minutes.

Where the analogy stops working. Real runners all run the same kind of leg. Here, each stage is written in a different style, with different capabilities, and the handovers are where boot problems live. Nearly every boot failure you will debug happens at a handover — not in the middle of a leg.

🧪 Exercise A1.1 — Find out which firmware booted this machine

bash

# The one-command answer: if this directory exists, you booted with UEFI
[ -d /sys/firmware/efi ] && echo "UEFI" || echo "BIOS (legacy)"

# On UEFI, the firmware keeps a boot menu in NVRAM
if command -v efibootmgr >/dev/null && [ -d /sys/firmware/efi ]; then
  sudo efibootmgr -v | head -8
else
  echo "(efibootmgr not installed, or BIOS boot)"
fi

# The EFI System Partition is an ordinary FAT filesystem - Module 03 applies
findmnt /boot/efi 2>/dev/null || echo "(no ESP mounted)"

# What is actually on it?
sudo find /boot/efi -name '*.efi' 2>/dev/null | head -5

✅ Expected result — click to reveal

plain text

$ [ -d /sys/firmware/efi ] && echo "UEFI" || echo "BIOS (legacy)"
UEFI

$ sudo efibootmgr -v | head -8
BootCurrent: 0004
Timeout: 0 seconds
BootOrder: 0004,0000,0001
Boot0000* UiApp
Boot0001* UEFI QEMU DVD-ROM
Boot0004* ubuntu	HD(1,GPT,...)/File(\EFI\ubuntu\shimx64.efi)

$ findmnt /boot/efi
TARGET    SOURCE    FSTYPE OPTIONS
/boot/efi /dev/vda1 vfat   rw,relatime,fmask=0077,dmask=0077...

$ sudo find /boot/efi -name '*.efi' | head -5
/boot/efi/EFI/ubuntu/shimx64.efi
/boot/efi/EFI/ubuntu/grubx64.efi
/boot/efi/EFI/ubuntu/mmx64.efi
/boot/efi/EFI/BOOT/BOOTX64.EFI

What to read out of it.

/sys/firmware/efi exists only when the kernel was started by UEFI firmware. That single test is the reliable way to answer the question, and it is worth knowing because half of boot troubleshooting depends on the answer.

efibootmgr shows the firmware's boot menu, stored on the motherboard — not on the disk. BootOrder: 0004,0000,0001 is the order it will try them in, and BootCurrent: 0004 is the one that worked this time. You can add, remove and reorder these entries from a running Linux system, which is genuinely surprising the first time you see it.

The ESP is vfat — Module 03's filesystem table showed this type and it looked like a curiosity. Here is why it is there: UEFI firmware is only guaranteed to read FAT — the specification mandates FAT12/16/32 and nothing else, so every ESP is FAT. Every modern machine carries a small Microsoft-format partition for exactly this reason.

And notice shimx64.efi comes before grubx64.efi. The shim is a small signed program whose job is to satisfy Secure Boot and then load GRUB. That is an extra relay leg that exists purely because of signature checking.

Now imagine this at 500 hosts. The BIOS/UEFI split decides how you recover a machine that will not boot, and the two procedures share nothing.

It also causes a specific and nasty failure: a machine that boots fine until someone clears NVRAM or swaps the motherboard, at which point the boot entries are gone even though the disk is untouched. The disk is fine. The list of what to boot lived on the motherboard.

A2 · The bootloader

Official docs: bootup(7) · The kernel's command-line parameters · Kernel admin guide

The firmware has handed over. The bootloader — almost always GRUB on Linux — now has one job:

Load a kernel and an initramfs into memory, hand the kernel a line of text, and jump to it.

That is it. GRUB is not a small operating system, although it can look like one. It reads filesystems, shows a menu, and then gets out of the way permanently.

The files it deals with live in /boot:

vmlinuz-* — the compressed kernel image.
initrd.img-* or initramfs-* — a small temporary filesystem. Section B2 explains why this exists at all.
grub.cfg — the generated menu. You do not edit this file; it is regenerated from /etc/default/grub and the scripts in /etc/grub.d/.

The single most common self-inflicted boot failure. People edit /boot/grub/grub.cfg directly, it works, and then the next kernel update runs update-grub and silently overwrites every change.

Edit /etc/default/grub, then run update-grub (Debian/Ubuntu) or grub2-mkconfig -o /boot/grub2/grub.cfg (RHEL 9 and Fedora 34+; on RHEL 8 and earlier a UEFI install's real config is /boot/efi/EFI/redhat/grub.cfg). The generated file even says so in a comment at the top, which nobody reads.

Real-world analogy — the index card by the door

Imagine a library where the firmware is a caretaker who cannot read. All the caretaker knows is: go to the front desk and do what the card there says.

The card is GRUB. It says: "take the book called vmlinuz-6.8.0-45, take the box of tools called initrd.img-6.8.0-45, and give the reader this note before they start."

Three properties of the card matter, and each one is a real GRUB behaviour:

The card can list several options. That is the boot menu, and it is why you can pick an older kernel when a new one breaks.
The note is handed over at the start and cannot be changed afterwards. That is the kernel command line — Section A3.
The card is printed from a template. Writing on the card by hand works until somebody reprints it, and then your writing is gone. That is grub.cfg versus /etc/default/grub.

Where the analogy stops working, and it is the useful bit. A caretaker who cannot find the card can ask someone.

If GRUB cannot find its configuration, nothing can help it — there is no operating system yet to ask. You get a grub> prompt, and every recovery from that point involves typing the same commands GRUB would have read from the file. That is why knowing what the file contains is worth more than it looks.

🧪 Exercise A2.1 — Look at what GRUB was told to load

bash

# The files GRUB hands to the firmware's successor
ls -lh /boot/vmlinuz-* /boot/initrd.img-* 2>/dev/null | head -6

# The menu entries GRUB generated (titles only)
sudo grep -E "^menuentry|^\s+menuentry" /boot/grub/grub.cfg 2>/dev/null | \
  sed 's/ --class.*//' | head -6

# The file you are ACTUALLY supposed to edit
grep -vE '^\s*#|^\s*$' /etc/default/grub

# And the warning at the top of the generated file
sudo head -5 /boot/grub/grub.cfg 2>/dev/null

✅ Expected result — click to reveal

plain text

$ ls -lh /boot/vmlinuz-* /boot/initrd.img-*
-rw------- 1 root root  14M Jul 15 19:22 /boot/vmlinuz-6.8.0-45-generic
-rw-r--r-- 1 root root  62M Aug 17 09:11 /boot/initrd.img-6.8.0-45-generic

$ sudo grep -E "^menuentry|^\s+menuentry" /boot/grub/grub.cfg | sed 's/ --class.*//' | head -6
menuentry 'Ubuntu'
	menuentry 'Ubuntu, with Linux 6.8.0-45-generic'
	menuentry 'Ubuntu, with Linux 6.8.0-45-generic (recovery mode)'

$ grep -vE '^\s*#|^\s*$' /etc/default/grub
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`( . /etc/os-release; echo ${NAME:-Ubuntu} ) 2>/dev/null || echo Ubuntu`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX=""
GRUB_DISABLE_OS_PROBER=true

$ sudo head -5 /boot/grub/grub.cfg
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub

What to read out of it.

The two sizes are worth noticing. The kernel is 14 MB. The initramfs is 62 MB — more than four times bigger. Most people assume the kernel is the big thing. Section B2 explains what is in that 62 MB and why it is so large.

The menu has a recovery mode entry for each kernel. That is the same kernel with a different command line, which is Section A3.

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" is the line that matters most in practice. quiet raises the console log threshold so that only errors and worse are printed — it does not stop the messages being produced. When a machine hangs at boot with no output, removing quiet is the first thing you do — the messages were being produced all along and simply hidden.

And the generated file says DO NOT EDIT THIS FILE in its first three lines. Everyone who has broken a system this way had that warning on screen.

Now imagine this at 500 hosts. Keeping two working kernels installed is not housekeeping, it is your rollback plan. When a kernel update breaks boot, the previous entry in that menu is the recovery path, and it costs one reboot rather than a rescue disk.

This is also why aggressive apt autoremove of old kernels on a fleet is a bad habit, and why GRUB_TIMEOUT=0 on a physical server is worth reconsidering — with no timeout there is no menu, and no menu means no rollback without console access.

A3 · The kernel command line

Official docs: The kernel's command-line parameters · bootup(7) · proc(5)

GRUB hands the kernel one line of text. That line is the only configuration the kernel gets before it starts, and it is fixed for the life of that boot.

It matters far more than its size suggests, because it is the one place you can change kernel behaviour before anything else exists. No filesystem, no config file, no service — just this string.

You can read the current one at /proc/cmdline.

Parameter	What it does
root=UUID=...	Which filesystem to use as /. Given as a UUID because device names like /dev/sda are not stable.
ro	Mount the root filesystem read-only at first, so it can be checked before anything writes to it.
quiet	Hide kernel messages during boot. Remove this when debugging.
systemd.unit=rescue.target	Tell systemd to boot to a minimal state instead of the usual one.
init=/bin/bash	Run a shell as PID 1 instead of systemd. The classic password-recovery trick — but the same command line still carries ro, so you must mount -o remount,rw / before passwd, then sync; reboot -f.
nomodeset	Disable kernel mode-setting: DRM drivers stop taking over the display and the firmware framebuffer is used instead. Fixes a large fraction of "blank screen after upgrade".

init=/bin/bash is worth understanding rather than memorising, because it explains a security fact people find alarming.

Module 02 said PID 1 is the first user-space process the kernel starts. It did not say the kernel is fussy about which program that is. The kernel simply runs whatever init= names.

So init=/bin/bash gives you a root shell, with no password, on the real root filesystem — because there is no login process, no PAM, and no systemd. There is nothing to authenticate against.

That is not a bug. It is the direct consequence of the kernel not caring what PID 1 is. And it is why anyone with physical access and a keyboard is effectively root, and why full-disk encryption and a firmware password are the only real answers.

Real-world analogy — the note handed over at the start

A new employee arrives on their first day, before they have an email account, a desk, or access to any system. Somebody hands them a single index card at the door.

Everything on that card had to be decided in advance, because there is no way to reach them afterwards until they are set up. And once they are through the door, the card cannot be changed — you would have to send them home and start again. That is why the kernel command line is fixed for the boot and only a reboot can alter it.

One line on that card is "report to this manager", and that is init=. Write a different name and they report to somebody else entirely, with no questions asked, because there is nobody at the door checking. That is init=/bin/bash.

Where the analogy stops working, and it is the security point. In a real building, security would notice someone with a hand-altered card.

The kernel has no equivalent. It cannot tell a legitimate command line from one an attacker typed at the boot menu thirty seconds ago. The only defences are earlier in the relay: a GRUB password, a firmware password, Secure Boot, and full-disk encryption — because once the card is written, the kernel obeys it.

🧪 Exercise A3.1 — Read the line your kernel was given

bash

# The exact line handed over at boot
cat /proc/cmdline

# Which root filesystem does that UUID actually refer to?
findmnt / -o SOURCE,UUID,FSTYPE

# The kernel logs its own command line as one of its first messages
sudo dmesg | grep -i "command line" | head -2

# Prove it is fixed: there is no way to write to it
echo "test" | sudo tee -a /proc/cmdline

✅ Expected result — click to reveal

plain text

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.0-45-generic root=UUID=8f3a1c7e-2b44-4d9a-9c11-6e05a2f7d130 ro quiet splash

$ findmnt / -o SOURCE,UUID,FSTYPE
SOURCE    UUID                                 FSTYPE
/dev/vda2 8f3a1c7e-2b44-4d9a-9c11-6e05a2f7d130 ext4

$ sudo dmesg | grep -i "command line" | head -2
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.0-45-generic root=UUID=8f3a1c7e... ro quiet splash
[    0.023011] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.8.0-45-generic root=UUID=8f3a1c7e... ro quiet splash

$ echo "test" | sudo tee -a /proc/cmdline
tee: /proc/cmdline: Invalid argument
test

What to read out of it.

The UUID in /proc/cmdline matches the UUID in findmnt exactly. That is the whole mechanism for finding the root filesystem: GRUB wrote a UUID, and something later searched every block device for a filesystem carrying it.

Why a UUID rather than /dev/vda2? Because device names are assigned in the order the kernel finds hardware, and that order can change — add a disk, change a controller, boot on different virtual hardware, and yesterday's /dev/sda is today's /dev/sdb. A UUID is written inside the filesystem and travels with it.

The timestamp [ 0.000000] on the first dmesg line is worth noticing. That is the very first thing the kernel logged — time zero. The command line is genuinely the first thing it knows.

And /proc/cmdline refuses to be written, even by root. It is a record of a decision already made, not a setting. Changing it means changing GRUB and rebooting.

Now imagine this at 500 hosts. cat /proc/cmdline collected fleet-wide is one of the highest-value, lowest-cost facts you can gather. It tells you which hosts have had emergency parameters left on them by mistake — nomodeset, a raised pid_max, a disabled mitigation, systemd.unit=rescue.target — that somebody added during an incident and never removed.

Those stay across reboots and are invisible to every configuration management tool, because they live in GRUB rather than in a config file anyone is watching.

⚙️ Part B · The kernel takes over

B1 · What the kernel does first

Official docs: bootup(7) · Kernel admin guide · proc(5)

GRUB has jumped to the kernel. In rough order, the kernel now:

Decompresses itself. The z in vmlinuz is for compressed — the first thing it runs is a small unpacker.
Sets up memory management, so there is such a thing as an address space at all.
Initialises the CPUs, bringing the other cores online.
Loads built-in drivers for whatever it finds.
Mounts the initramfs as a temporary root filesystem — Section B2.
Starts PID 1 — /init from the initramfs, not yet the real init.

Step 6 is the handover, and it is the moment the machine stops being a kernel-only thing and becomes a Linux system. Everything in Modules 02, 03 and 04 begins here.

Real-world analogy — opening a shop in the morning

The kernel's start-up is the routine before a shop opens.

Unlock the door, turn on the lights and heating, switch on the tills, check the stock is where it should be. None of it serves a customer. All of it has to happen before anyone can be served, and the order is not negotiable — you cannot switch on the tills before the power.

Then the manager arrives and the shop opens. That is PID 1.

The part worth carrying forward is what the kernel is not doing. It is not starting your web server, mounting your data volumes, or bringing up the network as you know it. It is making a machine on which those things become possible, and then handing over.

Where the analogy stops working. A shop manager arrives after opening up and takes over the whole building.

The kernel does not leave. Unlike the earlier relay legs, it stays underneath everything for as long as the machine is on. PID 1 is the manager; the kernel is the building itself.

🧪 Exercise B1.1 — Read the kernel's own boot log

bash

# The very first things the kernel said, with timestamps
sudo dmesg | head -12

# When did each CPU come online?
sudo dmesg | grep -iE "smpboot|Booting Node|Brought up" | head -5

# When did it hand over to PID 1? Look for the last kernel-only message.
sudo dmesg | grep -iE "Freeing unused kernel|Run /|init as init" | head -5

# How long was the kernel alone before userspace started?
systemd-analyze 2>/dev/null || echo "(systemd-analyze not available)"

✅ Expected result — click to reveal

plain text

$ sudo dmesg | head -12
[    0.000000] Linux version 6.8.0-45-generic (buildd@lcy02) (gcc 13.2.0) #45-Ubuntu SMP ...
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.0-45-generic root=UUID=8f3a1c7e... ro quiet splash
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] NX (Execute Disable) protection: active
[    0.004512] Memory: 1987456K/2096636K available
[    0.012004] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1

$ sudo dmesg | grep -iE "smpboot|Brought up" | head -5
[    0.031882] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[    0.183440] smpboot: CPU0: Intel Xeon Processor (Skylake, IBRS)
[    0.201773] smpboot: Total of 2 processors activated (9600.00 BogoMIPS)

$ sudo dmesg | grep -iE "Freeing unused kernel|Run /" | head -5
[    1.442019] Freeing unused kernel image (initmem) memory: 4520K
[    1.451003] Run /init as init process

$ systemd-analyze
Startup finished in 1.921s (kernel) + 8.442s (userspace) = 10.363s
graphics.target reached after 8.401s in userspace.

What to read out of it — the timestamps are the story.

Every dmesg line starts with seconds since the kernel began. Read them as a timeline.

[ 0.000000] — the kernel's first three messages are its own version, the command line it was handed, and the memory map the firmware gave it. Notice it learns the memory layout from the firmware; it does not go looking.

[ 0.201773] — two processors activated. Until this point the machine was effectively single-core. This is the kernel bringing the other cores online itself.

[ 1.451003] Run /init as init process — this is the handover, and note the path. With an initramfs present the kernel does not run /sbin/init; it runs /init inside the initramfs, which is Section B2. The real /sbin/init on the real root is execed later, at switch_root, which is why the journal shows systemd announcing itself around 1.92 s. Everything before this line is kernel-only. Everything after it is Module 02 onwards.

And Freeing unused kernel image (initmem) memory: 4520K just before it is a small piece of housekeeping worth knowing: a lot of kernel code exists only to run once at boot. Once it has run, the kernel gives that memory back. Four and a half megabytes of code that will never execute again.

Finally, systemd-analyze splits the whole boot in two: 2.1 seconds of kernel, 8.4 seconds of userspace. Almost all boot time is normally userspace, which is why Section D2 is about measuring that half.

B2 · initramfs — the chicken-and-egg problem

Official docs: bootup(7) — booting in the initrd · Kernel command-line parameters · Kernel admin guide

Here is the problem the initramfs exists to solve, and it is a genuine paradox.

The kernel needs to mount your root filesystem. To do that it needs a driver for the disk controller, and a driver for the filesystem type. Those drivers are modules — files, as Module 01 showed with lsmod.

And where do those files live? On the root filesystem it cannot mount yet.

You need the thing you cannot reach in order to reach it.

There are two possible answers:

Build every driver into the kernel. It works, and it means a kernel that supports all hardware would be enormous, with almost all of it unused on any given machine.
Carry a small toolbox. Give the kernel a tiny, self-contained filesystem loaded straight into memory by the bootloader, containing exactly the drivers this machine needs plus a small program to use them.

Linux does the second, and that toolbox is the initramfs.

Real-world analogy — the toolbox that is locked inside the shed

You need a spanner. The spanner is in the shed. The shed is locked, and the key is inside the shed.

That is exactly the kernel's position: the driver it needs to open the disk is stored on the disk.

The answer is not to knock the shed down or to carry every tool you own everywhere. It is to keep a small pouch on your belt with just the few things needed to get the shed open. Once it is open you have all your tools, and the pouch goes back in your pocket.

The initramfs is that pouch. It holds the disk driver, the filesystem driver, and whatever is needed to unlock encryption or assemble a RAID array — and nothing else. It is loaded into memory by the bootloader, so it needs no disk at all.

And once the real root is mounted, the pouch is emptied and thrown away. The memory it occupied is freed, which is why you cannot find the initramfs on a running system no matter where you look.

Where the analogy stops working, and it is the failure mode. If you left the right spanner out of the pouch, you can go and fetch it.

The kernel cannot. A missing driver in the initramfs means the shed never opens, and you get a machine that stops dead with Cannot open root device — with the driver it needs sitting on a disk two centimetres away, permanently out of reach.

🧪 Exercise B2.1 — Open the toolbox and look inside

bash

# How big is it, and what format?
ls -lh /boot/initrd.img-$(uname -r)
sudo file /boot/initrd.img-$(uname -r)

# What is in it? (lsinitramfs on Debian/Ubuntu, lsinitrd on RHEL)
sudo lsinitramfs /boot/initrd.img-$(uname -r) 2>/dev/null | wc -l

# The drivers it carries - these are the ones needed to reach the real root
sudo lsinitramfs /boot/initrd.img-$(uname -r) 2>/dev/null \
  | grep -E '\.ko(\.zst)?$' | grep -E 'virtio|ahci|nvme|ext4|xfs' | head -8

# And proof it is gone now: nothing is mounted from it
findmnt -t ramfs,tmpfs / 2>/dev/null || echo "root is NOT the initramfs - it was replaced"

✅ Expected result — click to reveal

plain text

$ ls -lh /boot/initrd.img-6.8.0-45-generic
-rw-r--r-- 1 root root 62M Aug 17 09:11 /boot/initrd.img-6.8.0-45-generic

$ sudo file /boot/initrd.img-6.8.0-45-generic
/boot/initrd.img-6.8.0-45-generic: ASCII cpio archive (SVR4 with no CRC)

$ sudo lsinitramfs /boot/initrd.img-6.8.0-45-generic | wc -l
3847

$ sudo lsinitramfs ... | grep -E '\.ko(\.zst)?$' | grep -E 'virtio|ahci|nvme|ext4|xfs' | head -8
usr/lib/modules/6.8.0-45-generic/kernel/drivers/block/virtio_blk.ko.zst
usr/lib/modules/6.8.0-45-generic/kernel/drivers/net/virtio_net.ko.zst
usr/lib/modules/6.8.0-45-generic/kernel/drivers/scsi/virtio_scsi.ko.zst
usr/lib/modules/6.8.0-45-generic/kernel/drivers/ata/ahci.ko.zst
usr/lib/modules/6.8.0-45-generic/kernel/drivers/nvme/host/nvme.ko.zst
usr/lib/modules/6.8.0-45-generic/kernel/fs/ext4/ext4.ko.zst

$ findmnt -t ramfs,tmpfs /
root is NOT the initramfs - it was replaced

What to read out of it.

3,847 files in a 62 MB archive, and it is a cpio archive — a format older than Linux itself, chosen because unpacking it needs almost no code.

The driver list is the answer to the paradox, made concrete. virtio_blk is the disk driver for this VM. ext4 is the filesystem driver. Both are inside the initramfs, which is why the kernel can reach a disk whose driver is stored on that same disk.

The kernel has a copy in memory before it ever touches storage.

And the last check confirms the toolbox is gone: the root filesystem is ext4 on a real disk, not the temporary one. Section B3 is how that swap happened.

The failure this causes, and it is one of the most common serious boot failures there is.

Whether the initramfs carries drivers for hardware this machine does not have is a distribution choice, and it decides whether a cloned disk boots. Debian and Ubuntu default to MODULES=most in /etc/initramfs-tools/initramfs.conf — most filesystem and all block drivers — which is why the image is tens of megabytes and why Ubuntu disks usually survive being moved between hypervisors. RHEL-family dracut defaults to hostonly=yes and builds a small image containing only what this machine needs. Move that disk to a host with a different disk controller and the driver is not in the pouch. The kernel loads, finds nothing it can read, and stops with Cannot open root device or drops to an initramfs> prompt.

This is why cloning a VM to different virtual hardware, or restoring a physical machine's image onto another model, can produce a machine that boots to a black screen with a perfectly healthy disk. The fix is to rebuild it — update-initramfs -u -k all, or dracut -f --regenerate-all — from a rescue environment.

It is also why a disk running out of space during a kernel update is dangerous in a way that most disk-full incidents are not: a truncated initramfs produces an unbootable machine, and you will not find out until the next reboot, possibly months later.

B3 · Switching to the real root

Official docs: bootup(7) — initrd to system handover · mount(8) · findmnt(8)

The initramfs is mounted as / and a small init program inside it is running as PID 1. Its job list is short:

Load the drivers it carries.
Do anything special needed to reach the real root — unlock encryption, assemble RAID, activate LVM.
Mount the real root filesystem somewhere temporary.
Make it / instead, and delete the initramfs.
Run the real /sbin/init on the new root.

Step 4 is switch_root, and it is a genuinely clever piece of engineering: the running system replaces its own root filesystem underneath itself, without rebooting.

Step 5 is worth pausing on. The real init replaces the temporary one with exec, so it inherits PID 1. That is exactly the mechanism from Module 02 Section B3 — same process, new program, same PID.

Real-world analogy — the site office and the finished building

A construction site starts with a portacabin. It has a desk, a kettle and a phone. It is not the building — it is what you work from while you build the building.

Once the real building is finished and the doors open, everyone moves in and the portacabin is taken away. It is not kept as a spare room. Its entire purpose was to exist until the real thing did.

That is the initramfs. And the person who ran the site office does not leave when the cabin does — they walk into the new building and carry on with the same job. Same person, new premises. That is exec preserving PID 1.

Where the analogy stops working, and it is why this is impressive. Moving offices normally means shutting down for a day.

The kernel does this with the system running, without dropping a single process. / genuinely changes meaning underneath the running init, and nothing notices. It is the same trick as mounting a filesystem over a directory from Module 03, applied to the root of the tree itself.

🧪 Exercise B3.1 — Find the handover in the logs

bash

# The moment the real root was mounted
sudo dmesg | grep -iE "EXT4-fs.*mounted|XFS.*Mounting|mounted filesystem" | head -3

# PID 1 now - and what it actually is on disk
ps -o pid,comm -p 1
sudo ls -l /proc/1/exe      # reading /proc/PID/exe needs privilege

# PID 1's root is the REAL root, not the initramfs
sudo ls -l /proc/1/root

# The journal records the initramfs phase separately
journalctl -b -o short-monotonic 2>/dev/null | head -6

✅ Expected result — click to reveal

plain text

$ sudo dmesg | grep -iE "EXT4-fs.*mounted|mounted filesystem" | head -3
[    1.610224] EXT4-fs (vda2): mounted filesystem 8f3a1c7e-2b44-4d9a-9c11-6e05a2f7d130 ro with ordered data mode
[    2.884301] EXT4-fs (vda2): re-mounted 8f3a1c7e-2b44-4d9a-9c11-6e05a2f7d130 r/w

$ ps -o pid,comm -p 1
    PID COMMAND
      1 systemd

$ sudo ls -l /proc/1/exe
lrwxrwxrwx 1 root root 0 Aug 20 16:02 /proc/1/exe -> /usr/lib/systemd/systemd

$ sudo ls -l /proc/1/root
lrwxrwxrwx 1 root root 0 Aug 20 16:02 /proc/1/root -> /

$ journalctl -b -o short-monotonic | head -6
[    0.000000] oslab kernel: Linux version 6.8.0-45-generic ...
[    1.921044] oslab systemd[1]: systemd 255.4-1ubuntu8 running in system mode.
[    1.925511] oslab systemd[1]: Detected virtualization kvm.
[    1.925610] oslab systemd[1]: Detected architecture x86-64.

What to read out of it — the two EXT4-fs lines tell the whole story.

At 1.61 seconds, the root filesystem is mounted ro — read-only. That is the ro from the kernel command line in Section A3. It is deliberate: mounting read-only first means the filesystem can be checked before anything writes to it.

At 2.88 seconds, it is re-mounted r/w. Between those two timestamps, systemd started and ran the filesystem check. Same filesystem, same UUID, changed from read-only to writable underneath a running system.

/proc/1/exe points at the real systemd on the real root. The initramfs's init is long gone. And /proc/1/root -> / confirms PID 1's idea of / is the real root — the switch happened before it started.

The journal line systemd[1]: systemd 255.4 running in system mode at 1.92 seconds is systemd announcing itself, and it sits neatly between the two mounts. That ordering is the handover, visible in timestamps.

Interview-grade detail. /proc/1/root is not decoration. Module 03 mentioned that root is a live symlink to what a process considers /, and for most processes it is boring.

It stops being boring in containers. Compare /proc/1/root inside a container with the host's and they point at completely different trees — that is the container's filesystem view, made visible. Reading /proc/<pid>/root from the host is a standard way to inspect a container's files without entering it, and it is the same mechanism you are looking at here.

🎛️ Part C · systemd — PID 1

C1 · Units — the thing systemd manages

Official docs: systemd(1) · systemd.unit(5) · systemctl(1)

systemd is now PID 1. Everything it manages is a unit — a small text file describing one thing it should look after.

The unit type is the file extension, and there are more than people expect:

Type	What it describes
.service	A process to run and keep running. The one you will use most.
.target	A group of units. Replaces runlevels — Section C2.
.mount	A filesystem to mount. Generated automatically from /etc/fstab.
.socket	A port or socket to listen on, starting the service only when something connects.
.timer	A schedule. The modern replacement for cron.
.device	A piece of hardware, so other units can wait for it to appear.

Unit files live in three places, and which one wins matters:

/etc/systemd/system/ — yours. Highest priority; overrides everything below.
/run/systemd/system/ — runtime only, gone on reboot, but it outranks the packaged copy while it exists. This is how systemd-run and transient units work.
/usr/lib/systemd/system/ — shipped by packages. Never edit these; upgrades overwrite them. (/lib/systemd/system is the same directory through the merged-/usr symlink, and systemctl cat reports the /usr/lib path.)

Real-world analogy — job cards on a workshop wall

A workshop has a card for every job. Each card says what the job is, what has to be finished first, and what to do if it goes wrong.

The foreman does not remember any of this. He reads the cards.

That is systemd. It has no built-in knowledge of what nginx is or how to start a database. It reads a card and follows it.

The three directories are the same idea a workshop uses: the printed cards that came with the machinery (/lib), the hand-written cards the workshop added (/etc), and today's temporary notes (/run). When two cards describe the same job, the workshop's own card wins over the manufacturer's.

Where the analogy stops working, and it is the one that trips people. A foreman would notice a new card on the wall.

systemd does not. It reads the cards into memory and works from that copy. Edit a unit file and nothing happens until you run systemctl daemon-reload. Forgetting that reload is the single most common systemd mistake there is — the file is correct, the behaviour is old, and nothing warns you.

🧪 Exercise C1.1 — Read a real unit and find the override

bash

# How many units of each type is this machine managing?
systemctl list-units --all --no-pager --no-legend --plain | awk '{print $1}' \
  | grep -oE '\.[a-z]+$' | sort | uniq -c | sort -rn | head -6

# The full unit file for ssh, including where it came from
{ systemctl cat ssh 2>/dev/null || systemctl cat sshd 2>/dev/null; } | head -25

# Which units are enabled to start at boot?
systemctl list-unit-files --state=enabled --no-pager | head -8

# And which have actually been overridden or modified locally?
systemd-delta --type=extended,overridden | tail -4
ls -l /etc/systemd/system/*.service 2>/dev/null | head -5

✅ Expected result — click to reveal

plain text

$ systemctl list-units --all --no-legend | awk '{print $1}' | grep -oE '\.[a-z]+$' | sort | uniq -c | sort -rn | head -6
     94 .service
     71 .device
     42 .mount
     28 .target
     14 .socket
      7 .timer

$ { systemctl cat ssh 2>/dev/null || systemctl cat sshd 2>/dev/null; } | head -25
# /usr/lib/systemd/system/ssh.service
[Unit]
Description=OpenBSD Secure Shell server
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify

[Install]
WantedBy=multi-user.target

What to read out of it.

42 .mount units on a machine where nobody wrote a mount unit. systemd generated them from /etc/fstab at boot. Same for most of the 71 .device units. The unit model covers far more than services.

Now read the ssh unit, because almost every idea in this module is in it:

After=network.target — ordering. Section C3.
ExecReload=/bin/kill -HUP $MAINPID — this is Module 04. systemctl reload ssh sends SIGHUP, exactly as that module described, and this line is where the meaning is defined.
Restart=on-failure — the supervision that nohup could never give you.
KillMode=process — how much to stop when stopping. The default is control-group, which signals everything in the unit's cgroup; process signals only the main process. sshd sets it deliberately, which is why systemctl stop ssh does not disconnect your own live session. It is a considered opt-out of the guarantee Section C4 describes.
WantedBy=multi-user.target — what systemctl enable actually does: it creates a symlink putting this unit into that target's wants directory. Nothing more mysterious than that.

The first line of systemctl cat tells you the path the file came from. If a unit is overridden you see the /etc copy first, and any drop-in files after it.

Now imagine this at 500 hosts. Never edit a file under /lib/systemd/system/ — the next package upgrade replaces it and your change vanishes silently.

Use systemctl edit <unit>, which creates a drop-in at /etc/systemd/system/<unit>.service.d/override.conf containing only what you changed. It survives upgrades, systemctl cat shows it clearly, and systemd-delta lists every override on the machine — a genuinely useful audit for "why does this host behave differently".

C2 · Targets, and what happened to runlevels

Official docs: systemd.unit(5) · bootup(7) · systemctl(1)

Older systems had runlevels: numbered stages, run in order, by a script per stage.

systemd replaced them with targets, and the change is not cosmetic. A target is not a stage in a sequence. It is a state you want the machine to be in, expressed as a set of units that should be active.

That difference is why systemd can start things in parallel. It is not walking through steps 2, 3, 4 — it is working out what has to be true and starting everything that can start now.

Target	Old runlevel	What it means
poweroff.target	0	Shut down
rescue.target	1	Single user, root shell, minimal services
multi-user.target	3	Full system, no graphical login. The normal server state.
graphical.target	5	Everything in multi-user, plus a display manager
reboot.target	6	Restart
emergency.target	—	Even more minimal than rescue. Root filesystem read-only, almost nothing started.

Real-world analogy — a recipe versus a finished dish

The old way was a recipe: step 1, then step 2, then step 3. Strictly in order, and if step 2 was slow, step 3 waited even if it needed nothing from step 2.

A target is not a recipe. It is a photograph of the finished dish — this is what the table should look like. How you get there is up to the kitchen, and anything that can be done at the same time is.

That is why systemd boots faster, and it is why the ordering directives in Section C3 exist. Once you stop specifying a sequence, you have to say explicitly which few things genuinely must wait for others.

And you can hand the kitchen a different photograph: systemctl isolate rescue.target says "make the machine look like this instead", and systemd works out what to stop and what to start.

Where the analogy stops working. A photograph is fixed.

A target is a set of wants, and it can be extended. systemctl enable adds your service to multi-user.target's list, changing what "the finished dish" means on this machine. Targets are lists you can add to, not fixed images.

🧪 Exercise C2.1 — See what your machine is aiming for

bash

# What does this machine boot to?
systemctl get-default

# What does that target pull in? (recursive, through nested targets)
systemctl list-dependencies multi-user.target --no-pager | head -15

# Which target is active right now, and the old runlevel name for it
systemctl list-units --type=target --state=active --no-pager --no-legend | head -8
runlevel

# The compatibility mapping still exists as symlinks
ls -l /lib/systemd/system/runlevel*.target 2>/dev/null | head -6

✅ Expected result — click to reveal

plain text

$ systemctl get-default
graphical.target

$ systemctl list-dependencies multi-user.target | head -12
multi-user.target
● ├─cron.service
● ├─dbus.service
● ├─ssh.service
● ├─systemd-logind.service
● ├─basic.target
● │ ├─sysinit.target
● │ │ ├─systemd-journald.service
● │ │ └─systemd-udevd.service
● │ └─sockets.target
● └─getty.target

$ systemctl list-units --type=target --state=active --no-legend | head -8
basic.target       loaded active active Basic System
cryptsetup.target  loaded active active Local Encrypted Volumes
graphical.target   loaded active active Graphical Interface
local-fs.target    loaded active active Local File Systems
multi-user.target  loaded active active Multi-User System
network.target     loaded active active Network
sysinit.target     loaded active active System Initialization

$ runlevel
N 5

$ ls -l /lib/systemd/system/runlevel*.target
lrwxrwxrwx 1 root root 15 Mar 24 13:45 runlevel0.target -> poweroff.target
lrwxrwxrwx 1 root root 13 Mar 24 13:45 runlevel1.target -> rescue.target
lrwxrwxrwx 1 root root 17 Mar 24 13:45 runlevel2.target -> multi-user.target
lrwxrwxrwx 1 root root 17 Mar 24 13:45 runlevel3.target -> multi-user.target
lrwxrwxrwx 1 root root 17 Mar 24 13:45 runlevel4.target -> multi-user.target
lrwxrwxrwx 1 root root 16 Mar 24 13:45 runlevel5.target -> graphical.target

What to read out of it.

list-dependencies shows targets are nested. multi-user.target pulls in basic.target, which pulls in sysinit.target. Each layer is a broader statement about what should be true, built on the one below.

Notice several targets are active at once: basic, local-fs, multi-user, network, graphical. That is the strongest evidence targets are not stages — you cannot be at runlevel 3 and 5 simultaneously, but you can satisfy several targets at the same time, because each is just a set of conditions.

runlevel still works and prints N 5, but those symlinks show what it really is: a compatibility shim so old scripts do not break. There is no runlevel 5 on this machine; there is graphical.target with an old name pointing at it.

Now imagine this at 500 hosts. A server booting to graphical.target is running a display manager and its dependencies for nobody. systemctl set-default multi-user.target is a one-line change that removes a chunk of attack surface and boot time on every headless machine.

And know the difference between the two ways to change target: systemctl isolate <target> changes it now, for this boot only. systemctl set-default <target> changes what it boots to next time. Confusing them during an incident is how people accidentally stop every service on a running production host.

C3 · Wants versus Requires, After versus Before

Official docs: systemd.unit(5) — Wants, Requires, After, Before · systemd(1) · systemctl(1)

Because targets are not sequences, systemd needs to be told about the few genuine dependencies. It uses two completely separate axes, and mixing them up is the most common cause of confusing systemd behaviour.

Axis 1 — WHETHER (requirement)

Wants= — start that too. If it fails, carry on anyway.

Requires= — start that too. If it fails, fail as well.

BindsTo= — stronger still: if it stops later, stop too.

This says what else should run. It says nothing about order.

Axis 2 — WHEN (ordering)

After= — do not start until that one has started.

Before= — do not let that one start until this has.

This says in what order. It says nothing about whether the other unit runs at all.

The trap, and it is worth reading twice.

Requires=foo.service on its own does not wait for foo. It starts foo, and starts your service at the same instant. If your service needs foo to be ready, it will fail — intermittently, depending on timing, which makes it maddening to debug.

You almost always need both:

Requires=postgresql.service and After=postgresql.service

The first says "postgres must run". The second says "and I go second". Neither implies the other, and the documentation says so explicitly.

This is the single most common systemd mistake in real unit files, and it produces a service that works on a fast machine and fails on a slow one.

Real-world analogy — the invitation and the running order

You are organising a meeting. Two decisions, and they are genuinely separate.

Who is invited? That is the requirement axis. "Invite the finance lead" — and there are two strengths. Wants is "invite them; if they cannot come, we go ahead." Requires is "if they cannot come, cancel the meeting."

Who speaks when? That is the ordering axis. "Finance speaks before marketing."

Now notice: inviting someone says nothing about when they speak. You can require someone's attendance and still, through pure carelessness, have marketing present the finance numbers before finance has arrived in the room. Everyone is present. The order was never specified.

That is Requires= without After=, and it is exactly what happens: postgres is definitely starting, and your app is talking to it before it is listening.

Where the analogy stops working, and it is a real behaviour. In a meeting you would notice someone was missing and wait.

systemd does not wait unless you tell it to. After= only means "after it has started", which for a simple service means "after the process was launched" — not "after it is ready to answer". That is what Type=notify is for: the service tells systemd when it is genuinely ready, and only then does After= release the next unit.

🧪 Exercise C3.1 — Watch Requires fail, and Wants survive (one is meant to fail)

bash

# A dependency that always fails
sudo tee /etc/systemd/system/broken-dep.service >/dev/null <<'EOF'
[Unit]
Description=A dependency that always fails
[Service]
Type=oneshot
ExecStart=/bin/false
EOF

# One service that REQUIRES it, one that only WANTS it
sudo tee /etc/systemd/system/needs-it.service >/dev/null <<'EOF'
[Unit]
Description=Requires the broken dependency
Requires=broken-dep.service
After=broken-dep.service
[Service]
Type=oneshot
ExecStart=/bin/echo "needs-it started"
EOF

sudo tee /etc/systemd/system/likes-it.service >/dev/null <<'EOF'
[Unit]
Description=Only wants the broken dependency
Wants=broken-dep.service
After=broken-dep.service
[Service]
Type=oneshot
ExecStart=/bin/echo "likes-it started"
EOF

sudo systemctl daemon-reload

echo "=== Requires (predict: does it start?) ==="
sudo systemctl start needs-it.service; echo "exit code: $?"
systemctl is-active needs-it.service

echo "=== Wants (predict: does it start?) ==="
sudo systemctl start likes-it.service; echo "exit code: $?"
journalctl -u likes-it.service --no-pager | grep "likes-it started"

# Clean up
sudo systemctl reset-failed broken-dep.service needs-it.service 2>/dev/null
sudo rm -f /etc/systemd/system/{broken-dep,needs-it,likes-it}.service
sudo systemctl daemon-reload

✅ Expected result — click to reveal

plain text

=== Requires (predict: does it start?) ===
A dependency job for needs-it.service failed. See 'journalctl -xe' for details.
exit code: 1
inactive

=== Wants (predict: does it start?) ===
exit code: 0
Aug 20 16:41:02 oslab echo[8812]: likes-it started

What to read out of it.

Identical situation, one word different, opposite outcomes.

Requires= — the dependency failed, so the service never ran at all. It is inactive, and systemctl start returned 1.

Wants= — the dependency failed in exactly the same way, and the service started anyway and printed its message.

That one word is the whole difference between "this is essential" and "this would be nice". Choosing wrongly gives you either a service that refuses to start because something optional is broken, or a service that starts happily without something it genuinely needs.

Notice both units also had After=. Without it, neither would have waited for the dependency to finish failing, and the Requires= case would have become a race.

Interview-grade detail. The default for a hand-written service is usually Wants=, and that is deliberate: a hard Requires= propagates failure, so a single broken optional unit can cascade into a machine that boots into a much emptier state than you expected.

Use Requires= only when the service genuinely cannot function without the other, and always pair it with After=. If you need "start it if it is there, but do not fail if it is not", Wants= is the answer.

C4 · cgroups — how a service is actually tracked

Official docs: cgroups(7) · systemd(1) · Control Group v2 — kernel docs

Modules 02 and 04 both mentioned cgroups and deliberately left them unexplained. Here is the explanation, because systemd cannot be understood without it.

The problem: how does systemd know which processes belong to a service?

Not by PID — the service forks. Not by process group — Module 04 showed a process can leave its group with setsid. Not by parent — Module 02 showed orphans get re-parented away.

Every relationship a process could use to identify itself, a process can escape.

So the kernel provides one it cannot escape: a control group. A cgroup is a labelled box. Put a process in it and every child it creates is in it too, automatically. Unlike a session or a process group, a process cannot slip out with setsid or a double fork — leaving a cgroup means writing your own PID into another cgroup's cgroup.procs, which needs write access to that file and to the nearest common ancestor. Root can do it; the service you are supervising cannot.

That gives systemd three things at once:

Accounting — exactly how much CPU, memory and I/O this service used, including everything it started.
Limits — a ceiling on those, enforced by the kernel.
Reliable stopping — systemctl stop signals everything in the box, so nothing survives.

Real-world analogy — the department everything gets charged to

A company needs to know what a project costs. Tracking "who reports to the project lead" fails immediately — people move, contractors are hired, someone transfers away.

So the company uses a cost code. Every purchase carries it, every person hired onto the project inherits it, and you cannot spend money on the project without it being charged there. Not because people are honest, but because the system will not process a purchase without a code.

That is a cgroup. Every process started inside inherits the label, and the kernel enforces it rather than trusting the process.

Once everything carries a code, three things become easy and were previously impossible: you can total the spend exactly, you can set a budget the system refuses to exceed, and when the project is cancelled you can find every single thing attached to it.

Where the analogy stops working. A cost code is bookkeeping — it records what happened.

A cgroup enforces. Hit a memory limit and the allocation fails or the process is killed, by the kernel, immediately. It is a budget the accounting system physically will not let you exceed, which is why Module 09's out-of-memory killer and Module 12's containers are both built on this.

🧪 Exercise C4.1 — Find the box a service lives in

bash

# The cgroup tree, as systemd sees it
systemd-cgls --no-pager | head -20

# Every process of one service, tracked by cgroup rather than by PID
systemctl status ssh --no-pager 2>/dev/null | tail -6

# A process's cgroup, straight from /proc - Module 01 applies
cat /proc/1/cgroup
cat /proc/self/cgroup

# Live resource accounting for every service, no instrumentation needed
systemd-cgtop -n 2 -b 2>/dev/null | head -8

✅ Expected result — click to reveal

plain text

$ systemd-cgls --no-pager | head -20
CGroup /:
-.slice
├─user.slice
│ └─user-1000.slice
│   ├─[email protected]
│   │ └─init.scope
│   │   ├─3890 /usr/lib/systemd/systemd --user
│   │   └─3891 (sd-pam)
│   └─session-3.scope
│     ├─3898 sshd: zaeem [priv]
│     └─3901 -bash
├─init.scope
│ └─1 /usr/lib/systemd/systemd
└─system.slice
  ├─ssh.service
  │ └─812 sshd: /usr/sbin/sshd -D [listener]
  └─cron.service
    └─756 /usr/sbin/cron -f

$ systemctl status ssh --no-pager -n 0 | tail -6
   Main PID: 812 (sshd)
      Tasks: 1 (limit: 2273)
     Memory: 5.2M (peak: 6.1M)
        CPU: 84ms
     CGroup: /system.slice/ssh.service
             └─812 "sshd: /usr/sbin/sshd -D [listener]"

$ cat /proc/1/cgroup
0::/init.scope

$ cat /proc/self/cgroup
0::/user.slice/user-1000.slice/session-3.scope

What to read out of it.

The tree is the whole model in one picture. Two big branches: system.slice for services, and user.slice for logged-in people. Your shell is in session-3.scope under your own user slice.

Look at systemctl status. Tasks: 1, Memory: 5.2M, CPU: 84ms — nobody instrumented sshd to produce those. The kernel counted them, because everything sshd does happens inside its box. That is accounting you get for free, and it is why systemctl status can tell you things ps cannot.

/proc/self/cgroup shows 0:: — the 0 means cgroup v2, the unified hierarchy. On an older machine you would see a dozen numbered lines, one per controller, which is v1.

And notice your interactive shell is in session-3.scope. That is why systemctl stop on a service never touches your shell, and also why logging out can cleanly kill everything you started — your whole session is one box. This is the mechanism behind KillUserProcesses in logind.conf, which decides whether your nohuped job from Module 04 actually survives logout on a given distribution.

Interview-grade detail. This closes the loop on Module 04's job-control section.

nohup, disown and setsid all work by escaping a relationship — a signal disposition, a shell's job list, a session. A cgroup is not a relationship, it is a container the kernel enforces, and none of those three escape it.

So on a system with KillUserProcesses=yes, nohup genuinely does not save your job: systemd removes the whole session scope at logout, regardless of what signals anything is ignoring. That is precisely why systemd-run --user is the correct answer — it puts the job in its own scope rather than your session's.

🎯 Interview questions — systemd

Q. What is the difference between Wants= and Requires=? And between Requires= and After=?

Wants= and Requires= are both requirement dependencies — they say another unit should also be started. The difference is what happens on failure: with Wants=, this unit starts anyway; with Requires=, this unit fails too.

After= is a completely different axis. It is ordering — do not start until the other unit has started. It says nothing about whether that unit runs at all.

The key point, and it is the reason the question is asked: requirement does not imply ordering. Requires=postgresql.service on its own starts postgres and starts your service at the same time. If your service needs the database to be ready, it fails — intermittently, depending on machine speed. You need both directives.

The details that separate candidates:

Prefer Wants= by default. Requires= propagates failure, so one broken optional unit can cascade into a much emptier boot than intended.
Know that After= only means "after it started", not "after it is ready". For Type=simple that means "after the process was launched", which is often not enough. Type=notify lets the service tell systemd when it is genuinely ready, which is what makes After= meaningful for a database.
Mention BindsTo=, which is Requires= plus "if it stops later, stop me too" — the right choice for a service tied to a device or a mount.

Q. Why does systemd use cgroups to track services instead of PIDs?

Because every other way of identifying "the processes belonging to this service" can be escaped by the processes themselves.

Tracking the main PID misses forked children. Tracking the process group fails because a process can call setsid and leave. Tracking by parent fails because when the parent dies the children are re-parented away.

A cgroup is different: it is a kernel-enforced label. Every process started inside inherits it, and no ordinary process can shake it off — escaping means a privileged write of your own PID into another cgroup's cgroup.procs, not a trick with fork or setsid. That gives systemd exact accounting, enforceable limits, and — most importantly — a reliable way to stop everything a service started.

The details that separate candidates:

Give the concrete payoff: systemctl status shows Tasks, Memory and CPU for a service with no instrumentation at all, because the kernel counted everything in the box.
Connect it to the daemonising problem. The old double-fork trick existed precisely to escape supervision. With cgroups it does not work, which is why Type=forking is discouraged and Type=simple or Type=notify is preferred.
Say that this is what containers are built from. A container is a set of processes in a cgroup with namespaces applied. Same mechanism, different packaging.
Know v1 versus v2: /proc/PID/cgroup showing a single 0:: line means the unified v2 hierarchy; multiple numbered lines mean v1. It matters because memory limits and accounting behave differently between them.

Q. What is a target, and how is it different from a runlevel?

A runlevel was a numbered stage, entered in sequence by running a set of scripts in a fixed order.

A target is a state the machine should be in, expressed as a group of units that should be active. It is a description of the destination, not a sequence of steps.

That difference is what allows parallel startup: systemd works out what must be true and starts everything whose dependencies are already satisfied, rather than walking through numbered stages.

The familiar mapping: multi-user.target was runlevel 3, graphical.target was 5, rescue.target was 1. The old names still exist as symlinks for compatibility.

The details that separate candidates:

Point out that several targets are active at once. You cannot be at runlevel 3 and 5 simultaneously, but basic, multi-user and graphical can all be active together, because each is a set of conditions rather than a position in a sequence.
Know the two ways to change it, and the difference: systemctl isolate <target> changes it now, for this boot; systemctl set-default <target> changes what it boots into next time. Confusing them on a production host stops every service that is not part of the new target.
Give the practical use: setting a headless server to multi-user.target instead of graphical.target removes a display manager and its dependencies from every boot.

🔧 Part D · Reading and fixing a boot

D1 · journalctl and the boot log

Official docs: journalctl(1) · systemd(1) · bootup(7)

systemd collects logs from everything — the kernel, every service's standard output, and anything sent to syslog — into one indexed store, the journal. journalctl reads it.

The flags that matter:

Flag	What it does
-b	This boot only. -b -1 is the previous boot — the one that crashed.
-u UNIT	One service.
-p err	Priority err and worse. Cuts an enormous amount of noise.
-f	Follow, like tail -f.
-k	Kernel messages only — the same content as dmesg.
--since	Accepts plain English: --since "10 min ago", --since yesterday.
-o short-monotonic	Seconds since boot instead of wall-clock. Essential for boot analysis.

Check whether your journal even survives a reboot. Journald's default is Storage=auto: persistent if /var/log/journal exists, and otherwise in memory under /run — tmpfs, from Module 03 — wiped on every reboot. Debian and Ubuntu create that directory in the systemd package's postinst, so a stock Ubuntu 24.04 is already persistent; minimal images, containers and some RPM-family installs are not.

When it is volatile, journalctl -b -1 tells you No journal boot entry found from the specified boot offset (-1) — and the logs from the crash you are investigating are gone.

Check, do not assume, and fix it before you need it: sudo mkdir -p /var/log/journal && sudo systemd-tmpfiles --create --prefix /var/log/journal. One command, and it is the difference between diagnosing a crash and guessing about it.

Real-world analogy — one incident book instead of thirty notebooks

Traditional logging is every department keeping its own notebook in its own drawer, in its own format. To reconstruct what happened at 3:14am you collect thirty notebooks and try to line up the timestamps.

The journal is one indexed incident book. Everything goes in, tagged with who wrote it, how urgent it was, and exactly when. You can then ask for one department's entries, or everything above a severity, or everything between two times — without knowing where anything was filed.

The trade is real and worth stating: the book is binary, so you cannot read it with grep and cat. You need journalctl. In exchange you get indexing that a pile of text files cannot give you.

Where the analogy stops working, and it is the trap above. A paper book stays on the shelf.

The journal is kept in memory unless you have created /var/log/journal. Reboot and it is blank. People discover this at the worst possible moment — investigating the crash whose logs no longer exist.

🧪 Exercise D1.1 — Read this boot, and check you could read the last one

bash

# Is the journal persistent, or does it vanish on reboot?
[ -d /var/log/journal ] && echo "PERSISTENT" || echo "VOLATILE - lost on reboot"
journalctl --list-boots --no-pager 2>/dev/null | tail -4

# If `dmesg` no longer starts at [0.000000], the kernel ring buffer has
# wrapped - use the journal's copy instead: journalctl -b -k --no-pager | head

# Everything that went wrong this boot
journalctl -b -p err --no-pager | tail -10

# The boot, with seconds-since-start instead of clock time
journalctl -b -o short-monotonic --no-pager | head -8

# Anything that failed to start
systemctl --failed --no-pager

✅ Expected result — click to reveal

plain text

$ [ -d /var/log/journal ] && echo "PERSISTENT" || echo "VOLATILE - lost on reboot"
PERSISTENT

$ journalctl --list-boots --no-pager | tail -4
IDX BOOT ID                          FIRST ENTRY                 LAST ENTRY
 -2 a1c4e77b93f24e0a8d2b6f1e5c930b48 Mon 2026-08-18 09:12:44 +08 Mon 2026-08-18 18:03:51 +08
 -1 7d2f0b96e41a4c3fa85e9c02d6b17f39 Tue 2026-08-19 08:41:02 +08 Tue 2026-08-19 19:22:10 +08
  0 3f9c1a2b8d4447e2b1c5a9f70e2d8341 Wed 2026-08-20 15:58:12 +08 Wed 2026-08-20 16:44:03 +08

$ journalctl -b -p err --no-pager | tail -6
Aug 20 15:58:14 oslab kernel: ACPI Error: No handler for Region [SYSI]
Aug 20 15:58:19 oslab systemd[1]: Failed to start snapd.seeded.service.

$ journalctl -b -o short-monotonic --no-pager | head -8
[    0.000000] oslab kernel: Linux version 6.8.0-45-generic ...
[    1.451003] oslab kernel: Run /init as init process
[    1.610224] oslab kernel: EXT4-fs (vda2): mounted filesystem ro with ordered data mode
[    1.921044] oslab systemd[1]: systemd 255.4-1ubuntu8 running in system mode.
[    2.104881] oslab systemd[1]: Queued start job for default target Graphical Interface.
[    2.884301] oslab kernel: EXT4-fs (vda2): re-mounted r/w

$ systemctl --failed --no-pager
  UNIT                 LOAD   ACTIVE SUB    DESCRIPTION
● snapd.seeded.service loaded failed failed Wait until snapd is fully seeded

What to read out of it.

VOLATILE and a --list-boots with only boot 0 in it are the same fact stated twice: there is no record of any previous boot. If this machine had crashed an hour ago, the evidence would be gone. Fix that now rather than later.

-p err reduced a boot log of several thousand lines to two that matter. That is the flag to reach for first, always.

The monotonic view lines up with everything from Part B: kernel at 0.000000, systemd announcing itself at 1.92, root remounted read-write at 2.88. Same timeline, now including userspace.

systemctl --failed is the single most useful command in this module. One line, and you know exactly what did not come up.

Now imagine this at 500 hosts. systemctl --failed is the cheapest fleet-wide health check that exists, and it catches things nothing else does: a service that failed at boot three weeks ago on eleven hosts, where nobody noticed because the machine is otherwise fine and nothing alerted.

Pair it with journalctl -b -p err for the detail, and make the journal persistent everywhere as a matter of policy.

D2 · systemd-analyze — where the time went

Official docs: systemd-analyze(1) · systemd.unit(5) · bootup(7)

Boot time is one of the few performance questions with a purpose-built tool that gives a straight answer.

systemd-analyze — the headline split: kernel time versus userspace time.
systemd-analyze blame — every unit, sorted by how long it took.
systemd-analyze critical-chain — the units that were actually holding everything else up.

The difference between the last two is the whole point, and it is easy to miss.

blame is misleading on its own, and the name does not help.

blame lists slow units. But a unit taking 8 seconds in parallel with everything else delayed nothing. Meanwhile a unit taking 2 seconds that thirty other units were waiting on delayed the entire boot by 2 seconds.

blame answers "what was slow?". critical-chain answers "what made the boot slow?" Those are different questions, and only the second one tells you what to fix.

This is the same distinction as Module 01's strace reading: the biggest number is not automatically the problem.

Real-world analogy — the longest job versus the job everyone waited for

A house is being renovated and it finished a week late. You want to know why.

The longest job was the decorating: nine days. But the decorators worked while the electricians and plumbers were busy elsewhere. Their nine days delayed nothing.

The job that actually cost you the week was a two-day delay getting the building inspector out — because until he signed off, the plasterers could not start, and until they finished, nobody could decorate.

blame gives you the nine-day decorating job at the top of the list. critical-chain gives you the inspector.

Where the analogy stops working, and it is a real limitation. A site manager can see the whole schedule and knows who was blocked.

critical-chain only shows the chain leading to one target. Something can be slow, off the chain, and still hurt you — a unit hammering the disk while everything else is trying to start does not appear anywhere on the critical chain, but it slows down everything on it. That is when you go back to blame and read it with judgement.

🧪 Exercise D2.1 — Find what actually delayed your boot

bash

# The headline split
systemd-analyze

# What was slow? (not necessarily what mattered)
systemd-analyze blame --no-pager | head -8

# What was actually holding things up?
systemd-analyze critical-chain --no-pager | head -12

# Compare: is the slowest unit even ON the critical chain?

✅ Expected result — click to reveal

plain text

$ systemd-analyze
Startup finished in 2.104s (kernel) + 8.442s (userspace) = 10.546s
graphical.target reached after 8.401s in userspace.

$ systemd-analyze blame | head -8
6.204s snapd.seeded.service
3.881s cloud-init.service
2.011s systemd-networkd-wait-online.service
1.402s snapd.service
 884ms dev-vda2.device
 421ms systemd-udev-trigger.service
 233ms cloud-config.service

$ systemd-analyze critical-chain | head -12
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

graphical.target @8.401s
└─multi-user.target @8.398s
  └─cloud-final.service @8.020s +376ms
    └─cloud-config.service @7.784s +233ms
      └─systemd-networkd-wait-online.service @5.771s +2.011s
        └─systemd-networkd.service @5.640s +128ms
          └─dbus.service @5.601s

What to read out of it — compare the two lists and the answer is obvious.

blame says the slowest unit is snapd.seeded.service at 6.2 seconds. That looks like the problem.

Now look at critical-chain. snapd.seeded.service is not on it at all. It took six seconds while everything else was getting on with its own work, and delayed nothing. Fixing it would save you zero.

The chain shows what really happened. systemd-networkd-wait-online.service took 2.011 seconds, and everything above it in the chain was stuck behind it. Read the @ times: nothing after it could start until 5.7 seconds in.

wait-online is doing exactly what its name says — waiting for the network to be up. On a machine that does not need the network before starting its services, disabling it is a genuine two-second saving. On a machine that does, it is load-bearing and must stay.

That is the reasoning blame alone would never have given you.

Now imagine this at 500 hosts. Boot time matters more than it used to: it is your recovery time after a reboot, and in an autoscaling group it is how long a new instance takes to become useful.

The two usual offenders are both on display here. systemd-networkd-wait-online blocking boot on a machine that does not need it, and cloud-init doing work that could be baked into the image instead. Both are found the same way — critical-chain first, blame only for context.

D3 · When boot fails

Official docs: bootup(7) · Kernel command-line parameters · systemctl(1)

Almost every boot failure happens at one of the handovers from Section A1. Knowing which one narrows the problem enormously, and the symptom tells you which.

What you see	Which handover	Usual cause and fix
No output at all, or a firmware screen	Firmware → bootloader	Boot entry lost from NVRAM, or wrong boot order. efibootmgr from a live disk.
grub> or grub rescue> prompt	Bootloader	GRUB cannot find its config or its disk. Reinstall with grub-install.
Cannot open root device / initramfs> prompt	initramfs → real root	Missing driver in the initramfs, or a wrong root= UUID. Rebuild it: update-initramfs -u -k all.
Hangs after some services start	systemd	A unit is blocking. Very often a bad /etc/fstab entry waiting for a device that is not there.
Reaches emergency.target	systemd	A required filesystem failed to mount. Read journalctl -b -p err.
Boots, but a service is missing	Not a boot failure	systemctl --failed, then journalctl -u <unit> -b.

The /etc/fstab trap, because it is the most common self-inflicted one.

Add an entry for a disk that is not always present and the machine will hang at boot waiting for it, sometimes for 90 seconds and sometimes forever, depending on the options.

Two protections, and use both:

nofail in the mount options — boot carries on if the device is absent.
x-systemd.device-timeout=10s — give up after ten seconds instead of the default 90.

And always test a new fstab entry with sudo mount -a before rebooting. It costs one command and it is the difference between finding the mistake now and finding it during a reboot at 3am.

Real-world analogy — where the relay baton was dropped

Back to Part A's relay race. The team lost, and you need to know where.

You do not analyse all four runners equally. You ask which handover failed, because the answer tells you which runner to talk to and makes the other three irrelevant.

The symptom on screen is exactly that information:

Nothing at all → the first runner never got going. Firmware.
A grub> prompt → the second runner has the baton and cannot find the third. GRUB.
An initramfs> prompt → the third runner is on the track and cannot reach the finish. The real root is unreachable.
Services half-started → the fourth runner is running and struggling. systemd.

Each of those is a different tool, a different recovery, and a different set of things you can safely ignore.

Where the analogy stops working, and it is a real hazard. In a race you can rewind the video.

Here, if your journal is volatile, there is no video. The machine that failed to boot kept no record of failing. That is why Section D1 opened with making the journal persistent — it is preparation you can only do beforehand.

🧪 Exercise D3.1 — Cause a boot-blocking failure safely, and diagnose it

bash

# Simulate the classic fstab mistake WITHOUT touching real fstab.
# systemd generates .mount units from fstab, so we make one directly.
# NOTE: a .mount unit's FILENAME must be the escaped mount point, or systemd
# refuses to load it at all: "Where= setting doesn't match unit name."
# /mnt/nothere therefore has to be mnt-nothere.mount.
# WARNING: the start below deliberately HANGS FOR 90 SECONDS. That is the lesson.
sudo tee /etc/systemd/system/mnt-nothere.mount >/dev/null <<'EOF'
[Unit]
Description=A mount that can never succeed
[Mount]
What=/dev/disk/by-uuid/00000000-0000-0000-0000-000000000000
Where=/mnt/nothere
Type=ext4
EOF

sudo mkdir -p /mnt/nothere
sudo systemctl daemon-reload

echo "=== try to start it (predict: what happens, and how long?) ==="
time sudo systemctl start mnt-nothere.mount; echo "exit code: $?"

echo "=== how it looks afterwards ==="
systemctl --failed --no-pager
systemctl status mnt-nothere.mount --no-pager -n 0 2>/dev/null | head -6
journalctl -u mnt-nothere.mount -n 3 --no-pager

# Clean up completely
sudo systemctl reset-failed mnt-nothere.mount 2>/dev/null
sudo rm -f /etc/systemd/system/mnt-nothere.mount
sudo rmdir /mnt/nothere
sudo systemctl daemon-reload

✅ Expected result — click to reveal

plain text

=== try to start it (predict: what happens, and how long?) ===
   [ ...nothing happens for 90 seconds... ]
A dependency job for mnt-nothere.mount failed. See 'journalctl -xe' for details.

real	1m30.041s
exit code: 1

=== how it looks afterwards ===
  UNIT LOAD ACTIVE SUB DESCRIPTION
0 loaded units listed.

○ mnt-nothere.mount - A mount that can never succeed
     Loaded: loaded (/etc/systemd/system/mnt-nothere.mount; static)
     Active: inactive (dead)
      Where: /mnt/nothere
       What: /dev/disk/by-uuid/00000000-0000-0000-0000-000000000000

systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-00000000-....device
systemd[1]: Dependency failed for mnt-nothere.mount - A mount that can never succeed.
systemd[1]: mnt-nothere.mount: Job mnt-nothere.mount/start failed with result 'dependency'.

What to read out of it.

This is what a failed unit looks like from all three angles, and you should know all three:

systemctl --failed — the one-line answer to "what is broken".
systemctl status — the state and configuration, including the file it came from.
journalctl -u — the actual reason, in this case Timed out waiting for device. That is the line that tells you what to fix.

You just watched the ninety-second hang, and that is the whole exercise. mount was never run. Because What= names a device path, systemd created an implicit .device unit for that UUID and added Requires=/After= on it — so the job simply sat there until the default 90-second device timeout expired. Note the consequences for your triage: the unit ends inactive (dead), not failed, so systemctl --failed shows nothing at all, and the result is dependency rather than an exit code.

That is exactly what an fstab entry for an absent disk does at boot — pulled in by local-fs.target, ninety seconds of silence, on a machine you are trying to bring back during an incident.

Adding nofail and x-systemd.device-timeout=10s to the options is what turns a 90-second boot hang into a logged warning and a machine that comes up.

Interview-grade detail. If asked "a server hangs at boot after an fstab change, what do you do?", the strong answer has three parts:

Recover it now — reboot, edit the kernel command line in GRUB to add systemd.unit=emergency.target, then mount -o remount,rw / (emergency mode leaves the root read-only), remove the bad line from /etc/fstab, and reboot. No rescue media needed.
Explain why it hung — systemd waits for the device unit backing that mount, and the default timeout is 90 seconds per device.
Prevent it — nofail, a short x-systemd.device-timeout, and mount -a before rebooting. Knowing that GRUB is where you get out of it, rather than reaching for rescue media, is what marks out someone who has actually done it.

🏁 Part E · Practice, capstone, reference and review

E1 · Production practice

Situation	What you run	What it tells you
Anything wrong after a boot?	systemctl --failed	The cheapest health check there is. Run it fleet-wide
Why did a service not start?	journalctl -u UNIT -b	The actual error, not the summary
Machine hangs at boot, no output	Edit the GRUB entry, remove quiet	The messages were always there, just hidden
Need to get into a broken machine	Add systemd.unit=emergency.target in GRUB	Minimal boot, root shell, no rescue media needed
Forgot the root password	Add init=/bin/bash in GRUB	Root shell with no authentication. Also why physical access is root
Boot is slow	systemd-analyze critical-chain first, blame second	What held things up, not merely what was slow
Cannot open root device	update-initramfs -u -k all from rescue	Missing driver in the initramfs, usually after cloning to new hardware
Edited a unit and nothing changed	systemctl daemon-reload	systemd works from its in-memory copy. The most common systemd mistake
Need to change a packaged unit	systemctl edit UNIT	Creates a drop-in that survives upgrades. Never edit /lib/systemd/system
Which hosts differ from stock?	systemd-delta	Every unit overridden or masked on this machine
Crash logs missing after reboot	mkdir -p /var/log/journal	The journal was in memory. Do this before you need it
Emergency kernel parameters left behind	cat /proc/cmdline fleet-wide	Invisible to config management — it lives in GRUB, not a config file
New fstab entry	mount -a before rebooting, plus nofail	Turns a 90-second boot hang into a logged warning
What is this service actually using?	systemctl status UNIT, systemd-cgtop	Free per-service CPU and memory accounting, via cgroups

E2 · Capstone exercise

🧪 CAPSTONE — Reconstruct your own boot, end to end

No new tools. Produce a written timeline of this machine's last boot, naming each handover and the evidence for it.

bash

# 1. Which firmware, and what was it told to boot?
[ -d /sys/firmware/efi ] && echo "UEFI" || echo "BIOS"
cat /proc/cmdline

# 2. When did the kernel start, and when did it hand over?
sudo dmesg | head -3
sudo dmesg | grep -iE "Freeing unused kernel|Run /" | tail -2

# 3. Prove the initramfs was used and then discarded
ls -lh /boot/initrd.img-$(uname -r) 2>/dev/null || ls -lh /boot/initramfs-$(uname -r).img
sudo dmesg | grep -iE "mounted filesystem|re-mounted" | head -3
findmnt / -o SOURCE,FSTYPE

# 4. When did PID 1 take over, and what is it?
ls -l /proc/1/exe
journalctl -b -o short-monotonic --no-pager | grep -m1 "systemd.*running in system mode"

# 5. What did it aim for, and how long did it take?
systemctl get-default
systemd-analyze
systemd-analyze critical-chain --no-pager | head -8

# 6. Did anything fail?
systemctl --failed --no-pager
journalctl -b -p err --no-pager | tail -5

Write out answers to these before opening the toggle:

a) Give the four handovers in order — firmware→bootloader, bootloader→kernel, kernel→initramfs init, initramfs init→real PID 1 — with a timestamp for each from your own output. (sudo dmesg | grep -iE "Run /" dates the third.)
b) Your initramfs is tens of megabytes. What is in it, and why can it not simply be left out?
c) /proc/cmdline contains a UUID rather than a device name. Why does that matter?
d) Is your slowest unit on the critical chain? What follows from the answer either way?
e) If this machine failed to boot tomorrow, could you read today's logs? How do you know?

✅ What a good answer looks like — click to reveal

a) The four handovers. Firmware → bootloader (no timestamp available; the kernel's clock has not started). Bootloader → kernel at [0.000000], the first dmesg line. Kernel → PID 1 at the Run /sbin/init as init process line, typically 1–2 seconds. PID 1 → target reached, from systemd-analyze, typically several seconds later. A complete answer notes that the kernel does not leave at its handover, unlike the first two stages.

b) The initramfs. It contains the drivers needed to reach the real root — the disk controller driver and the filesystem driver — plus tools for encryption, LVM and RAID. It cannot be left out because those drivers live on the filesystem the kernel is trying to mount, so without it there is nothing to open the disk with. The alternative is compiling every driver into the kernel, which makes a general-purpose kernel enormous. And it is discarded once the real root is mounted, which is why you cannot find it on a running system.

c) The UUID. Device names are assigned in hardware-discovery order and are not stable — adding a disk or changing a controller can renumber them. A UUID is stored inside the filesystem and travels with it. Referring to /dev/sda2 in root= produces a machine that boots until the day the enumeration order changes.

d) Slowest versus critical. If the slowest unit is not on the chain, it ran in parallel and delayed nothing — fixing it saves zero boot time. If it is on the chain, it is genuinely blocking and worth attacking. blame answers "what was slow"; critical-chain answers "what made boot slow". They are different questions and only the second is actionable.

e) The logs. Journald's default is Storage=auto: persistent if /var/log/journal exists, otherwise in /run, which is tmpfs — memory — and wiped on every reboot. Debian and Ubuntu create that directory for you at install time, so a stock Ubuntu box is normally already persistent; minimal images and containers are not. Do not assume either way — verify with journalctl --list-boots. More than one boot listed means you are safe; only boot 0 means tomorrow's investigation would have nothing to read, and the fix (mkdir -p /var/log/journal, then systemd-tmpfiles --create --prefix /var/log/journal) only helps if done in advance.

Why this is the capstone. Every boot problem you will ever debug is "which handover failed", and answering it needs exactly the evidence you just collected. Doing this once on a healthy machine means that when you see initramfs> on a broken one, you already know what stage that is, what it was trying to do, and which two commands fix it.

It is also the Foundation tier closing on itself: the kernel from Module 01, PID 1 from Module 02, mounts from Module 03, and signals from Module 04 all appear in one timeline.

E3 · Official documentation reference

Topic	Official page	Offline equivalent
The whole boot sequence	bootup(7)	man 7 bootup
Kernel command line	Kernel parameters	man 7 bootparam
systemd as PID 1	systemd(1)	man 1 systemd
Unit files, dependencies	systemd.unit(5)	man 5 systemd.unit
Controlling units	systemctl(1)	man 1 systemctl
Reading logs	journalctl(1)	man 1 journalctl
Boot performance	systemd-analyze(1)	man 1 systemd-analyze
Control groups	cgroups(7) · cgroup v2	man 7 cgroups
Mounting	mount(8) · findmnt(8)	man 8 mount
Kernel administration	Admin guide	sysctl -a

Where to start. Read man 7 bootup — it is short, it covers the whole sequence in order, and it includes ASCII diagrams of the target dependencies that make Section C2 much clearer.

Then man 5 systemd.unit, specifically the sections on Wants=, Requires=, After= and Before=. It states the requirement-versus-ordering distinction explicitly, and it is the paragraph most people writing unit files have never read.

E4 · Self-assessment

Name the four stages of boot in order. Which one does not stop after handing over, and why does that matter?
What is the difference between BIOS and UEFI in how they find something to boot? Why is there a FAT partition on a modern Linux machine?
Why does an initramfs exist at all? Describe the problem it solves in one sentence.
You clone a VM to different virtual hardware and it stops at Cannot open root device. What happened, and what is the fix?
Why does root= use a UUID rather than /dev/sda2?
What does init=/bin/bash do, and what does its existence tell you about physical security?
Explain the difference between Wants= and Requires=. Why is Requires= on its own usually not enough?
Why does systemd track services with cgroups instead of PIDs? Name one thing that gives you for free.
systemd-analyze blame says a unit took 6 seconds. Under what circumstances does fixing it save you nothing?
A machine hangs at boot after an /etc/fstab change. How do you get in, and what two options would have prevented it?
You edit a unit file and nothing changes. Why?
After a crash, journalctl -b -1 returns nothing. Why, and what should have been done in advance?

E5 · Sources

Technical content is sourced from the official documentation listed in E3.

Foundation tier complete. You now have the kernel boundary, processes, files, signals and boot. Everything in the Intermediate tier — threads, scheduling, memory, swap and I/O — is built directly on these five.

Next: Module 06 — Threads, Races & Deadlock.

Spotted a mistake or want something added? Send me a note.

🚀Module 05 — Boot & Init: Firmware to systemd

🔌 Part A · Before the kernel runs

A1 · Firmware — the first code that runs

A2 · The bootloader

A3 · The kernel command line

⚙️ Part B · The kernel takes over

B1 · What the kernel does first

B2 · initramfs — the chicken-and-egg problem

B3 · Switching to the real root

🎛️ Part C · systemd — PID 1

C1 · Units — the thing systemd manages

C2 · Targets, and what happened to runlevels

C3 · Wants versus Requires, After versus Before

C4 · cgroups — how a service is actually tracked

🎯 Interview questions — systemd

🔧 Part D · Reading and fixing a boot

D1 · journalctl and the boot log

D2 · systemd-analyze — where the time went

D3 · When boot fails

🏁 Part E · Practice, capstone, reference and review

E1 · Production practice

E2 · Capstone exercise

E3 · Official documentation reference

E4 · Self-assessment

E5 · Sources

Module 05 — Boot & Init: Firmware to systemd