Module 12 — Namespaces, cgroups & Containers
Updated 22 August 2026
Every module so far described one machine: one set of processes, one filesystem tree, one network stack. This is where that stops being true. By the end you will be able to build a container by hand with three commands, and — far more useful — take one apart from the host when the tooling is not helping.
🧠 concept → 🧩 real-world analogy → 🧪 exercise → ✅ expected result (hidden) → 🎯 interview questions (hidden)
From Module 02 — fork, exec, PIDs, PID 1, and orphan reparenting.
From Module 03 — mounts, the mount table, and file descriptors.
From Module 07 — that a cgroup can cap CPU, and cpu.max.
From Module 09 — memory.max versus memory.high, memory.events, and exit code 137.
From Module 10 — io.stat and io.pressure.
From Module 11 — that abstract sockets are bounded by a network namespace.
Everything here uses util-linux, which is already installed: unshare, nsenter, lsns, findmnt.
🪪 Part A · Namespaces
A1 · What a namespace actually is
A namespace changes what a process can see, and nothing else. It does not restrict what a process may do, it does not limit resources, and it does not hide anything from the kernel — it simply gives that process a different view of one particular global resource.
There are eight, each covering one kind of thing:
| Namespace | What it partitions | Introduced |
| Mount (CLONE_NEWNS) | The set of mounts — what the filesystem tree looks like | 2.4.19 |
| UTS | Hostname and domain name | 2.6.19 |
| IPC | System V IPC objects and POSIX message queues | 2.6.19 |
| PID | Process ID numbers | 2.6.24 |
| Network | Interfaces, addresses, routes, ports, iptables | 2.6.24 |
| User | UIDs, GIDs and capabilities (Module 13) | 3.8 (usable) |
| Cgroup | What the cgroup tree looks like from inside | 4.6 |
| Time | The boot and monotonic clocks | 5.6 |
A namespace is identified by an inode number. /proc/PID/ns/ holds one symlink per type, and two processes are in the same namespace exactly when those inodes match. That is the whole comparison, and it is how every tool here works.
Imagine a building where every employee consults a shared directory: room numbers, phone extensions, the list of departments, the staff list.
A namespace is giving one team its own copy of one page of that directory. Their staff-list page says they are employees 1, 2 and 3 — while the building's real list has them at 4012, 4013 and 4014. Both are true. The team is not hidden, not locked in, not restricted; they simply read a different page.
Crucially, each page is separate. A team can have its own staff list while sharing the building's phone directory, or its own room numbering while sharing everything else. There is no single "private team" switch — there are eight independent pages, and you choose which to replace.
The part that surprises people: from the building manager's desk, all of this is visible. Their master directory shows every employee under their real number, including the team that thinks it is numbered 1 to 3. Nothing is hidden from the host — the view is one-way.
Where the analogy stops working. A directory page is a document you could photocopy. A namespace is a live kernel object, and processes can be moved between them at runtime with setns().
🧪 Exercise A1.1 — See your own namespaces, then leave one
# The eight namespaces this shell is in. The numbers are inodes.
ls -l /proc/self/ns/
# Every namespace on the machine, with how many processes are in each
lsns | head -12
# Create a new UTS namespace and change the hostname inside it
echo "host hostname: $(hostname)"
sudo unshare --uts bash -c '
hostname container-demo
echo "inside : $(hostname)"
readlink /proc/self/ns/uts
'
echo "host after: $(hostname)"
readlink /proc/self/ns/uts
# Which namespaces did that child NOT change?
sudo unshare --uts bash -c '
for n in uts pid net mnt ipc user cgroup time; do
printf "%-7s %s\n" "$n" "$(readlink /proc/self/ns/$n)"
done' > /tmp/child.ns
for n in uts pid net mnt ipc user cgroup time; do
printf "%-7s %s\n" "$n" "$(readlink /proc/self/ns/$n)"
done > /tmp/host.ns
diff /tmp/host.ns /tmp/child.ns
rm -f /tmp/host.ns /tmp/child.ns✅ Expected result — click to reveal
lrwxrwxrwx 1 zaeem zaeem 0 Aug 22 09:14 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 zaeem zaeem 0 Aug 22 09:14 ipc -> 'ipc:[4026531839]'
lrwxrwxrwx 1 zaeem zaeem 0 Aug 22 09:14 mnt -> 'mnt:[4026531840]'
lrwxrwxrwx 1 zaeem zaeem 0 Aug 22 09:14 net -> 'net:[4026531841]'
lrwxrwxrwx 1 zaeem zaeem 0 Aug 22 09:14 pid -> 'pid:[4026531836]'
lrwxrwxrwx 1 zaeem zaeem 0 Aug 22 09:14 pid_for_children -> 'pid:[4026531836]'
lrwxrwxrwx 1 zaeem zaeem 0 Aug 22 09:14 time -> 'time:[4026531834]'
lrwxrwxrwx 1 zaeem zaeem 0 Aug 22 09:14 time_for_children -> 'time:[4026531834]'
lrwxrwxrwx 1 zaeem zaeem 0 Aug 22 09:14 user -> 'user:[4026531837]'
lrwxrwxrwx 1 zaeem zaeem 0 Aug 22 09:14 uts -> 'uts:[4026531838]'
NS TYPE NPROCS PID USER COMMAND
4026531834 time 243 1 root /sbin/init
4026531835 cgroup 243 1 root /sbin/init
4026531836 pid 243 1 root /sbin/init
4026531837 user 243 1 root /sbin/init
4026531838 uts 243 1 root /sbin/init
4026531839 ipc 243 1 root /sbin/init
4026531840 net 243 1 root /sbin/init
4026531841 mnt 241 1 root /sbin/init
host hostname: vm-01
inside : container-demo
uts:[4026532297]
host after: vm-01
uts:[4026531838]
1c1
< uts uts:[4026531838]
---
> uts uts:[4026532297]What to read out of this.
Ten symlinks for eight namespaces. pid_for_children and time_for_children are not extra namespaces — they name the namespace this process's next child will be placed in, which is how unshare --pid can take effect without moving the calling process. They point at the same inodes as pid and time here because nothing has been unshared.
Every inode starts 4026531.... Those are the initial namespaces — the ones the kernel creates at boot and every ordinary process inherits. Seeing an inode in that range tells you at a glance that a process is not in a container. (The values for cgroup, ipc, pid, time, user and uts are fixed constants in the kernel; mnt and net are allocated at boot and can differ between machines.)
lsns confirms it: eight namespaces, all rooted at PID 1. That is what a minimal machine with no sandboxed services looks like — on a stock Ubuntu server several systemd units use PrivateTmp=, so you will usually see more than eight even before any container exists. On a container host the same command runs to dozens of lines.
The unshare --uts demonstration is the mechanism in three lines. Inside, hostname returns container-demo; on the host, immediately after, it is still vm-01. The inode is different — 4026532297 rather than 4026531838 — and that new number is what "a different namespace" physically means.
The diff at the end is the most important output here. Exactly one line differs. The child got a new UTS namespace and kept the host's PID, network, mount, IPC, user, cgroup and time namespaces. It could see every process on the machine, every network interface, and the entire filesystem. It was isolated in precisely one respect, because that is all we asked for.
That is the model to carry into everything below: namespaces are eight independent switches, not one. A "container" is a particular combination somebody chose, and different runtimes choose differently — which is exactly why some containers share the host's network and others do not.
If lsns shows far more than eight, something on the machine is already using namespaces: systemd sandboxing options, a container runtime, or a snap.
A2 · PID namespaces and the strange life of PID 1
A PID namespace gives processes a second set of PID numbers. The first process in one becomes PID 1 inside it, while keeping its ordinary PID on the host. Every process has as many PIDs as there are namespaces it sits inside — one per level — and none of them is more real than the others.
Being PID 1 inside a namespace carries three behaviours that are the source of a great many container problems:
Signals are filtered. A signal sent from inside the namespace is delivered to PID 1 only if it has installed a handler for it. There is no default action. So kill -TERM 1 inside a container does nothing at all unless the program explicitly handles SIGTERM — which is why docker stop so often waits its full ten seconds and then kills the container outright.
If PID 1 dies, the namespace dies. The kernel sends SIGKILL to every remaining process in it, and further fork() calls fail with ENOMEM. There is no recovery.
PID 1 is the reaper. From Module 02: orphans reparent to PID 1, which must wait() for them. A container whose PID 1 is an application that never reaps accumulates zombies indefinitely.
A large group sets up a subsidiary. Inside it, staff are numbered from 1: employee 1 is the managing director. On the group's master list those same people are 4012, 4013 and 4014. Both numbering schemes are correct and in use simultaneously.
The managing director has an unusual contract. Internal memos do not compel them: a request from their own staff to resign is simply filed unless the director has explicitly agreed to act on that kind of request. But an instruction from group head office is binding and immediate — because head office sits outside the subsidiary.
And if the managing director leaves, the subsidiary is dissolved: everyone in it is dismissed at once, and no new hires are possible. There is no deputy and no succession.
They are also responsible for all the paperwork nobody else will do — when a contractor finishes and nobody signs them off, the file lands on the director's desk. Ignore it and the filing cabinet fills with unclosed files.
Where the analogy stops working. A subsidiary's staff know they are in one. A process in a PID namespace has no reliable way to tell — which is why "am I in a container?" is answered by heuristics rather than by asking.
🧪 Exercise A2.1 — Build a PID namespace and watch PID 1 misbehave
# Without --mount-proc, ps reads the HOST's /proc and the isolation looks broken
echo "--- PID namespace, but /proc not remounted ---"
sudo unshare --pid --fork bash -c 'echo "I am PID $$"; ps -e --no-headers | wc -l'
# With it, the view matches the reality
echo "--- PID namespace with --mount-proc ---"
# The trailing `true` matters: bash exec-replaces itself with the LAST command
# of a -c string, so without it `ps` becomes PID 1 and the shell disappears.
sudo unshare --pid --fork --mount-proc bash -c '
echo "I am PID $$"
ps -ef
true
'
# The same process has two PIDs. Find both.
sudo unshare --pid --fork --mount-proc bash -c '
echo "inside: PID $$"; sleep 20' &
sleep 2
# Match the process NAME, not the command line: `pgrep -f` would also match
# the `sudo` and `unshare` wrappers, which live in the HOST PID namespace.
HOSTPID=$(pgrep -x sleep | tail -1)
echo "host sees that shell tree at PID $HOSTPID"
grep -E '^NSpid' /proc/$HOSTPID/status
# Signals to PID 1 from INSIDE are filtered unless it handles them
echo "--- kill -TERM 1 from inside ---"
sudo unshare --pid --fork --mount-proc bash -c '
( sleep 5 ) &
kill -TERM 1 2>/dev/null
echo "still alive after TERM to PID 1: yes"
kill -KILL 1 2>/dev/null
echo "still alive after KILL to PID 1: yes"
'
wait 2>/dev/null✅ Expected result — click to reveal
--- PID namespace, but /proc not remounted ---
I am PID 1
243
--- PID namespace with --mount-proc ---
I am PID 1
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 09:22 pts/0 00:00:00 bash -c ...
root 2 1 0 09:22 pts/0 00:00:00 ps -ef
inside: PID 1
host sees that shell tree at PID 9412
NSpid: 9412 1
--- kill -TERM 1 from inside ---
still alive after TERM to PID 1: yes
still alive after KILL to PID 1: yesWhat to read out of this.
The first block is the trap. The shell correctly reports I am PID 1 — the namespace is real and working — and then ps counts 243 processes, the whole machine. Nothing is broken: ps reads /proc, /proc is a mount, and mounts belong to the mount namespace, which we did not unshare. The isolation is genuine and the tool is looking at the wrong place.
With --mount-proc the view is consistent: two processes, the shell as PID 1 and ps itself. That flag does two things — unshares the mount namespace and mounts a fresh procfs — and forgetting it is the single most common reason a hand-built PID namespace "does not work".
(The true at the end of that block is not decoration. Bash exec-replaces itself with the last command of a -c string, so without it ps inherits PID 1, the shell vanishes, and the output is a single line showing ps as PID 1 — a genuinely confusing result for an exercise about PID 1.)
NSpid: 9412 1 is the clearest line in the exercise. One process, two PIDs, listed outermost first: 9412 on the host, 1 inside its namespace. Neither is more real. This field is how you map a container process back to the host, and it works with no tooling at all.
Then the signal behaviour. kill -TERM 1 and kill -KILL 1 from inside both did nothing — the shell survived both and printed its lines. Even SIGKILL, which Module 04 established cannot be caught or ignored, is filtered here because the target is PID 1 of the sender's own namespace and it has no handler installed. From the host, the same SIGKILL would work instantly, because the host is an ancestor namespace.
This is exactly the mechanism behind docker stop: it sends SIGTERM from the host, waits ten seconds, then sends SIGKILL. If the container's PID 1 handles SIGTERM it exits promptly; if not, nothing happens until the timeout expires and the kill arrives.
If your first block shows a small number rather than 243, you are already inside a container, and /proc is already namespaced.
A3 · Mount and network namespaces
These two do the heavy lifting in a real container.
A mount namespace gives a process its own mount table. Mounting something inside it is invisible outside, and vice versa. This is what lets a container have a completely different filesystem tree — and it is a view of mounts, not a restriction: a process in its own mount namespace can still reach anything already mounted when it was created, unless the runtime also changes its root.
Mounts also have a propagation type, which decides whether a mount made in one namespace shows up in another. shared propagates both ways, private neither way, slave receives but does not send. systemd marks / as shared at boot, so a raw unshare(CLONE_NEWNS) system call inherits shared propagation and any mount you make leaks straight back to the host. The unshare(1) command guards against this for you — since util-linux 2.27 it sets propagation to private by default — so the --propagation private in the exercise below is explicit rather than required. Pass --propagation unchanged if you want to watch the leak happen.
A network namespace gives a process its own network stack: its own interfaces, addresses, routes, firewall rules and port numbers. A fresh one contains exactly one interface, lo, and it is DOWN. Nothing can reach in or out until you connect it to something, usually a veth pair with one end in each namespace.
The mount namespace is the floor plan. Two teams can be handed different plans of the same building: one team's plan shows a library where the other's shows a storeroom. Neither is lying, and a door added to one plan does not appear on the other.
Mount propagation is the rule about whether a change to the master plan reaches the copies. shared means both directions, slave means updates flow down but not back, private means the copy is frozen. Getting this wrong is why a device plugged into the host sometimes fails to appear inside a container.
The network namespace is a completely separate phone system. Its own extension numbers, its own switchboard, its own directory. Extension 80 in one system and extension 80 in another are unrelated, so a hundred teams can all have an extension 80 without conflict.
And a new phone system arrives not connected to anything — the internal handsets are installed but the line is dead. Someone has to run a cable to the building's switchboard, which is the veth pair.
Where the analogy stops working. A floor plan is inert. Mount namespaces have live propagation rules, so a change in one really can appear in another without anyone acting.
🧪 Exercise A3.1 — Mount something nobody else can see, and take a machine off the network
# --- Mount namespace ---
echo "host mount count: $(findmnt -ln | wc -l)"
sudo unshare --mount --propagation private bash -c '
mkdir -p /tmp/private-mnt
mount -t tmpfs none /tmp/private-mnt
echo "secret" > /tmp/private-mnt/file
echo "inside : mounts=$(findmnt -ln | wc -l), file says $(cat /tmp/private-mnt/file)"
'
echo "host after: mounts=$(findmnt -ln | wc -l)"
echo "host sees the file? $(cat /tmp/private-mnt/file 2>&1 | head -1)"
# --- Network namespace ---
echo "--- host network ---"
ip -br addr
echo "--- a fresh network namespace ---"
sudo unshare --net bash -c '
ip -br addr
echo "routes: $(ip route | wc -l)"
echo "listening sockets: $(ss -tln | tail -n +2 | wc -l)"
'
# Two namespaces can both own port 8080 with no conflict
sudo unshare --net bash -c 'ip link set lo up
(python3 -m http.server 8080 --bind 127.0.0.1 >/dev/null 2>&1 &)
sleep 1; echo "namespace A: $(ss -tln | grep -c 8080) listener on 8080"' &
sudo unshare --net bash -c 'ip link set lo up
(python3 -m http.server 8080 --bind 127.0.0.1 >/dev/null 2>&1 &)
sleep 1; echo "namespace B: $(ss -tln | grep -c 8080) listener on 8080"' &
wait
echo "host sees on 8080: $(ss -tln | grep -c 8080)"✅ Expected result — click to reveal
host mount count: 34
inside : mounts=35, file says secret
host after: mounts=34
host sees the file? cat: /tmp/private-mnt/file: No such file or directory
--- host network ---
lo UNKNOWN 127.0.0.1/8 ::1/128
eth0 UP 10.0.2.15/24 fe80::5054:ff:fe12:3456/64
--- a fresh network namespace ---
lo DOWN
routes: 0
listening sockets: 0
namespace A: 1 listener on 8080
namespace B: 1 listener on 8080
host sees on 8080: 0What to read out of this.
The mount count went 34 → 35 → 34. Inside the namespace the tmpfs existed and the file could be read; the moment that shell exited, the mount was gone and the host could never see it at all. Note the directory /tmp/private-mnt does still exist on the host — mkdir happened in a shared filesystem — but it is empty. The mount was private; the directory was not. That distinction catches people out constantly.
--propagation private is explicit rather than load-bearing here: unshare(1) has defaulted to private propagation since util-linux 2.27. To see the mechanism it protects you from, run the same block with --propagation shared instead — the tmpfs then appears on the host too, and findmnt proves it. That is what a raw unshare(CLONE_NEWNS) syscall gets by default on a systemd machine, and it is why container runtimes are careful about propagation.
Now the network side. A fresh namespace has one interface, lo, and it is DOWN. Zero routes. Zero listening sockets. It is not a restricted network — it is no network at all, and until something connects it, a process inside cannot even reach 127.0.0.1.
Then the port demonstration, which is the one to remember. Two separate namespaces each have a listener on 8080, and the host has none. No conflict, no EADDRINUSE, and nothing visible from outside. That is precisely how a hundred containers all listen on port 80 on one machine, and it is why ss -tlnp on a container host is so often misleadingly empty — the sockets exist, in namespaces you did not look in.
If the two unshare --net blocks print 0, python3 failed to start; try nc -l 8080 or check that lo came up.
A4 · User namespaces — being root without being root
A user namespace maps UIDs and GIDs between the inside and the outside. UID 0 inside can be UID 1000 outside — so a process can be root within its own world while remaining a completely ordinary unprivileged user to the rest of the machine.
The mapping is written to /proc/PID/uid_map and gid_map, three numbers per line: ID inside, ID outside, range length. So 0 1000 1 means "UID 0 in here is UID 1000 out there, for a range of one". The file is write-once — a second write fails with EPERM — and since Linux 4.16 it may hold up to 340 lines.
An unprivileged user has been able to create a user namespace since Linux 3.8, and gets full privileges inside it. That is what makes rootless containers possible: Podman and rootless Docker run entirely as your own user, using ranges from /etc/subuid and /etc/subgid to map a block of otherwise-unused UIDs into the container.
A contractor is issued a visitor badge that reads "Site Manager" and opens every door on the second floor. Inside that floor they genuinely are the site manager: they can move furniture, unlock rooms, sign for deliveries.
Take the lift to the ground floor and the badge means nothing. To reception they are contractor #1000 with no authority at all. Both facts are true at once, and which applies depends only on where they are standing.
The badge system needs a translation table at the door: "badge 0 on this floor is contractor 1000 in the building register". That table is written once when the badge is issued and cannot be edited afterwards.
And the awkward part. Deliveries addressed to someone with no entry in the translation table arrive labelled "unknown recipient". The parcel is fine, the sender is fine — but from inside the second floor nobody can tell who it belongs to, and it cannot be signed for.
Where the analogy stops working. A visitor badge is checked by people who can use judgement. UID mapping is arithmetic: an unmapped ID becomes 65534 with no consideration of intent, every time.
🧪 Exercise A4.1 — Be root in a namespace, as an ordinary user
# Is your distribution allowing unprivileged user namespaces?
sysctl kernel.apparmor_restrict_unprivileged_userns 2>/dev/null || \
echo "(not an AppArmor-restricting kernel)"
sysctl user.max_user_namespaces 2>/dev/null
# Ubuntu 24.04 blocks them by default, so allow them for this exercise.
# Decide deliberately - see the warning above - and put it back afterwards.
sudo sysctl -q -w kernel.apparmor_restrict_unprivileged_userns=0 2>/dev/null
sysctl kernel.apparmor_restrict_unprivileged_userns 2>/dev/null
# Who are you now?
echo "outside: $(id -u) ($(id -un))"
# -U makes a new user namespace, -r maps your UID to root inside it.
# NOTE: no sudo. This is an ordinary user becoming root in its own world.
unshare -U -r bash -c '
echo "inside : $(id -u) ($(id -un))"
echo "--- the mapping the kernel wrote ---"
cat /proc/self/uid_map
cat /proc/self/gid_map
echo "--- can this root create a file in /root? ---"
touch /root/proof 2>&1 | head -1
'
# Unmapped IDs show as nobody (overflowuid)
cat /proc/sys/kernel/overflowuid
unshare -U -r bash -c 'ls -ld /usr/bin/sudo; stat -c "%U:%G" /etc/shadow'
ls -ld /usr/bin/sudo
# A user namespace is what lets an unprivileged user do the rest of it
echo "--- unprivileged user namespace plus a PID namespace ---"
unshare -U -r --pid --fork --mount-proc bash -c 'echo "PID $$ as $(id -un)"; ps -e --no-headers | wc -l'
# Put the restriction back
sudo sysctl -q -w kernel.apparmor_restrict_unprivileged_userns=1 2>/dev/null✅ Expected result — click to reveal
kernel.apparmor_restrict_unprivileged_userns = 1
user.max_user_namespaces = 63704
kernel.apparmor_restrict_unprivileged_userns = 0
outside: 1000 (zaeem)
inside : 0 (root)
--- the mapping the kernel wrote ---
0 1000 1
0 1000 1
--- can this root create a file in /root? ---
touch: cannot touch '/root/proof': Permission denied
65534
-rwsr-xr-x 1 nobody nogroup 277936 Apr 8 2026 /usr/bin/sudo
nobody:nogroup
-rwsr-xr-x 1 root root 277936 Apr 8 2026 /usr/bin/sudo
--- unprivileged user namespace plus a PID namespace ---
PID 1 as root
3What to read out of this.
id -u returns 1000 outside and 0 inside, and there was no sudo anywhere in that command. That is a user namespace: a genuine, kernel-recognised UID 0, scoped to one namespace.
The map reads 0 1000 1 — UID 0 inside is UID 1000 outside, range of one. Exactly one identity was mapped, which is why everything else appears as nobody.
Now the line that matters most: touch /root/proof was denied. Being root inside the namespace grants privileges over the namespace's own resources, not over the host's files. The host filesystem is checked against the outside UID, which is still 1000. If a user namespace made you root over the whole machine it would be a trivial privilege escalation, and it is not one.
/usr/bin/sudo shows as nobody nogroup inside and root root outside — same file, same mode, same size, only the owner names differ. (On Debian and Ubuntu, UID 65534 is nobody but GID 65534 is nogroup, which is why the two halves do not match.) Nothing changed on disk. UID 0 on the host has no entry in our one-line map, so it renders as overflowuid, 65534. That is the entire explanation for the classic "my volume mount shows as nobody and I cannot write to it" — a mapping gap, not a permissions problem, and chmod will not fix it.
The last block is the punchline. As an ordinary user with no sudo at all, we created a user namespace, a PID namespace and a fresh /proc, and ended up as PID 1 running as root with three visible processes — the shell, plus ps and wc from the counting pipeline. That is the foundation of rootless containers, and it is available to any user on a permissive kernel.
Notice the sysctl in the transcript: it reads 1 — Ubuntu 24.04's default — and the exercise sets it to 0 before the unshare -U calls, then puts it back. Without that step every unshare -U above fails with Operation not permitted, because AppArmor denies unprivileged user-namespace creation to any program without a profile permitting it. On a distribution that does not carry the AppArmor restriction the first sysctl prints nothing and the rest works unchanged.
🎯 Interview questions — Namespaces
Q. What are Linux namespaces?
A namespace changes what a process can see of one particular global resource, and nothing else. There are eight: mount, UTS, IPC, PID, network, user, cgroup and time. Each is independent, so a process can have its own network stack while sharing everything else, or its own PID numbering while sharing the filesystem.
They are identified by inode numbers, exposed as symlinks in /proc/PID/ns/. Two processes are in the same namespace exactly when those inodes match, which is how every tool in this area works.
The details that separate candidates: being clear that namespaces isolate visibility, not privilege and not resources. Limiting what a process may do is capabilities and seccomp; limiting what it may consume is cgroups. Namespaces alone give you neither. The other detail worth having: the version table in namespaces(7) shows when each /proc/PID/ns/ file appeared, not when the namespace was introduced — mount namespaces date from 2.4.19 in 2002, long before containers existed, and quoting 3.8 for them is a sign of having read the wrong column.
Q. Why does docker stop often take the full timeout before the container dies?
Because PID 1 in a PID namespace does not have default signal actions. A signal sent to it is delivered only if the process has installed a handler for that signal. docker stop sends SIGTERM and waits — typically ten seconds — and if the container's PID 1 has no SIGTERM handler, absolutely nothing happens until the timeout expires and SIGKILL is sent from the host.
The usual cause is a shell-form entrypoint, where PID 1 is /bin/sh -c "myapp" and the shell neither handles SIGTERM nor forwards it to the application.
The details that separate candidates: the asymmetry. SIGKILL and SIGSTOP are delivered to a namespace's PID 1 when sent from an ancestor namespace, which is exactly how the host eventually forces the kill — but from inside, even SIGKILL to PID 1 is filtered. The practical fixes are worth naming too: exec form rather than shell form so the application really is PID 1, or a small init as PID 1 (docker run --init) which handles signals and reaps zombies properly. Zombie accumulation in containers has the same root cause.
Q. What is a user namespace, and how does it make rootless containers possible?
It maps UIDs and GIDs between inside and outside, so UID 0 inside can be an ordinary unprivileged UID on the host. A process is genuinely root within its namespace — it can do the things root does to that namespace's own resources — while the kernel still checks host filesystem access against the outside UID. An unprivileged user has been able to create one since Linux 3.8, which is what allows Podman and rootless Docker to run without any privileged daemon: they map a block of UIDs from /etc/subuid into the container.
The details that separate candidates: explaining why this is not a privilege escalation — the mapping is bounded, and host resources are still checked against the outer identity — and then naming the real cost: unprivileged user namespaces are the main container-escape surface, because they expose kernel code paths that previously needed root. Ubuntu 24.04 restricts them by default via kernel.apparmor_restrict_unprivileged_userns. The other detail with obvious operational value is the nobody problem: an unmapped UID renders as overflowuid, 65534, so a volume owned by host root shows as nobody inside and chmod will never fix it.
🎚️ Part B · cgroups v2
B1 · The unified hierarchy
Namespaces decide what a process sees. Cgroups decide what it may use. You have already used them in Modules 07, 09 and 10 — cpu.max, memory.max, io.stat — and this is where the structure behind them fits together.
The one-sentence difference from the old design: cgroup v1 gave every controller its own independent hierarchy, so a process could sit in unrelated places for CPU and for memory, whereas v2 has a single unified tree that every controller sees the same way. That change removed a genuine class of impossible-to-reason-about configurations.
Two rules govern the tree:
A cgroup enables controllers for its children, not for itself. cgroup.subtree_control is where you write +cpu or -memory, and it affects the level below. A controller can only be enabled in a child if the parent already enabled it, so control flows strictly downward.
No internal processes. A non-root cgroup may not both contain processes and distribute resources to children. Processes live on the leaves. Try to violate it and the write to cgroup.procs fails with EBUSY, which is a confusing error until you know the rule.
The company has one budget tree. Head office allocates to divisions, divisions to departments, departments to teams.
cgroup v1 was a separate tree per resource: one org chart for money, an unrelated one for office space, a third for equipment. A team could sit under Sales for budget and under Engineering for desks. It worked, and reasoning about it was miserable. v2 uses one chart for everything.
"A cgroup enables controllers for its children" is the rule that a division decides which resources its departments are allowed to be given limits on, not which limits apply to the division itself. And you cannot give a department control over money that your own division was never given.
"No internal processes" is the rule that a division either has staff or has departments, not both. Once you start subdividing, everyone moves down into a department. It sounds bureaucratic until you try to answer "what is this division's CPU limit" for a body that is simultaneously a manager and a worker — the accounting has no consistent answer, so the design forbids the situation.
Where the analogy stops working. A budget is spent and gone. A cgroup limit is a rate or a ceiling that is re-evaluated continuously, so a cgroup that goes quiet immediately releases everything it was using.
🧪 Exercise B1.1 — Walk the tree, then break the rules on purpose
# Is this machine on v2? cgroup2fs means unified.
stat -fc %T /sys/fs/cgroup
# The top of the tree, and which controllers are available at all
cat /sys/fs/cgroup/cgroup.controllers
echo "enabled for children of root: $(cat /sys/fs/cgroup/cgroup.subtree_control)"
# What the tree looks like on a systemd machine. Directories only: files sort
# first alphabetically, so a plain `ls | head` shows nothing but cgroup.* files.
ls -d /sys/fs/cgroup/*/ | head -6
systemd-cgls --no-pager 2>/dev/null | head -14
# Where is THIS shell? One line, and the path is the cgroup.
cat /proc/self/cgroup
# --- Build a small tree and violate "no internal processes" ---
sudo mkdir -p /sys/fs/cgroup/demo/child
echo "+cpu +memory" | sudo tee /sys/fs/cgroup/cgroup.subtree_control >/dev/null
echo "+cpu +memory" | sudo tee /sys/fs/cgroup/demo/cgroup.subtree_control >/dev/null
echo "--- files that appeared in demo/child ---"
# Name the files explicitly: there are nine cpu.* files and they all sort
# before memory.*, so a `head -6` would never reach the memory controller.
ls /sys/fs/cgroup/demo/child/ | grep -E '^(cpu\.max|cpu\.stat|memory\.(current|events|max))$'
echo "--- putting a process in demo (which has a child) ---"
sleep 60 &
P=$!
echo $P | sudo tee /sys/fs/cgroup/demo/cgroup.procs 2>&1 | tail -1
echo "--- putting it in the LEAF instead ---"
echo $P | sudo tee /sys/fs/cgroup/demo/child/cgroup.procs >/dev/null && echo "accepted"
cat /proc/$P/cgroup
kill $P 2>/dev/null; sleep 1
sudo rmdir /sys/fs/cgroup/demo/child /sys/fs/cgroup/demo 2>/dev/null✅ Expected result — click to reveal
cgroup2fs
cpuset cpu io memory hugetlb pids rdma misc
enabled for children of root: cpuset cpu io memory pids
/sys/fs/cgroup/dev-hugepages.mount/
/sys/fs/cgroup/dev-mqueue.mount/
/sys/fs/cgroup/init.scope/
/sys/fs/cgroup/sys-fs-fuse-connections.mount/
/sys/fs/cgroup/sys-kernel-config.mount/
/sys/fs/cgroup/system.slice/
Control group /:
-.slice
├─user.slice
│ └─user-1000.slice
│ └─session-3.scope
│ ├─ 2841 sshd: zaeem [priv]
│ └─ 2903 -bash
└─system.slice
├─ssh.service
└─systemd-journald.service
0::/user.slice/user-1000.slice/session-3.scope
--- files that appeared in demo/child ---
cpu.max
cpu.stat
memory.current
memory.events
memory.max
--- putting a process in demo (which has a child) ---
tee: /sys/fs/cgroup/demo/cgroup.procs: Device or resource busy
--- putting it in the LEAF instead ---
accepted
0::/demo/childWhat to read out of this.
cgroup2fs confirms the unified hierarchy. cgroup.controllers lists what the kernel has; cgroup.subtree_control lists what has been enabled for the level below — and they are different, which is the distinction the whole design turns on.
systemd-cgls shows the shape every systemd machine has: user.slice for logged-in sessions, system.slice for services, init.scope for PID 1. Your shell sits several levels down, and /proc/self/cgroup gives its exact path in one line: 0::/user.slice/user-1000.slice/session-3.scope. The 0:: prefix means cgroup v2 — on v1 you would see many numbered lines, one per controller hierarchy.
Now the two writes. Enabling +cpu +memory in demo/cgroup.subtree_control made cpu.max, memory.max and their companions appear inside demo/child. Those files did not exist a moment earlier. A cgroup's control files are created by its parent's decision, which is why an empty-looking cgroup directory usually means the controller was never enabled above it.
Then the deliberate failure. Writing a PID into demo/cgroup.procs returned Device or resource busy — EBUSY — because demo has a child and distributes resources, so it may not also hold processes. The same write into demo/child was accepted immediately. That is the "no internal processes" rule, and EBUSY is its error message. It is worth recognising, because nothing in the message says so.
If stat -fc %T says tmpfs rather than cgroup2fs, the machine is on cgroup v1 or a hybrid — an old distribution, or booted with systemd.unified_cgroup_hierarchy=0. Most of this section will not apply.
B2 · The controllers you will actually use
Every limit you have met in earlier modules is a file in a cgroup directory. Here they are together:
| File | What it does | Where you met it |
| cpu.max | $QUOTA $PERIOD in microseconds. 200000 100000 is two CPUs' worth | Module 07 |
| cpu.weight | Relative share under contention, 1–10000, default 100 | Module 07 |
| memory.max | Hard limit. Breaching it invokes the OOM killer inside this cgroup | Module 09 |
| memory.high | Throttle and reclaim hard. Never invokes the OOM killer | Module 09 |
| io.max | Per-device bytes/s and IOPS ceiling | Module 10 |
| pids.max | Maximum number of processes. The fork-bomb defence | New here |
| cgroup.procs | Read the members; write a PID to move the whole process | New here |
| cgroup.events | populated and frozen. Pollable — the correct way to watch for exit | New here |
| cgroup.kill | Write 1 to SIGKILL the entire subtree at once | New here |
| *.pressure | PSI, scoped to this cgroup | Modules 09 and 10 |
A department has a spending limit (memory.max), a share of the shared machine shop (cpu.weight), and a cap on hours booked in it (cpu.max). Those are the limits everybody thinks about.
pids.max is the headcount cap, and it is the one people forget. A department with a small budget can still hire ten thousand unpaid interns, and while the budget is untouched, the building runs out of desks, passes and parking spaces — and every other department is now unable to work. That is a fork bomb: it costs almost no memory and no CPU, and it consumes a resource the whole building shares.
cgroup.kill is the difference between going desk to desk asking people to leave — while they keep hiring behind you — and revoking every pass in the department at once. Only the second one terminates.
Where the analogy stops working. A department dissolved this way could be rebuilt from records. cgroup.kill is SIGKILL: no handlers, no cleanup, nothing flushed.
🧪 Exercise B2.1 — Cap a workload four ways, then kill it atomically
sudo mkdir -p /sys/fs/cgroup/demo
echo "+cpu +memory +pids" | sudo tee /sys/fs/cgroup/cgroup.subtree_control >/dev/null
# Four limits, four files
echo "20000 100000" | sudo tee /sys/fs/cgroup/demo/cpu.max >/dev/null # 20% of one CPU
echo "64M" | sudo tee /sys/fs/cgroup/demo/memory.max >/dev/null
echo "48M" | sudo tee /sys/fs/cgroup/demo/memory.high >/dev/null
echo "20" | sudo tee /sys/fs/cgroup/demo/pids.max >/dev/null
grep . /sys/fs/cgroup/demo/{cpu.max,memory.high,memory.max,pids.max}
# Run a busy loop inside it and watch the CPU accounting
sudo bash -c 'echo $$ > /sys/fs/cgroup/demo/cgroup.procs
timeout 5 bash -c "while :; do :; done"'
echo "--- cpu.stat after 5 seconds of a busy loop ---"
grep -E 'usage_usec|nr_periods|nr_throttled|throttled_usec' /sys/fs/cgroup/demo/cpu.stat
# pids.max stops a fork bomb without CPU or memory being involved
echo "--- hitting pids.max ---"
sudo bash -c 'echo $$ > /sys/fs/cgroup/demo/cgroup.procs
# NOTE: `while cmd & do ...` would loop forever - a background command
# always reports success. Check whether the child really exists instead.
n=0
while [ $n -lt 40 ]; do
sleep 30 & pid=$!
kill -0 $pid 2>/dev/null || break
n=$((n+1))
done
echo "managed to start $n background processes"' 2>&1 | tail -2
cat /sys/fs/cgroup/demo/pids.current
# cgroup.events tells you whether anything is still in there
cat /sys/fs/cgroup/demo/cgroup.events
# One write kills the whole subtree, race-free
echo "--- cgroup.kill ---"
echo 1 | sudo tee /sys/fs/cgroup/demo/cgroup.kill >/dev/null
sleep 1
cat /sys/fs/cgroup/demo/cgroup.events
sudo rmdir /sys/fs/cgroup/demo✅ Expected result — click to reveal
/sys/fs/cgroup/demo/cpu.max:20000 100000
/sys/fs/cgroup/demo/memory.high:50331648
/sys/fs/cgroup/demo/memory.max:67108864
/sys/fs/cgroup/demo/pids.max:20
--- cpu.stat after 5 seconds of a busy loop ---
usage_usec 1004182
nr_periods 50
nr_throttled 49
throttled_usec 3971204
--- hitting pids.max ---
bash: fork: retry: Resource temporarily unavailable
managed to start 18 background processes
20
populated 1
frozen 0
--- cgroup.kill ---
populated 0
frozen 0What to read out of this.
memory.max was written as 64M and reads back as 67108864 — the kernel accepts the suffix and stores bytes. Reading a limit back is the only way to be sure of what was actually set. Note memory.high sitting below memory.max at 48 MiB: that is the arrangement from Module 09 that gives a workload a soft landing — throttled and reclaimed hard at 48 MiB, killed only if it reaches 64 MiB.
cpu.stat after five seconds of a flat-out busy loop: usage_usec 1004182, about one second of CPU. The limit was 20% of one CPU, and 20% of five seconds is one second. It worked exactly as configured.
The interesting pair is underneath. nr_throttled 49 and throttled_usec 3971204 — in 49 of the 50 hundred-millisecond periods, the loop exhausted its quota and was frozen for the remainder, about four seconds in total. From inside, that process spent four of its five seconds stopped dead. This is the Module 07 point made concrete: a CPU limit is a quota per period, not a smooth slowdown.
Then pids.max. The shell got fork: retry: Resource temporarily unavailable at 18 processes, with pids.current at the ceiling of 20. Note what was not involved: no memory pressure, no CPU exhaustion, no OOM kill. A fork bomb is stopped by a counter, and only by a counter.
cgroup.events reads populated 1 while processes remain and populated 0 after. That file is pollable: a supervisor can wait on it with poll() and be woken the instant the last process exits, instead of busy-looping over cgroup.procs.
And cgroup.kill emptied the whole subtree with one write. Compare that with the loop it replaces — read cgroup.procs, kill each PID, read again — which races against anything still forking. cgroup.kill sets a flag the kernel checks at every fork() in the subtree, so a fork bomb cannot outrun it.
If cgroup.kill does not exist, the kernel predates 5.14. Fall back to the loop, and know that it can fail to converge.
B3 · Finding a container's cgroup from the host
Every limit a container has is a file under /sys/fs/cgroup, and finding that directory is the skill that makes container incidents tractable without any runtime tooling.
systemd builds the tree using slices and scopes, and its naming convention is the one thing to learn: - is the hierarchy separator, and each level repeats the full ancestor prefix. So kubepods-burstable.slice sits inside kubepods.slice, and kubepods-burstable-pod<UID>.slice inside that.
A Kubernetes pod on a systemd-cgroup-driver node therefore lands here:
/sys/fs/cgroup/kubepods.slice/
kubepods-burstable.slice/
kubepods-burstable-pod4a2f9c11_8e3b_4d77_9a10_5c6d2e8f0b3a.slice/
cri-containerd-<64-hex-container-id>.scope/ <- one per container
memory.max cpu.max pids.max io.stat memory.eventsTwo details that trip people up: Guaranteed pods skip the QoS level entirely — kubepods.slice/kubepods-pod<UID>.slice/ — and the pod UID's dashes become underscores, because - is already the separator.
A large hospital numbers rooms by repeating the whole path: Building-Wing-Floor-Room. Every sign carries the full ancestry, so East-3-Cardiology-14 tells you exactly where you are standing without a map.
It is verbose, and it means a room's name changes if you move it — but it makes every label self-describing, which is worth a great deal when someone reads it out over the phone at three in the morning.
And here is the part people miss: you do not have to decode the scheme. Every patient's wristband already has the full room code printed on it. Reading the wristband beats reasoning about the numbering, every time. That wristband is /proc/PID/cgroup.
Where the analogy stops working. Hospital rooms are permanent. A container's cgroup directory disappears the moment the container exits, taking every counter with it — which is why you capture the numbers before restarting anything.
🧪 Exercise B3.1 — Map a process to its cgroup and back
# Any process: one line gives you its exact cgroup
cat /proc/self/cgroup
cat /proc/1/cgroup
# The tree as systemd sees it, with resource usage per cgroup
systemd-cgtop -n 1 --order=cpu 2>/dev/null | head -8
# Create a real scope, the same way a container runtime does
sudo systemd-run --unit=demo-scope --scope -p MemoryMax=64M -p CPUQuota=20% \
--quiet sleep 120 &
sleep 2
P=$(pgrep -x sleep | tail -1)
echo "sleep is host PID $P"
echo "--- its cgroup, straight from /proc ---"
CG=$(awk -F: '{print $3}' /proc/$P/cgroup)
echo "$CG"
echo "--- the limits that were applied ---"
grep . /sys/fs/cgroup$CG/{cpu.max,memory.max} 2>/dev/null
echo "--- and back the other way: who is in that cgroup? ---"
cat /sys/fs/cgroup$CG/cgroup.procs
# On a container host, the same path names the pod and container
echo "--- what a container's line looks like ---"
echo "0::/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2f9c11_8e3b_4d77_9a10_5c6d2e8f0b3a.slice/cri-containerd-9f3c1e21b7a45d8e6c02f19a3d47b8e05c6a1f92d38e47b0a95c2e1f60d83b47.scope"
sudo systemctl stop demo-scope.scope 2>/dev/null
kill $P 2>/dev/null✅ Expected result — click to reveal
0::/user.slice/user-1000.slice/session-3.scope
0::/init.scope
Control Group Tasks %CPU Memory Input/s Output/s
/ 243 2.1 1.2G - -
system.slice 88 1.4 712.4M - -
user.slice 41 0.6 284.1M - -
system.slice/snapd.service 18 0.9 94.2M - -
sleep is host PID 10482
--- its cgroup, straight from /proc ---
/system.slice/demo-scope.scope
--- the limits that were applied ---
/sys/fs/cgroup/system.slice/demo-scope.scope/cpu.max:20000 100000
/sys/fs/cgroup/system.slice/demo-scope.scope/memory.max:67108864
--- and back the other way: who is in that cgroup? ---
10482
--- what a container's line looks like ---
0::/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2f9c11_8e3b_4d77_9a10_5c6d2e8f0b3a.slice/cri-containerd-9f3c1e21b7a45d8e6c02f19a3d47b8e05c6a1f92d38e47b0a95c2e1f60d83b47.scopeWhat to read out of this.
/proc/self/cgroup gives one line beginning 0::, and everything after the second colon is the path under /sys/fs/cgroup. That 0:: prefix is the tell for cgroup v2; under v1 you would see a dozen numbered lines, one per controller hierarchy, often with different paths — which is exactly the confusion v2 removed.
PID 1 sits in /init.scope. That is systemd placing itself in a cgroup, which is worth knowing when you are reading a tree and wondering where init went.
The systemd-run --scope block is the important part, because it is structurally identical to what a container runtime does: create a scope, apply limits, put a process in it. CPUQuota=20% became cpu.max: 20000 100000 and MemoryMax=64M became memory.max: 67108864. The friendly names are systemd's; the files are the kernel's, and everything you learned in Modules 07, 09 and 10 applies to them unchanged.
Then the round trip. /proc/$P/cgroup gave the path; cgroup.procs in that path gave the PID back. Those two files are the whole mapping in both directions, and neither needs root or any tooling.
The last line is what a real Kubernetes container looks like. Read it left to right: kubepods.slice → QoS class burstable → the pod, with the pod UID's dashes turned into underscores → the container scope, named for the runtime and the container ID. Given that one line from /proc/PID/cgroup, you know the pod, the QoS class and the container — before touching kubectl.
If systemd-cgtop shows nothing for CPU, it needs a second sample; run it without -n 1 for a moment.
🎯 Interview questions — cgroups
Q. What are cgroups and how do they work?
Cgroups limit and account for what a group of processes may use — CPU, memory, I/O, process count — as opposed to namespaces, which change what those processes can see. In cgroup v2 they form a single unified tree under /sys/fs/cgroup; each directory is a cgroup, cgroup.procs lists its members, and the limits are ordinary files: cpu.max, memory.max, io.max, pids.max.
Two structural rules matter. A cgroup enables controllers for its children by writing to cgroup.subtree_control, and control only flows downward. And no internal processes: a non-root cgroup cannot both hold processes and distribute resources to children, so processes live on the leaves.
The details that separate candidates: the v1-versus-v2 difference — v1 gave each controller its own independent hierarchy, so a process could sit in unrelated places for CPU and memory, while v2 uses one tree that every controller shares. And that this is settled history now: systemd removed v1 support entirely in version 258, so on any current distribution v2 is not a preference but the only option.
Q. A container is CPU-limited and its usage graph looks fine, but it is slow. Why?
Almost certainly CFS throttling. cpu.max is a quota per 100 ms period, not a smooth rate. A container that wakes several threads at once can burn its whole allowance in the first few milliseconds and then be frozen for the rest of the period. Averaged over a minute the usage looks comfortable; in reality the process is stopping dead many times a second, which destroys tail latency while leaving the median untouched.
The evidence is in cpu.stat: nr_throttled against nr_periods, and throttled_usec. A container throttled in 49 of 50 periods is spending most of its life stopped, and no utilisation metric shows it.
The details that separate candidates: knowing that the fix is often not to raise the limit. Reducing the worker or thread count so the same work spreads across the period frequently removes the throttling entirely at the same quota. And naming cpu.pressure — PSI scoped to that cgroup — as the metric that quantifies the harm rather than the activity.
Q. A container fork-bombs. What contains it, and what does not?
pids.max, and nothing else. A fork bomb uses almost no memory, so memory.max never triggers; its CPU is throttled quite happily by cpu.max while it keeps forking. What it exhausts is the host's global PID space, which is shared by everything on the machine — so the node stops being able to start new processes at all, including SSH and systemd's own recovery.
pids.max is a simple counter on the cgroup, fork() fails with EAGAIN once it is reached, and the blast radius is confined to that container.
The details that separate candidates: pointing out that Kubernetes sets a per-pod default while a plain docker run does not unless you pass --pids-limit, so a Docker-based fleet is usually unprotected. And knowing the right way to clean up afterwards: cgroup.kill, added in Linux 5.14 — a single write of 1 that SIGKILLs the entire subtree and is checked at every fork(), so it cannot be outrun. The obvious alternative, looping over cgroup.procs and killing each PID, races against a process forking faster than you can read and may never converge.
📦 Part C · What a container actually is
C1 · Building one by hand
A container is a process with seven things done to it, none of which the kernel calls a container:
| Mechanism | What it controls | Covered in |
| Namespaces | What it can see | Part A |
| cgroups | What it can use | Part B |
| A new root filesystem | What files exist for it | Here, and Section C2 |
| Capabilities | Which pieces of root privilege it keeps | Module 13 |
| seccomp | Which system calls it may make at all | Module 13 |
| An LSM (AppArmor / SELinux) | Mandatory access control on top of everything else | Module 13 |
| no_new_privs | Whether it can ever gain privilege via setuid | Module 13 |
Every one is independent and optional. Leave out the namespaces and you have a chroot. Leave out the cgroups and you have a container that can consume the whole machine. Leave out the last four and you have a container running as real root with every capability — which, for years, was the default.
People describe a container as a thing, like a crate. It is not. It is closer to a recipe that a kitchen follows when seating a guest: give them their own menu (mount namespace), their own table numbering (PID), their own phone line (network), a spending cap (cgroups), a restricted set of things they may ask staff to do (capabilities and seccomp), and a doorway into their own private dining room (pivot_root).
Skip any step and you still have a guest, just a less contained one. Skip all of them and you have an ordinary diner. There is no crate anywhere — which is why you cannot ask the kitchen "how many crates are there", and why the kernel cannot tell you how many containers are running.
chroot versus pivot_root in these terms: chroot is telling the guest to stay in the private room, with the door left open behind them and the corridor still there. pivot_root is rebuilding the building around them so the corridor no longer exists. The first is a convention; the second is a fact.
Where the analogy stops working. A guest knows they are in a restaurant. A containerised process usually cannot tell, and "am I in a container?" is answered by heuristics — looking at /proc/1/cgroup, or for /.dockerenv — rather than by any reliable kernel interface.
🧪 Exercise C1.1 — Build a container from scratch, in three commands
# --- 1. A root filesystem. One static binary is enough. ---
command -v busybox >/dev/null || sudo apt-get install -y busybox-static
ROOT=/tmp/mycontainer
sudo rm -rf $ROOT
sudo mkdir -p $ROOT/bin $ROOT/proc $ROOT/sys $ROOT/tmp $ROOT/dev
# Without a /dev/null, any `2>/dev/null` redirection inside fails and the
# command never runs at all - a very confusing way for an exercise to break.
sudo mknod -m 666 $ROOT/dev/null c 1 3
sudo cp "$(command -v busybox)" $ROOT/bin/
for c in sh ls ps mount hostname id ip cat; do sudo ln -sf busybox $ROOT/bin/$c; done
echo "rootfs is $(sudo du -sh $ROOT | cut -f1), containing $(sudo find $ROOT -type f | wc -l) real file"
# --- 2. A cgroup with limits ---
sudo mkdir -p /sys/fs/cgroup/handmade
echo "+cpu +memory +pids" | sudo tee /sys/fs/cgroup/cgroup.subtree_control >/dev/null
echo "50000 100000" | sudo tee /sys/fs/cgroup/handmade/cpu.max >/dev/null
echo "128M" | sudo tee /sys/fs/cgroup/handmade/memory.max >/dev/null
echo "50" | sudo tee /sys/fs/cgroup/handmade/pids.max >/dev/null
# --- 3. Namespaces + that cgroup + that root = a container ---
sudo unshare --mount --uts --ipc --net --pid --fork --propagation private \
sh -c 'echo $$ > /sys/fs/cgroup/handmade/cgroup.procs
exec chroot '"$ROOT"' /bin/sh -c "
/bin/mount -t proc proc /proc
/bin/hostname handmade
echo \"hostname : \$(/bin/hostname)\"
echo \"PID : \$\$\"
echo \"user : \$(/bin/id -u)\"
echo \"--- processes it can see ---\"; /bin/ps
echo \"--- filesystem it can see ---\"; /bin/ls /
echo \"--- network it can see ---\"; /bin/ip -o link
"'
# Clean up
sudo rmdir /sys/fs/cgroup/handmade 2>/dev/null
sudo rm -rf $ROOT✅ Expected result — click to reveal
rootfs is 2.1M, containing 1 real file
hostname : handmade
PID : 1
user : 0
--- processes it can see ---
PID USER COMMAND
1 0 /bin/sh -c ...
6 0 /bin/ps
--- filesystem it can see ---
bin dev proc sys tmp
--- network it can see ---
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN qlen 1000\ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00What to read out of this.
A 2.1 MB root filesystem containing exactly one real file. Everything else is a symlink to busybox. That is the whole image — no distribution, no package manager, no libraries, because busybox-static is statically linked. It is worth seeing how little a container actually needs.
Inside: hostname handmade (UTS namespace), PID 1 (PID namespace), two visible processes out of the machine's 243 (PID namespace plus a fresh /proc), four directories at / (chroot), and a network with nothing but lo (network namespace). Every one of those is a separate mechanism, and you asked for each of them by name on the unshare line.
user: 0 is worth pausing on. This container is running as real root, with every capability the host has, because we did not drop any. It is contained in what it can see and what it can use, and not at all in what it may do. That is a container with the Module 13 half missing, and it is why namespaces and cgroups alone are not a security boundary.
The cgroup limits are applied, and you can prove it from another terminal while it runs: cat /sys/fs/cgroup/handmade/cgroup.procs lists the container's host PID, and cpu.max reads 50000 100000.
One thing to notice about what we did not do: no image format, no registry, no daemon, no runtime. Those exist to make this repeatable, distributable and safe — but the isolation itself is these three commands. Docker did not invent any of this; it packaged it.
If unshare fails with Operation not permitted, you dropped the sudo, or unprivileged user namespaces are restricted — see Section A4.
C2 · overlayfs — where the image layers come from
The other half of a container image is the filesystem, and it is one kernel feature: overlayfs stacks directories so that they appear as one.
| Directory | Role | In container terms |
| lowerdir | Read-only. Several may be stacked | The image layers |
| upperdir | Writable. Every change lands here | The container's writable layer |
| workdir | Empty scratch space on the same filesystem as upperdir | Internal staging, never touched by you |
| merged | The mount point where the result appears | The container's / |
Two behaviours do all the work. Copy-up: the first write to a file that exists only in a lower layer copies the whole file into the upper layer first, then modifies the copy. Whiteouts: deleting a lower-layer file cannot actually delete it, so overlayfs creates a special marker in the upper layer that hides it.
That is why a hundred containers from one image cost one copy of the image plus each container's changes — the lower layers are shared, read-only, and identical.
The printed map is the image: fixed, shared, and identical for everyone who has a copy. Over it you lay a sheet of clear tracing paper, and you draw only on the tracing paper.
Looking down, you see one map — printed features where you have not drawn, your annotations where you have. That is the merged view, and it is what the container sees as /.
To change a printed road, you cannot edit the print. You trace the whole road onto your sheet first and then alter your copy. That is copy-up, and it is why altering one small thing on a very large feature costs you tracing the whole feature.
To delete a printed building, you cannot remove ink from the map. You paint a white patch over it. The building is still printed underneath; it is simply hidden. That is a whiteout — and it is why deleting a secret from a Dockerfile layer does not remove it from the image.
And the reason it is efficient: one printed map, a hundred sheets of tracing paper. The expensive part is shared.
Where the analogy stops working. Tracing paper is transparent everywhere. Overlayfs merges per file, not per pixel, so a file is either yours or the layer's beneath — there is no partial blending.
🧪 Exercise C2.1 — Watch copy-up and a whiteout happen
rm -rf /tmp/ovl; mkdir -p /tmp/ovl/{lower,upper,work,merged}
# The "image": two files in a read-only layer
echo "from the image" > /tmp/ovl/lower/config.txt
echo "also from the image" > /tmp/ovl/lower/readme.txt
dd if=/dev/zero of=/tmp/ovl/lower/big.dat bs=1M count=64 status=none
sudo mount -t overlay overlay \
-o lowerdir=/tmp/ovl/lower,upperdir=/tmp/ovl/upper,workdir=/tmp/ovl/work \
/tmp/ovl/merged
echo "--- merged view ---"
ls /tmp/ovl/merged
echo "upper layer is currently: $(ls -A /tmp/ovl/upper | wc -l) entries, $(du -sh /tmp/ovl/upper | cut -f1)"
# --- Copy-up: change one byte of a 64 MiB file ---
echo "--- writing ONE byte into big.dat ---"
printf 'X' | dd of=/tmp/ovl/merged/big.dat bs=1 count=1 conv=notrunc status=none
du -sh /tmp/ovl/upper
ls -l /tmp/ovl/upper/big.dat
# --- The lower layer is untouched ---
echo "lower big.dat still: $(du -sh /tmp/ovl/lower/big.dat | cut -f1)"
echo "changed config in merged:"
echo "changed by the container" > /tmp/ovl/merged/config.txt
echo " merged: $(cat /tmp/ovl/merged/config.txt)"
echo " lower : $(cat /tmp/ovl/lower/config.txt)"
# --- Whiteout: delete a file that only exists below ---
rm /tmp/ovl/merged/readme.txt
echo "--- after deleting readme.txt ---"
echo "merged: $(ls /tmp/ovl/merged)"
echo "lower : $(ls /tmp/ovl/lower)"
ls -l /tmp/ovl/upper/readme.txt
sudo umount /tmp/ovl/merged; rm -rf /tmp/ovl✅ Expected result — click to reveal
--- merged view ---
big.dat config.txt readme.txt
upper layer is currently: 0 entries, 4.0K
--- writing ONE byte into big.dat ---
65M /tmp/ovl/upper
-rw-rw-r-- 1 zaeem zaeem 67108864 Aug 22 10:14 /tmp/ovl/upper/big.dat
lower big.dat still: 64M
changed config in merged:
merged: changed by the container
lower : from the image
--- after deleting readme.txt ---
merged: big.dat config.txt
lower : big.dat config.txt readme.txt
c--------- 1 root root 0, 0 Aug 22 10:14 /tmp/ovl/upper/readme.txtWhat to read out of this.
The merged view shows all three files while the upper layer is empty. Nothing was copied to create the container's filesystem — that is why starting a container is instant regardless of image size.
Then the headline. Writing one byte into big.dat made the upper layer jump to 65 MB, and ls -l shows a full-size 67108864-byte file there — still owned by your ordinary user, because copy-up preserves the lower file's owner and mode. Copy-up copies the entire file before modifying it. One byte of writing, 64 MiB of I/O, and none of it visible in the application's own metrics. This is the single most common cause of "the container is fast in test and slow in production" for anything that rewrites large files in place, and the fix is a volume rather than the writable layer.
The lower layer is untouched throughout: big.dat is still 64M there, and config.txt still says from the image while the merged view says changed by the container. The image is genuinely immutable, which is what makes it safe to share one copy between a hundred containers.
Finally the whiteout, which is the strangest-looking thing in this module. readme.txt is gone from the merged view and still present in the lower layer, and in the upper layer there is now:
c--------- 1 root root 0, 0 ... readme.txtA character device with major and minor both 0. That is overlayfs's marker for "this name is deleted", because it cannot remove a file from a read-only layer. It is not a file, it has no contents, and its only job is to hide what is beneath.
This is why deleting a secret in a later Dockerfile layer does not remove it from the image. The RM produces a whiteout; the file is still sitting in the earlier layer, and anyone who pulls the image can read it. The only fix is not to add it in the first place — multi-stage builds, or build secrets.
If the mount fails with Invalid argument, upperdir and workdir must be on the same filesystem and workdir must be empty.
🎯 Interview questions — What a container is
Q. How is a container different from a virtual machine?
A virtual machine runs its own kernel on virtualised hardware; the hypervisor gives it emulated CPUs, memory and devices, and the boundary is enforced by the CPU's virtualisation support. A container is an ordinary process on the host's kernel that has been given a restricted view and a resource budget: namespaces for what it sees, cgroups for what it uses, a different root filesystem, and — if configured — reduced capabilities, a seccomp filter and an LSM policy.
The consequences follow directly. Containers start in milliseconds because nothing boots; they share the kernel's page cache and one copy of each image layer, so density is far higher; and they cannot run a different kernel or a different operating system.
The details that separate candidates: stating that there is no container object in the kernel at all — no container ID, no container_create() — so a container is a convention assembled from seven independent mechanisms, any of which can be omitted. That framing answers the security question in the same breath: the isolation is only as strong as the pieces you actually applied, whereas a VM boundary is one mechanism that is either there or not.
Q. Are containers really isolated, the way a VM is?
No. They share the host kernel, so a kernel vulnerability is reachable from inside every container on the machine, and a container escape lands the attacker on the host. A VM's boundary is enforced by hardware virtualisation and its attack surface is the hypervisor, which is very much smaller than the kernel's syscall interface.
Containers are also isolated only in the respects that were configured. Namespaces control visibility, not privilege: a container running as root with all capabilities and no seccomp profile is contained in what it can see and not in what it can do.
The details that separate candidates: naming the specific weak points rather than gesturing at "shared kernel". A privileged container, a mounted Docker socket, --pid=host, a hostPath mount of /, and unprivileged user namespaces being reachable are the practical escape routes, and every one of them is a configuration choice rather than a kernel flaw. The right answer to "how do I get VM-grade isolation" is then honest: use a VM — Kata Containers, Firecracker, gVisor's syscall interception — rather than trying to harden a shared kernel into one.
Q. What actually makes a container a container? Name the mechanisms.
Seven, all independent. Namespaces — which of the eight it gets decides what it can see. cgroups — CPU, memory, I/O and PID limits. A new root filesystem, normally an overlayfs stack of image layers plus a writable upper, entered with pivot_root rather than chroot. Capabilities dropped from root's full set. seccomp, restricting which system calls it may make at all. An LSM — AppArmor or SELinux — for mandatory access control. And no_new_privs, so it cannot regain privilege through a setuid binary.
Any of these can be left out, and different runtimes and configurations do leave different ones out.
The details that separate candidates: knowing why runtimes use pivot_root rather than chroot — chroot(2)'s own manual says it "is not intended to be used for any kind of security purpose", because the old root remains reachable through the mount tree and can be walked back to; pivot_root replaces the root mount and the old one is then unmounted, leaving nothing to escape to. Being able to say that you have built one by hand with unshare, a cgroup and a busybox rootfs is worth more than any amount of Docker vocabulary.
🔍 Part D · Debugging containers from the host
D1 · The host can see everything
Namespaces are one-way. A container cannot see the host; the host sees everything. That asymmetry is what makes container debugging tractable when the image has no shell, the runtime is wedged, or kubectl exec refuses to connect.
Four techniques, in the order you will need them:
| Technique | What it gets you |
| cat /proc/PID/cgroup | Which container this process belongs to |
| ls /proc/PID/root/ | The container's filesystem, from the host — no exec required |
| nsenter -t PID -n <cmd> | Run a host binary inside the container's network namespace |
| cat /proc/PID/mountinfo | Every mount the container has, including bind mounts from the host |
The second one deserves emphasis. /proc/PID/root is a symlink to that process's root directory, so from the host you can read, copy and inspect any file inside any container with ordinary tools, without entering it and without the container having any tooling of its own. For a distroless image containing one static binary, this is the only way.
The team from Section A1 has its own floor, its own directory page and its own phone system. From inside, the rest of the building does not exist.
The building manager has a master key and the master directory. They can walk onto that floor, read every document in every cabinet, and listen on the team's phone line — and they do not need the team's permission or cooperation to do it.
More usefully, they do not need to bring the team's tools. The team may have no photocopier; the manager brings one from downstairs. That is nsenter: the host's binaries, running in the container's world.
And the master directory lists which floor every employee is on, which is /proc/PID/cgroup — the way back from "someone is causing a problem" to "which team they belong to".
Where the analogy stops working. A manager walking onto the floor is visible. Reading /proc/PID/root from the host is completely invisible to the container, which is exactly why it works when the container is unresponsive.
🧪 Exercise D1.1 — Inspect a container that has no tools
# Recreate the minimal container from C1 and leave it running
command -v busybox >/dev/null || sudo apt-get install -y busybox-static
ROOT=/tmp/mycontainer
sudo rm -rf $ROOT; sudo mkdir -p $ROOT/bin $ROOT/proc $ROOT/tmp
sudo cp "$(command -v busybox)" $ROOT/bin/
for c in sh sleep hostname; do sudo ln -sf busybox $ROOT/bin/$c; done
echo "a file only the container should have" | sudo tee $ROOT/tmp/secret.txt >/dev/null
sudo unshare --mount --uts --net --pid --fork --propagation private \
chroot $ROOT /bin/sh -c '/bin/hostname isolated; exec /bin/sleep 120' &
sleep 2
# 1. Find it from the host. It has no ps, no ss, no ip, no shell to exec into.
P=$(pgrep -x sleep | tail -1)
echo "container process is host PID $P"
echo "its in-container PID: $(awk '/^NSpid/ {print $NF}' /proc/$P/status)"
# 2. Which namespaces does it have that we do not?
for n in uts pid net mnt ipc; do
printf '%-5s host=%-22s container=%s\n' "$n" \
"$(readlink /proc/self/ns/$n)" "$(sudo readlink /proc/$P/ns/$n)"
done
# 3. Read its filesystem WITHOUT entering it
echo "--- the container's / as seen from the host ---"
sudo ls /proc/$P/root/
sudo cat /proc/$P/root/tmp/secret.txt
# 4. Run host tools inside its network namespace
echo "--- its network, using the HOST's ip and ss ---"
sudo nsenter -t $P -n ip -br addr
sudo nsenter -t $P -n ss -tln | tail -n +2 | wc -l
# 5. Its mounts, including anything bind-mounted in from the host
echo "--- its mounts ---"
sudo findmnt -N $P -o TARGET,SOURCE,FSTYPE | head -5
sudo kill $P 2>/dev/null; sudo rm -rf $ROOT✅ Expected result — click to reveal
container process is host PID 12408
its in-container PID: 1
uts host=uts:[4026531838] container=uts:[4026532301]
pid host=pid:[4026531836] container=pid:[4026532303]
net host=net:[4026531840] container=net:[4026532305]
mnt host=mnt:[4026531841] container=mnt:[4026532300]
ipc host=ipc:[4026531839] container=ipc:[4026531839]
--- the container's / as seen from the host ---
bin proc tmp
a file only the container should have
--- its network, using the HOST's ip and ss ---
lo DOWN
0
--- its mounts ---
TARGET SOURCE FSTYPE
/ /dev/vda1 ext4
|-/proc proc proc
|-/sys sysfs sysfs
| |-/sys/kernel/security securityfs securityfsWhat to read out of this.
NSpid gives 12408 on the host, 1 inside. That mapping is the starting point for everything else, and it comes from a plain file.
The namespace comparison is the clearest picture of what "containerised" means here. Four inodes differ — uts, pid, net, mnt — and ipc is identical to the host's, because we did not pass --ipc this time. That is the actual isolation profile, read directly rather than inferred from a runtime's configuration, and it is how you answer "is this container sharing the host's network?" with certainty.
Then the technique that matters most. ls /proc/$P/root/ listed the container's filesystem and cat read a file out of it — from the host, with host binaries, with the container completely unaware. This container has no shell you could exec into and no tooling whatsoever, and none of that was an obstacle. For distroless and scratch images this is not a convenience, it is the only option.
nsenter -t $P -n ip -br addr ran the host's ip inside the container's network namespace: one interface, lo, shown DOWN and with no address, because nothing ever brought it up — and zero listening sockets. The container contains no ip binary. Substitute tcpdump, curl or dig and the same thing works — which covers most of what you ever want to do to a container's networking.
findmnt -N $P reads /proc/$P/mountinfo and prints that process's whole mount tree. Reading mountinfo by hand is possible but fiddly: field 5 is the mount point, and the source is the second field after the - separator, whose position varies from line to line — which is exactly why findmnt exists. Counter-intuitive: the mounts here look like the host's, because --mount gives the new namespace a copy of the host mount table and chroot then changes the process's root without touching that table — a reminder that chroot is not pivot_root. On a real container the root line names the overlayfs, and any bind mount from the host appears here too — which is how you find out what a container has been given access to without reading anyone's YAML.
If pgrep -x sleep picks the wrong process, match on the full command with pgrep -f 'sleep 120'.
🎯 Interview questions — Debugging
Q. A container is misbehaving, kubectl exec will not connect, and the image has no shell. How do you investigate?
From the node, using kernel interfaces rather than the runtime. Find the container's host PID — from crictl, or by grepping /proc/*/cgroup for the pod UID. Then: ls /proc/PID/root/ gives me the container's whole filesystem from the host with ordinary tools, no exec needed. nsenter -t PID -n <cmd> runs the host's ss, ip, tcpdump or curl inside the container's network namespace, so the image containing none of them is irrelevant. /proc/PID/mountinfo shows every mount it has, including bind mounts from the host. And /proc/PID/cgroup plus that cgroup's memory.events and cpu.stat gives me its limits and how often it has hit them.
The details that separate candidates: explaining why this works — namespaces are one-way, so the container cannot see the host but the host sees everything, and none of these techniques need the container or the runtime to cooperate. That is precisely the situation where kubectl exec has already failed. Adding that nsenter -p forks, because setns() into a PID namespace only affects children, shows first-hand use rather than a memorised list.
Q. A pod is being OOMKilled. Trace it from the node.
kubectl says exit code 137, which from Module 09 is 128 + 9, a SIGKILL. To find out whether it was the container's own limit or the node running out, I go to the node's kernel log: dmesg -T | grep -i oom and read constraint=. CONSTRAINT_MEMCG with an oom_memcg= path is the container's own memory.max; global_oom means the node itself ran short and the container may have been an innocent victim.
Then the cgroup: memory.events for oom_kill and, just as important, max — a high max count with no kills is a container being aggressively reclaimed right at its limit, which is slow and invisible in every utilisation graph. memory.stat splits anon from file, which tells me whether it was the workload's own memory or its page cache that pushed it over.
The details that separate candidates: two. The OOM record is written by the host kernel, so it never appears in container logs — which is why a pod so often restarts with an empty log and no explanation. And page cache is charged to the cgroup, so a container doing heavy file I/O can be killed over memory that was entirely reclaimable, which changes the fix from "raise the limit" to "stop caching that". Raising the limit without knowing which of those it was wastes capacity on every replica in the fleet.
🏁 Part E · Practice, docs and self-check
E1 · Production practice
| Symptom in production | What is really happening | What to run | The fix |
| Every container takes the full grace period to stop | PID 1 has no SIGTERM handler; signals to PID 1 have no default action | Check the entrypoint form; is PID 1 a shell? | Exec-form entrypoint, or --init so a real init is PID 1 |
| Zombies accumulating inside a container | PID 1 is an application that never reaps orphans | ps inside; look for state Z | Same fix — a proper init as PID 1 |
| Volume mount shows as owned by nobody | An unmapped UID rendering as overflowuid 65534 | cat /proc/PID/uid_map; id inside and out | Fix the UID mapping. chmod cannot help |
| Rootless Podman fails on Ubuntu 24.04 | Unprivileged user namespaces restricted by AppArmor | sysctl kernel.apparmor_restrict_unprivileged_userns | Set it deliberately, fleet-wide, and export it as a metric |
| A fork bomb takes down the whole node | No pids.max; CPU and memory limits do not contain it | cat /sys/fs/cgroup/<path>/pids.max | Set pids.max on everything; --pids-limit for Docker |
| Container slow, CPU usage well under its limit | CFS throttling — quota exhausted early in each period | nr_throttled in cpu.stat | Fewer worker threads, or a higher quota |
| ss -tlnp on the host shows none of the containers' ports | Ports are per network namespace | nsenter -t PID -n ss -tlnp | Nothing to fix — look in the right namespace |
| Container fast in test, crawls in production writing files | overlayfs copy-up: one byte written copies the whole file | du -sh on the container's upper layer | Put write-heavy paths on a volume |
| A secret was "removed in a later layer" and is still in the image | Deletion in overlayfs is a whiteout; the bytes remain below | Extract and inspect every layer, not the merged view | Never add it — multi-stage builds or build secrets |
| Two containers on one host see each other's processes | The PID namespace was shared, deliberately or by a flag | Compare /proc/PID/ns/pid inodes | Remove --pid=host / shareProcessNamespace |
| Scripts break on one host in the fleet | That host is on cgroup v1 — different files, different semantics | stat -fc %T /sys/fs/cgroup | Standardise on v2; it is the only option from systemd 258 |
E2 · Capstone — four container tickets
Ticket 1. Every deploy takes 10 minutes longer than it should. Rollouts are slow because each pod takes exactly 30 seconds to terminate — the full grace period, every time, across every service. Nothing is logged. What is happening and how do you prove it?
Ticket 2. A pod mounts a shared NFS volume. Inside the container, every file shows as owned by nobody:nogroup and writes fail. On the node, the same files are owned by uid 1500. The team has tried chmod 777 and it changed nothing. Why?
Ticket 3. One node in the cluster becomes completely unresponsive roughly once a week — SSH times out, the kubelet stops reporting, and a reboot fixes it. Node CPU and memory metrics look normal right up to the moment it goes silent. What would you check?
Ticket 4. A security review finds a database password in a published container image. The Dockerfile clearly deletes the file: COPY secrets.env . followed a few lines later by RUN rm secrets.env. The final container filesystem does not contain it. How is it still in the image?
✅ Ticket 1 — worked answer
The shape is the diagnosis: exactly the grace period, every time. A process that is shutting down properly takes a variable, usually short time. A fixed 30 seconds is a timeout expiring, which means the SIGTERM did nothing at all.
Why nothing happened. PID 1 in a PID namespace has no default signal actions. A signal is delivered only if the process installed a handler for it. Kubernetes sends SIGTERM, waits terminationGracePeriodSeconds, then SIGKILLs from the host — which always works, because the host is an ancestor namespace.
Prove it in three steps:
# On the node: what IS PID 1 in the container?
sudo cat /proc/$PID/root/proc/1/cmdline | tr '\0' ' ' # or crictl inspect
# Does it handle SIGTERM? In SigCgt, signal N is bit N-1, so SIGTERM (15) is bit 14 = 0x4000.
grep -E '^Sig(Cgt|Ign)' /proc/$PID/statusSigCgt is the caught-signals bitmask from Module 04, printed as 16 hex digits. Signal N is bit N−1, counting from zero — so SIGTERM is bit 14, which is the mask 0x4000. If SigCgt AND 0x4000 is zero, PID 1 has no SIGTERM handler, and the timeout is inevitable. (Quick check: printf '%d\n' $(( 0x$(awk '/^SigCgt/{print $2}' /proc/$PID/status) & 0x4000 )) — 0 means no handler.)
The usual cause is a shell-form entrypoint — CMD myapp args becomes /bin/sh -c "myapp args", so PID 1 is a shell that neither handles SIGTERM nor forwards it, and the application never hears about the shutdown at all.
Fixes, in order of preference. Use exec form (CMD ["myapp", "args"]) so the application really is PID 1 and can handle signals itself. Or run a tiny init as PID 1 — docker run --init, or tini in the image — which forwards signals and reaps zombies. Do not "fix" it by shortening the grace period: that converts a slow graceful shutdown into a fast ungraceful one, and anything the application needed to flush is now lost.
✅ Ticket 2 — worked answer
nobody is not a permissions problem, so no amount of chmod will touch it. It is a UID mapping problem: 65534 is /proc/sys/kernel/overflowuid, the value the kernel substitutes when a file's owner has no entry in the current user namespace's map.
Run, in order:
cat /proc/$PID/uid_map # what is mapped at all?
cat /proc/$PID/gid_map
id # inside the container
stat -c '%u:%g' /path/to/file # numeric, on the node - names lie hereWhat you will find. The map covers a range that does not include 1500. Everything outside it renders as 65534, and the write fails because the container's effective UID has no rights over a file it cannot even name the owner of.
Always compare numeric IDs, not names. nobody inside and someuser outside can be the same UID displayed through two different /etc/passwd files, and chasing the names wastes an afternoon.
Three real fixes. Run the container with a UID that the mapping covers and that matches the file owner (runAsUser: 1500). Extend the mapping so 1500 is included — the /etc/subuid range for rootless, or the runtime's userns-remap configuration. Or, for NFS specifically, check no_root_squash/all_squash on the export, which does its own identity mapping and produces an almost identical symptom for an entirely different reason.
The one thing that cannot work is chmod. Permissions are checked against a UID; the problem is that the UID does not translate.
✅ Ticket 3 — worked answer
A node that becomes unreachable while CPU and memory look normal, and recovers only on reboot, is the signature of PID exhaustion. A fork bomb — or a much duller runaway loop spawning subprocesses — consumes the host's global PID space, which every process on the machine shares. Once it is full, nothing can fork(): not SSH, not the kubelet, not systemd's recovery paths. CPU is throttled quite happily and memory is barely touched, so both graphs stay flat until the node simply stops answering.
Check, before it happens again:
cat /proc/sys/kernel/pid_max
ls /proc | grep -c '^[0-9]' # processes right now
for c in /sys/fs/cgroup/kubepods.slice/*/*/; do
printf '%8s / %-8s %s\n' "$(cat $c/pids.current)" "$(cat $c/pids.max)" "$c"
done | sort -rn | headWhat confirms it. Any cgroup with pids.max reading max, and a pids.current that grows over hours or days. A container with no PID limit is one bug away from taking the node with it.
The fix is one file. Set pids.max on every workload — Kubernetes has a per-pod default you should verify is actually configured on your nodes, and plain docker run sets none unless you pass --pids-limit. It costs nothing and it is the only thing that contains this: CPU and memory limits are irrelevant to a fork bomb.
And for cleanup, echo 1 > cgroup.kill on the offending cgroup rather than a loop over cgroup.procs — the loop races against a process forking faster than you can read PIDs and may never converge.
Monitoring to add: pids.current / pids.max per container, and the node's total process count against pid_max. Both are cheap, and both rise steadily before the node dies — which is more warning than any other metric gives you here.
✅ Ticket 4 — worked answer
Because rm in a container image does not delete anything — it creates a whiteout.
Each Dockerfile instruction produces a layer, and layers are stacked read-only with overlayfs. COPY secrets.env . wrote the file into layer n. RUN rm secrets.env could not modify layer n, because it is read-only; instead it created in layer n+1 a whiteout whose only job is to hide that name in the merged view. Its form depends on where you look: on the running container's overlayfs it is a character device with major and minor 0; inside the distributed image layer it is a zero-length file named .wh.secrets.env sitting next to where the original was.
The file is still there. Anyone who pulls the image can extract layer n and read it:
docker save myimage:tag -o img.tar
mkdir -p /tmp/x && tar -xf img.tar -C /tmp/x
# then unpack each layer.tar and look - the merged filesystem is irrelevant
for l in /tmp/x/*/layer.tar; do tar -tf "$l" | grep -iE 'secret|\.wh\.' && echo " ^ in $l"; done
# the real file appears in one layer; a '.wh.secrets.env' entry appears in a later oneThis is exactly why "we removed it in a later step" is not a remediation. The same applies to RUN wget ... && rm, to .env files copied and deleted, and to any credential that appears in an intermediate layer at all.
Real fixes. A multi-stage build, where the secret exists only in a build stage that is never published. Build secrets (RUN --mount=type=secret), which mount the value for one command and never write it to a layer. Or fetch the credential at runtime from a secret store rather than baking it in.
And treat the credential as compromised. It has been in a published image; rotate it. Removing the image from the registry does not un-pull it from every machine that already has it.
E3 · Documentation reference
| Topic | Where to read it | Why this one |
| All eight namespaces | namespaces(7) | Start here. Note its version table is about /proc files, not introduction dates |
| Real introduction versions | clone(2) | The CLONE_NEW* table is the authoritative one |
| PID namespaces and PID 1 | pid_namespaces(7) | The signal rules and what happens when PID 1 dies |
| User namespaces | user_namespaces(7) · subuid(5) | UID mapping, and the rootless-container foundation |
| Mount namespaces and propagation | mount_namespaces(7) | Shared / private / slave — the source of most surprises |
| Network namespaces | network_namespaces(7) | Short, and explains why a new one has nothing in it |
| The other four | uts_namespaces(7) · ipc_namespaces(7) · cgroup_namespaces(7) · time_namespaces(7) | Each is a page or two |
| Creating and joining | unshare(2) · unshare(1) · setns(2) · nsenter(1) | The command-line pair is what you will actually use |
| Listing them | lsns(8) | Note it cannot see namespaces kept alive only by a bind mount |
| Changing root properly | pivot_root(2) · chroot(2) | chroot(2) says outright it is not a security mechanism |
| cgroup v2, all of it | Control Group v2 | Every file, every rule. Long, and worth it |
| cgroups overview | cgroups(7) | The v1-versus-v2 comparison in one page |
| Limits via systemd | systemd.resource-control(5) | MemoryMax=, CPUQuota=, TasksMax= and the files they set |
| Seeing the tree | systemd-cgls(1) · systemd-cgtop(1) | cgtop is top grouped by workload |
| Image layers | Overlay Filesystem | Copy-up and whiteouts, from the source |
E4 · Self-assessment
Answer these out loud before moving to Module 13. The section to reread is named after each.
- What does a namespace isolate, and what does it deliberately not? (A1)
- Name the eight namespaces and what each partitions. (A1)
- How do you tell whether two processes are in the same namespace? (A1)
- Why does ps inside a new PID namespace show every process on the machine? (A2)
- What happens when you kill -KILL 1 from inside a container, and why does it work from the host? (A2)
- What happens to a PID namespace when its PID 1 exits? (A2)
- Why does a fresh network namespace have no connectivity at all? (A3)
- How can a hundred containers all listen on port 80? (A3)
- What does unshare -U -r actually do to your UID, and what can you not do with it? (A4)
- Why does a bind-mounted directory sometimes show as owned by nobody? (A4)
- What is the difference between cgroup v1 and v2, in one sentence? (B1)
- What is cgroup.subtree_control for, and what does EBUSY on cgroup.procs mean? (B1)
- Which limit contains a fork bomb, and why do the others not? (B2)
- Why is cgroup.kill better than looping over cgroup.procs? (B2)
- Given a host PID, how do you find its container's memory limit? (B3)
- List the seven mechanisms that make a container, and say which are optional. (C1)
- Why do runtimes use pivot_root rather than chroot? (C1)
- What is copy-up, and when does it hurt? (C2)
- Why does deleting a file in a later image layer not remove it from the image? (C2)
- kubectl exec fails and the image has no shell. Name three things you can still do from the node. (D1)
E5 · Sources
Manual pages
· namespaces(7) · clone(2) · unshare(2) · unshare(1) · setns(2) · nsenter(1) · lsns(8)
· pid_namespaces(7) · user_namespaces(7) · mount_namespaces(7) · network_namespaces(7) · uts_namespaces(7) · ipc_namespaces(7) · cgroup_namespaces(7) · time_namespaces(7)
· pivot_root(2) · pivot_root(8) · chroot(2) · mount(8)
· cgroups(7) · systemd.resource-control(5) · systemd-cgls(1) · systemd-cgtop(1) · proc(5)
Kernel documentation
· Control Group v2 · Overlay Filesystem · User namespaces and resource control
Distribution behaviour
· Ubuntu 24.04's kernel.apparmor_restrict_unprivileged_userns default, and the systemd 256/258 removal of cgroup v1, are distribution and project decisions rather than kernel documentation — check your own machine with sysctl and systemctl --version rather than trusting any write-up, including this one.