🔐Module 13 — Isolation: Capabilities & seccomp

40
cap_chown
cap_dac_override
cap_dac_read_search
cap_fowner
cap_fsetid
41

CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	000001ffffffffff
CapAmb:	0000000000000000
NoNewPrivs:	0
Seccomp:	0

CapPrm:	000001ffffffffff
CapEff:	000001ffffffffff
CapBnd:	000001ffffffffff

0x0000000000003000=cap_net_admin,cap_net_raw
41

/usr/bin/ping cap_net_raw=ep

/usr/bin/fusermount3
/usr/bin/sudo
/usr/bin/chfn
/usr/bin/chsh
/usr/bin/gpasswd
/usr/bin/mount

What to read out of this.

cap_last_cap is 40, and the name list has 41 entries, because they are numbered from zero. Read that file rather than memorising a number — it is smaller on an older kernel, and it is the only honest answer to "how many capabilities are there?"

Your ordinary shell holds nothing. CapPrm and CapEff are all zeros. Everything you can do, you can do because of file ownership and permission bits, not privilege.

CapBnd is the exception, and it is the field people misread. It is 000001ffffffffff — all 41 bits — in an unprivileged shell. The bounding set is not a grant. It is a ceiling: the set of capabilities this process, or anything it ever execs, is allowed to acquire. Holding none while being permitted all is the normal state of every login shell on the machine. Line 5 confirms the arithmetic: 000001ffffffffff decodes to 41 names.

The root process holds all three fields full, which is exactly what "root" means now — a UID that comes with the complete key ring.

0x3000 is bits 12 and 13, which is cap_net_admin,cap_net_raw. Get comfortable with capsh --decode; every capability field you will ever see in /proc, in a Kubernetes audit log or in a runtime's debug output is a 16-digit hex mask.

The last two commands are the same question asked of two eras. getcap -r finds programs carrying a specific privilege in their file metadata — on Ubuntu that is usually just ping, holding cap_net_raw and nothing more. find -perm -4000 finds the old answer: programs that become fully root. sudo, mount, passwd and su are all still setuid, and each one is an entire root shell if it has a bug. That contrast is the whole argument for capabilities, in two commands.

If getcap -r prints nothing at all, this machine simply has no file capabilities set — common in containers and minimal images. Nothing is wrong.

--- root with the full key ring ---
-rw-rw-r-- 1 zaeem zaeem 0 Aug 22 10:34 /tmp/ownme
--- root with CAP_CHOWN removed ---
my uid is: 0
chown: changing ownership of '/tmp/ownme': Operation not permitted
chown exit: 1
-rw-rw-r-- 1 zaeem zaeem 0 Aug 22 10:34 /tmp/ownme
--- root without CAP_DAC_OVERRIDE, reading a 0600 file it does not own ---
cat: /tmp/priv: Permission denied
cat exit: 1

What to read out of this.

Read the second block again: my uid is: 0 and Operation not permitted on the same screen. That combination is supposed to be impossible under the mental model most people carry, and it is the single most useful thing in this module. The process is root. id -u says so. It cannot change a file's owner, because CAP_CHOWN is not on its ring and UID 0 is not what the kernel checks.

The third block is the same lesson through a different door. Root "can read any file" only because root normally holds CAP_DAC_OVERRIDE (ignore the permission bits) and CAP_DAC_READ_SEARCH (read and traverse regardless). Remove both and root gets Permission denied from a 0600 file — exactly like any other user.

Note which error you get, because the two are diagnostic. Operation not permitted (EPERM) is a capability failure: the kernel checked a privilege and you did not have it. Permission denied (EACCES) is usually an ordinary permission-bit failure. When a container "cannot do something as root", EPERM points at a missing capability or seccomp; EACCES points at file modes, ownership or an LSM. You will use that split again in Part D.

capsh --drop removes from the bounding set, and that is a one-way door. There is no --add. A capability removed from the bounding set cannot be regained by that process or by anything it ever execs — which is precisely why container runtimes use it, and why the shrinking is safe to do early in a startup script.

ext4   rw,relatime
getcap printed nothing above, and exited 0
unable to set CAP_SETFCAP effective capability: Operation not permitted
unprivileged setcap exit: 1
/home/zaeem/capdemo cap_net_bind_service=ep

getfattr: Removing leading '/' from absolute path names
# file: home/zaeem/capdemo
security.capability=0sAQAAAgAEAAAAAAAAAAAAAAAAAAA=

getfattr: Removing leading '/' from absolute path names
# file: home/zaeem/capdemo
security.capability=0x0100000200040000000000000000000000000000

Traceback (most recent call last):
  File "<string>", line 1, in <module>
PermissionError: [Errno 13] Permission denied
bound port 80

What to read out of this.

getcap printing nothing and exiting 0 is the normal "no capabilities" answer. It is not an error, and scripts that test $? here will conclude the wrong thing. Test for empty output instead.

The first failure is the important one. Setting a file capability requires CAP_SETFCAP, so an ordinary user cannot do it — and the error says so by name: unable to set CAP_SETFCAP effective capability. If a user could grant capabilities to files, capabilities would not be a security mechanism at all.

Then the payoff, which is the point of the whole feature. The same Python interpreter, run by the same unprivileged user, on the same machine: the packaged one gets Permission denied on port 80, and your copy binds it. No sudo, no setuid, no root process anywhere. One capability, on one file. That is what web servers should have been doing for twenty years instead of starting as root and dropping privileges afterwards.

Read the hex, because it demystifies the whole thing. 0x01000002... is a small versioned structure, stored little-endian. The first four bytes 01 00 00 02 are 0x02000001 — revision 2, with the effective bit set (the e in =ep). The next four, 00 04 00 00, are 0x00000400 — bit 10 — and capability number 10 is CAP_NET_BIND_SERVICE. The whole "file capability" is twenty bytes of metadata; there is no magic in it.

CapInh:	0000000000000000
CapPrm:	0000000000000400
CapEff:	0000000000000400
CapBnd:	000001ffffffffff
CapAmb:	0000000000000000

CapPrm:	0000000000000000
CapEff:	0000000000000000

CapInh:	0000000000000400
CapPrm:	0000000000000400
CapEff:	0000000000000400
CapAmb:	0000000000000400
bound port 80 with NO file capability

Traceback (most recent call last):
  File "<string>", line 1, in <module>
PermissionError: [Errno 13] Permission denied

What to read out of this.

Block 1 versus block 2 is the entire lesson. The process that ran ~/capdemo holds 0000000000000400 — bit 10, CAP_NET_BIND_SERVICE — in both Permitted and Effective. The grep it launches, one exec later, holds zero. Nothing was revoked and nothing went wrong; this is the default. Privilege attached to a file stays with that file's process and goes no further.

Notice CapInh and CapAmb are both empty in block 1, even though the process is privileged. Being privileged and passing on privilege are separate things.

Block 3 is the ambient set doing its job. setpriv dropped to your unprivileged UID, put CAP_NET_BIND_SERVICE into the inheritable set, raised it into the ambient set, and execed the stock, unmodified /usr/bin/python3. All four fields show 400, and port 80 binds. No setcap, no setuid bit, no root process left running. This is how a modern service manager gives a daemon one privilege — it is exactly what AmbientCapabilities= in a systemd unit does.

CapInh had to be set too: ambient capabilities may only contain capabilities that are in both Permitted and Inheritable. Raising ambient without inheritable silently gets you nothing, and that is the usual reason AmbientCapabilities= "does not work".

Block 4 is the control. Identical command, ambient set left empty, and the bind fails. The difference between the two is four words on a command line and nothing else.

uid inside: 0
CapPrm:	000001ffffffffff
CapEff:	000001ffffffffff
CapBnd:	000001ffffffffff
--- try to read /etc/shadow:
cat: /etc/shadow: Permission denied
cat exit: 1
--- try to change the hostname:
hostname: you must be root to change the host name
hostname exit: 1
--- try to mount a tmpfs:
mount: /mnt: permission denied.
       dmesg(1) may have more information after failed mount system call.
mount exit: 32

What to read out of this.

All 41 capabilities, and three failures in a row. CapPrm, CapEff and CapBnd are 000001ffffffffff — the complete set — and id -u says 0. By the model most people carry, this process owns the machine. It cannot read /etc/shadow, cannot set the hostname, and cannot mount a filesystem.

The reason is that capabilities are scoped to a user namespace, not to the machine. Every one of those 41 is held in the namespace you just created, and that namespace owns nothing: no host files, no host UTS namespace, no host mounts. The kernel checks "do you hold CAP_SYS_ADMIN in the namespace that owns the thing you are touching", and the answer is no, every time.

/etc/shadow is the clearest case. It is owned by real UID 0. Your namespace maps your UID to 0, and real UID 0 is unmapped — so as far as this namespace is concerned the file belongs to a stranger, and CAP_DAC_OVERRIDE here does not apply to it.

The hostname failure names the mechanism out loud: you must be root to change the host name. CAP_SYS_ADMIN would allow it, but only in the UTS namespace you own — and you did not create one. Add --uts and it works instantly.

This is the foundation of rootless containers, and it is worth being able to explain in one sentence: rootless Podman gives a container a convincing UID 0 with a full capability set, and that root is powerless outside the namespaces it was given. Compare with Exercise A1.2, where a real host-root process held all 41 capabilities and could do real damage with any of them.

Because a container is not a boundary — it is a set of restrictions, and capabilities are the ones that switch large parts of the kernel back on.

The three that matter most: CAP_SYS_ADMIN allows mount, so a container holding it can mount a host filesystem or abuse cgroup interfaces to run code on the host. CAP_SYS_MODULE allows loading a kernel module, which is not an escape route so much as being the kernel — there is nothing above it to escape to. CAP_SYS_PTRACE allows attaching to another process, which matters the moment the PID namespace is shared with the host.

Two more deserve naming. CAP_DAC_READ_SEARCH allows open_by_handle_at, the basis of the well-known Shocker escape, and it is not in Docker's default set for that reason. CAP_NET_RAW does not escape at all but allows spoofing on the shared network, which is often just as damaging.

The details that separate candidates: saying that the default 14 capabilities are chosen precisely so that none of them is an escape route on their own, so the interesting question is always what was added, not what a container has. And adding that capabilities are only the first of five independent restrictions — seccomp, the LSM and the device cgroup have to be discussed too, because --privileged removes all four at once and --cap-add=SYS_ADMIN removes only one.

UID 0 is not what the kernel checks. mount requires CAP_SYS_ADMIN, and Docker's default set of 14 capabilities does not include it. The container's CapEff mask has that bit clear, so the check fails regardless of the UID.

How to prove it rather than assert it: grep Cap /proc/1/status inside the container and capsh --decode=<CapEff>. The absent name will be there in black and white.

The details that separate candidates: naming the error code. Operation not permitted is EPERM, which is what a capability check returns; Permission denied is EACCES, which is usually a permission-bit or LSM refusal. Being able to say "EPERM means look at capabilities and seccomp, EACCES means look at file modes, ownership and AppArmor" turns a guess into a triage path. And the right fix is almost never --cap-add=SYS_ADMIN — it is asking why the container is mounting anything, and mounting it from the host side instead.

Four, roughly in the order you should consider them.

Do not bind a low port. Listen on 8443 and let the load balancer, the Kubernetes Service, or a docker -p 443:8443 publish handle the mapping. This is almost always the right answer and needs no privilege at all.

Lower the threshold. sysctl net.ipv4.ip_unprivileged_port_start=443 makes 443 an ordinary port. One line, machine-wide, and it removes the problem rather than working around it — but it applies to everything on that host, so it is a fleet decision, not an app decision.

Ambient capabilities from the service manager. In a systemd unit: AmbientCapabilities=CAP_NET_BIND_SERVICE with User=svc. The process starts unprivileged, holds exactly one capability, and no root process exists at any point.

A file capability. setcap cap_net_bind_service=ep /usr/bin/myapp. It works, but it is attached to a file that a package upgrade will silently replace, and it does nothing on nosuid mounts, NFS, or scripts.

The details that separate candidates: rejecting the fifth option out loud — "start as root and drop privileges after binding" — because that still requires a full-root process to exist, and every privilege-drop bug in UNIX history lives in that pattern. Also noting that AmbientCapabilities= requires the capability to be in the inheritable set as well, which systemd handles for you but setpriv does not.

A thread has five. Permitted is the ceiling of what it may switch on; Effective is what is switched on right now and is the set the kernel actually checks; Bounding is the ceiling across exec and can only ever shrink; Inheritable is a legacy mechanism that only works when the executed file's inheritable set agrees; Ambient (Linux 4.3) is what survives exec of an ordinary file, and is the one that made the feature usable.

A file has three: a permitted set, an inheritable set, and a single effective bit — which is why getcap prints things like cap_net_raw=ep.

The details that separate candidates: explaining why the default is to lose everything on exec — otherwise every privileged program would leak its privilege into anything it launched, starting with /bin/sh. And knowing the two traps: ambient is silently cleared when you exec a setuid binary or a file with its own capabilities, and ambient can only hold capabilities that are already in both Permitted and Inheritable.

Because it stopped being a capability and became a catch-all. The man page's own words are "a vast proportion of existing capability checks are associated with this capability", followed by the instruction "Don't choose CAP_SYS_ADMIN if you can possibly avoid it!"

The measurement people quote comes from Michael Kerrisk's 2012 LWN article: in Linux 3.2, 451 of 1167 capability checks in the source were CAP_SYS_ADMIN — about 38%, later estimated at just over 45% by 5.2. Granting it is therefore not granting one privilege, it is granting an unbounded and growing set.

The details that separate candidates: citing the figure with its version and source rather than as a current fact — nobody recounts it each release, and quoting "38% of all capability checks" as though it were measured yesterday is a tell. Better still, pointing at the fix in progress: CAP_BPF and CAP_PERFMON were carved out of CAP_SYS_ADMIN in Linux 5.8 specifically so observability tooling would stop needing the junk drawer, and CAP_CHECKPOINT_RESTORE followed in 5.9.

Blast radius. A setuid-root binary runs with all 41 capabilities and UID 0, so any bug in it is a full root compromise. A binary with cap_net_raw=ep runs as the calling user with one capability, so the same bug buys an attacker raw sockets and nothing else.

Both are stored on the file — the setuid bit in the mode, capabilities in the security.capability extended attribute — and both are ignored on a nosuid mount.

The details that separate candidates: knowing that capabilities are not a drop-in replacement, and why. They cannot be applied to scripts, because the kernel execs the interpreter. They are lost when a package manager replaces the file. They cannot be stored on NFS or FAT. And a program written for setuid usually calls setuid() to drop privilege, which requires CAP_SETUID — so converting it is real work, not a one-line change. That is why sudo, mount and passwd are still setuid on every distribution.

-rwsr-xr-x 1 root root 43744 Aug 22 10:41 /home/zaeem/idcopy
--- normally: ---
uid=1002(zaeem) gid=1003(zaeem) euid=0(root) groups=1003(zaeem)
--- with no_new_privs set: ---
uid=1002(zaeem) gid=1003(zaeem) groups=1003(zaeem)
--- and it is inherited by children: ---
NoNewPrivs:	1
uid=1002(zaeem) gid=1003(zaeem) groups=1003(zaeem)

What to read out of this.

The whole result is in one word: euid=0. In the first run the setuid bit did its job and the process became effectively root. In the second, the identical binary — same file, same s bit, same ls -l output — ran as a plain user. Nothing was changed on disk. One bit was set on the process before exec, and the kernel declined to apply the setuid bit.

The third block is the property that makes it deployable. setpriv --no-new-privs bash -c … set the bit on bash, and the ~/idcopy that bash launched was still disarmed — NoNewPrivs: 1 is right there in /proc/self/status. You set it once, as early as possible, and every descendant inherits it forever. There is no --no-new-privs=off to undo it, by design.

Where you have already met this without knowing. Docker's --security-opt no-new-privileges and Kubernetes' allowPrivilegeEscalation: false set exactly this bit. That Kubernetes field name causes real confusion in reviews: it does not mean "this pod cannot become privileged" and it does not drop any capability. It means precisely what you just watched — no setuid binary, and no file capability, inside this container can raise privilege beyond what the process already had.

Seccomp:	2
Seccomp_filters:	1

Seccomp:	0
Seccomp_filters:	0

chmod: /tmp: Operation not permitted
exit: 1

chmod: /tmp: Operation not permitted
exit: 1

exit: 159

What to read out of this.

Seccomp: 2 is filter mode, and it is on by default in every Docker container — this is the check Docker's own documentation tells you to run. Seccomp: 0 is none. Seccomp_filters counts how many filters are stacked on the process, and exists since Linux 5.9; a container that has installed its own filter on top of the runtime's will show 2 here, which is a useful thing to notice.

Step 4 is the one to remember. --user 0 --cap-add=ALL is a container running as root with all 41 capabilities, and it gets the identical refusal. Nothing about identity is consulted. seccomp is the only control here that is not made irrelevant by being root, which is exactly why container runtimes turn it on by default and why removing it is such a large decision.

Step 5 shows the two verdict styles side by side. SCMP_ACT_ERRNO produced a normal-looking error message and exit 1 — the program ran to completion and simply failed, which is why a badly-tuned profile makes applications misbehave rather than crash, and why the resulting bug reports are so misleading. SCMP_ACT_KILL_PROCESS printed nothing at all and exited 159, which is 128 + 31, and signal 31 is SIGSYS. A container exiting 159 with no log line is very often a seccomp kill, and that number is worth committing to memory next to 137 (SIGKILL, from Module 09).

If step 1 shows Seccomp: 0 on your machine, the daemon was started with seccomp disabled, or you are on a kernel built without CONFIG_SECCOMP_FILTER. Check grep CONFIG_SECCOMP /boot/config-$(uname -r).

No Docker? The same demonstration works with systemd: sudo systemd-run --pty --wait -p SystemCallFilter='~chmod fchmod fchmodat' -p SystemCallErrorNumber=EPERM /bin/chmod 777 /tmp. The ~ makes it a deny-list, and SystemCallErrorNumber= is the ERRNO-versus-kill switch.

=== docker run 
CapEff:	00000000a80425fb
CapBnd:	00000000a80425fb
NoNewPrivs:	0
Seccomp:	2

=== docker run --privileged
CapEff:	000001ffffffffff
CapBnd:	000001ffffffffff
NoNewPrivs:	0
Seccomp:	0

=== docker run --cap-drop=ALL --security-opt no-new-privileges
CapEff:	0000000000000000
CapBnd:	0000000000000000
NoNewPrivs:	1
Seccomp:	2

0x00000000a80425fb=cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,
cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,
cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap

What to read out of this. Four numbers describe a container's entire privilege posture, and you can read them from any container on any machine without asking the runtime anything.

00000000a80425fb is worth recognising on sight. It decodes to exactly the 14 capabilities Docker grants by default — the list from section A3, confirmed by the kernel rather than by documentation. When you see that value in a /proc dump or an audit log, you are looking at a stock unprivileged container.

The privileged row is the whole argument against --privileged in three lines. CapEff jumps to all 41, and — the part people miss — Seccomp drops from 2 to 0. --privileged does not merely add capabilities; it switches the syscall filter off entirely, along with the AppArmor or SELinux profile and the device cgroup. It is five protections removed by one word — capabilities, seccomp, the LSM profile, the device cgroup and the masked paths — and only one of them is capabilities.

The hardened row is what a well-configured workload looks like: zero capabilities, a bounding set of zero so none can ever be regained, no_new_privs on, and the filter still in place. Note CapBnd is also zero — --cap-drop=ALL shrinks the ceiling, not just the current holding, so no setuid binary or file capability inside the image can put anything back.

Counter-intuitive: --privileged does not remove namespaces. That container still has its own PID, mount, network and UTS namespaces — --pid=host and friends are separate flags. It is "root on the host's kernel with a private view", which is why escaping still takes a deliberate step such as mounting a host block device, rather than simply looking around.

They act at three different layers, which is the point of using more than one.

seccomp filters system calls by number and by raw argument values. It runs before every other check and ignores identity entirely, so it constrains a container that is already root. It cannot see filenames or paths, because it must not follow user-supplied pointers.

AppArmor is path-based mandatory access control: a profile says which files this program may read, write or execute, which capabilities it may use, and what networking it may do. It is what Ubuntu and Debian ship, and profiles are per-executable.

SELinux is label-based: every process and every object carries a label, and policy defines which label may act on which. It is what RHEL, Fedora and openSUSE ship, and it is stronger and considerably harder to author. For containers the important label is the type — a container process gets container_t, and host files it should not touch simply do not permit container_t.

The details that separate candidates: stating that these are independent and additive, and that a container is only as strong as all of them together — then pointing out that --privileged removes all three at once plus the device cgroup and the masked paths — five protections — which is why it is a far bigger step than --cap-add=ALL. Also worth saying: you can only run one major LSM of this kind at a time, so "AppArmor and SELinux" is a distribution choice, not a hardening choice.

Do not start by guessing. Reproduce it under strace -f -e trace=all inside the unconfined container and look for the call the failing code path makes just before it gives up — the filtered one will be present there and absent from the working trace's outcome.

Better, because it works in production: build a copy of the profile with defaultAction set to SCMP_ACT_LOG, run the workload, and read the audit log. Every call the container makes is recorded and none is blocked, so you learn the exact syscall name without an outage.

Then add just that call to the allow-list, redeploy and re-test. Never resolve the ticket by switching seccomp off; that trades a one-line profile change for the loss of the only control that still applies to a root process.

The details that separate candidates: knowing that the failure will not look like a security error. With SCMP_ACT_ERRNO the application sees EPERM from a call that normally succeeds, and reports whatever that leads it to conclude — "cannot create thread", "unsupported filesystem", or a plain hang. Mentioning the clone3 incident as the canonical example, and that ENOSYS versus EPERM as the default answer for unknown calls is what decides whether a library falls back gracefully, is a strong finish.

It sets the no_new_privs bit on the container's processes. From that moment, exec can never grant more privilege than the process already holds: setuid and setgid bits are ignored, file capabilities are ignored, and privilege-raising LSM transitions are refused.

It does not drop any capability, does not stop the container running as UID 0, and is not the same as privileged: false. A container can be root with fourteen capabilities and still have allowPrivilegeEscalation: false.

Verify it rather than trusting the manifest: grep NoNewPrivs /proc/1/status inside the pod should print 1.

The details that separate candidates: knowing that the bit is inherited and cannot be unset, which is why it is safe to set unconditionally and why it is the cheapest line in a securityContext — no capability analysis, no profile authoring. And knowing the one thing it genuinely breaks: an image that relies on a setuid helper, such as an old ping, or sudo in an entrypoint. That is nearly always a packaging mistake worth fixing rather than an exception worth granting.

Because a seccomp filter survives exec, and without no_new_privs an unprivileged user could aim one at a setuid-root program.

The attack is concrete. Install a filter that makes setuid() return success without doing anything, then exec a setuid-root binary whose first act is to drop privilege by calling setuid(). The call appears to succeed, the program believes it is now unprivileged, and it continues to run as root while behaving as though it were not. no_new_privs breaks the chain by making the setuid bit inoperative for that process, so a filter can only ever be aimed at programs the user could already run.

A process holding CAP_SYS_ADMIN may install a filter without the bit, since it could reach that state by other means anyway.

The details that separate candidates: framing it as the general principle rather than one trick — a mechanism that lets you change how the kernel answers a program must not be combinable with a mechanism that raises that program's privilege. no_new_privs is what makes unprivileged sandboxing safe at all, which is why the two shipped together — both landed in Linux 3.5, and the dedicated seccomp() system call, which until then had been a prctl() option, followed in 3.17.

capability,landlock,lockdown,yama,apparmor,bpf

apparmor module is loaded.
73 profiles are loaded.
41 profiles are in enforce mode.
32 profiles are in complain mode.
0 profiles are in prompt mode.
0 profiles are in kill mode.
0 profiles are in unconfined mode.

unconfined

docker-default (enforce)
unconfined

What to read out of this.

/sys/kernel/security/lsm is the one reliable answer to "what is protecting this machine". The exact list differs by distribution and kernel — what matters is whether apparmor or selinux appears in it. On a RHEL-family host you would see selinux there and getenforce would print Enforcing.

Your login shell is unconfined, and that is normal. AppArmor confines named programs, not users. Your shell has no profile, so no policy applies to it.

The last two lines are the point. A stock container is confined by a profile called docker-default, in enforce mode — a restriction you did not ask for and probably did not know was there. It blocks writes to most of /proc and /sys, mounting, and ptrace outside the container.

With --privileged the same container reports unconfined. That single flag did not weaken the profile; it removed it. Together with Seccomp: 0 from Exercise B3.1, you have now watched --privileged switch off two entire independent defences, and you have the commands to prove it in a review.

If aa-status says "command not found", install apparmor-utils — or you are on a distribution that uses SELinux instead, and the getenforce line is your answer.

=== default container ===
crw-rw-rw-    1 root     root        1,   3 Aug 22 10:52 /proc/kcore
sysfs /sys sysfs ro,nosuid,nodev,noexec,relatime 0 0
sh: can't create /sys/kernel/uevent_helper: Read-only file system
write to /sys exit: 1

=== privileged container ===
-r--------    1 root     root        140737486266368 Aug 22 10:52 /proc/kcore
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
unconfined

What to read out of this.

/proc/kcore is the whole of physical memory as a file. In the default container ls -l shows crw-rw-rw- … 1, 3 — character device major 1, minor 3, which is /dev/null. The runtime bind-mounted /dev/null over it. Reading it returns nothing, writing it discards. The same treatment is applied to /proc/keys, /proc/timer_list and /proc/sched_debug. Directories are masked a second way — an empty read-only tmpfs is mounted over them — so ls -ld /sys/firmware shows drwxrwxrwt … 40 rather than a character device.

In the privileged container it is the real file, and its size is the machine's physical address space — a number in the hundreds of billions. A process that can read that can read any secret held by any process on the host, without needing to escape anywhere.

The second line is the other quiet protection. /sys is mounted ro in a normal container, and the deliberate failure shows why that matters: /sys/kernel/uevent_helper is a file the kernel executes on the host when a device event occurs. Writable, it is a one-line escape. Privileged, /sys is rw.

Three lines, three defences, all removed by one flag — and none of them is "capabilities", which is the only one most people mention. Add Seccomp: 0 and unconfined from the earlier exercises and you can describe --privileged completely and precisely, which very few candidates can.

--- before ---
→ Overall exposure level for demo-hardening.service: 9.4 UNSAFE 😨

--- after ---
→ Overall exposure level for demo-hardening.service: 2.4 OK 🙂

--- the whole machine, worst first ---
UNIT                                 EXPOSURE PREDICATE HAPPY
containerd.service                        9.6 UNSAFE    😨
cron.service                              9.6 UNSAFE    😨
docker.service                            9.6 UNSAFE    😨
ssh.service                               9.6 UNSAFE    😨
systemd-udevd.service                     9.4 UNSAFE    😨
systemd-resolved.service                  2.0 OK        🙂

What to read out of this.

From 9.4 to 2.4, and not one line of the program changed. Every directive in that drop-in is a kernel feature you have already met in this module — no_new_privs, an empty bounding set, a seccomp allow-list, and a private mount namespace. systemd is a convenient front end, not a new mechanism.

Read the ✗ lines, not the number. The score is a weighted sum with no absolute meaning; the value is in the list of things not done. After hardening, eighteen items are still ✗ — run it without tail and read them. The largest are PrivateNetwork= (0.5), User=/DynamicUser= (0.4, because nobody is itself a poor choice for a service) and RestrictAddressFamilies=~AF_(INET|INET6) (0.3). Every one of them is honest: this service really can still reach the network, and really does run as a shared system user, because nothing told it otherwise.

The fleet view is the one to keep. systemd-analyze security with no arguments ranks every unit on the machine. Do not be alarmed that docker.service and sshd sit at 9.6: some services genuinely need the access, and the tool has no idea what a service is for. Its job is to give you an ordered list so that you harden the vendor agent that needs nothing before you spend a week on the one that needs everything.

A caution before you copy that drop-in onto something real. ProtectSystem=strict makes the entire filesystem read-only, so a service that writes anywhere needs ReadWritePaths= or a StateDirectory=. MemoryDenyWriteExecute=yes — deliberately left out above — breaks anything with a JIT, which includes most JVM, .NET and Node workloads. Harden one directive at a time, with a restart and a smoke test between each.

Name what it actually removes, in order, because "it's basically root on the host" is the answer everyone gives.

privileged: true grants all 41 capabilities, sets the seccomp profile to unconfined, sets the AppArmor or SELinux profile to unconfined, lifts the device cgroup so every host device is usable, unmasks /proc/kcore and friends, and mounts /sys read-write. Five protections, one field.

The concrete consequences: it can mount the node's root filesystem and read or write any file including /etc/shadow and every other pod's secrets; it can read all of physical memory through /proc/kcore; it can write /sys/kernel/uevent_helper to have the kernel execute a program on the node; and it can load a kernel module. On a shared node it is, in practice, cluster-admin, because the kubelet's credentials are on that disk.

The details that separate candidates: saying what it does not remove — namespaces. The pod still has its own PID, mount and network view; hostPID, hostNetwork and hostIPC are separate fields. That is why it "looks contained", and why the correct framing is visibility is not authority. Finish with the fix: Pod Security Admission at baseline rejects it at the API server, so it never becomes a code-review argument.

In the order the controls actually matter.

Do not run as root. runAsNonRoot: true with a real UID removes most of the value of anything an attacker finds. allowPrivilegeEscalation: false — one line, no_new_privs, no analysis needed. capabilities.drop: ["ALL"], adding back only what is proven necessary, and dropping NET_RAW even if you add nothing else. seccompProfile.type: RuntimeDefault, remembering that Kubernetes does not apply it unless you ask. readOnlyRootFilesystem: true with named writable volumes. Leave the LSM alone and let docker-default or container_t do its job.

Then the things that are not securityContext at all: no hostPath mounts, especially not / or the container socket; patch the node kernel, because the one escape you cannot see in a manifest is a kernel bug; and enforce all of it in admission control rather than review.

The details that separate candidates: pointing out that most real-world "escapes" are not kernel exploits at all — they are a mounted Docker socket, a writable hostPath, or an over-broad service account token, all of which are misconfiguration rather than vulnerability. And naming user namespaces as the structural fix: rootless containers make the container's UID 0 map to an unprivileged host UID, so a full capability set inside is worth nothing outside, exactly as in Exercise A3.1.

Not setenforce 0. An LSM only ever removes access, so it cannot be the cause of a failure that would otherwise have succeeded — it is reporting a policy gap, and disabling it deletes the report rather than the problem.

Read the denial: ausearch -m AVC -ts recent, or journalctl -t setroubleshoot. The AVC record names the source type, the target type and the operation, which is usually enough on its own. audit2why explains it in prose, and audit2allow will generate a policy module — read what it generates before loading it, because it will happily allow far more than you need.

The most common container case is a hostPath volume whose files are not labelled container_file_t. The fix is a label, not a policy: chcon -Rt container_file_t /path to test, semanage fcontext plus restorecon to make it survive a relabel. In Docker the :z and :Z volume suffixes do this for you.

The details that separate candidates: knowing that setenforce 0 is temporary and /etc/selinux/config is permanent, so a machine "fixed" with the former quietly re-breaks at the next reboot — which is the worst possible time to discover it. And offering the middle path: put the one domain in permissive mode with semanage permissive -a, rather than the whole system.

Find out why first — the answer is nearly always "the base image's default user is root and nobody changed it", and occasionally "it writes to a directory only root can write".

Fix it in the image where you can: a USER 10001 line in the Dockerfile, and chown the directories the application writes during the build. Then enforce it in the manifest so a rebuild cannot undo it:

securityContext:
  runAsNonRoot: true
  runAsUser: 10001
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities: { drop: ["ALL"] }

runAsNonRoot: true is the important one, because it makes the kubelet refuse to start a container whose image user resolves to UID 0 — so this cannot regress silently on the next image build.

The details that separate candidates: handling the two things that break. Files baked into the image with root-only permissions need fixing at build time, not with an init container running as root. And if the application listens on port 80, the answer is to change it to 8080 rather than to add NET_BIND_SERVICE — the Service or ingress maps the port, and the capability is not needed at all. Mentioning that readOnlyRootFilesystem: true usually needs an emptyDir mounted at /tmp shows you have actually done it.

== pid 2593 : bash
Uid:	1000	1000	1000	1000
Gid:	1000	1000	1000	1000
CapEff:	0000000000000000
CapBnd:	000001ffffffffff
CapAmb:	0000000000000000
NoNewPrivs:	0
Seccomp:	0
CapEff decoded: 0x0000000000000000=
LSM confinement: unconfined
2593: =

--- 1. mode bits (EACCES) ---
cat: /tmp/rootonly: Permission denied
exit: 1
--- 2. missing capability (EPERM) ---
chown: changing ownership of '/tmp/rootonly': Operation not permitted
exit: 1
--- 3. read-only mount (EROFS) ---
touch: cannot touch '/tmp/ro/x': Read-only file system
exit: 1

What to read out of this.

Learn the first block as one unit. Six lines from /proc/PID/status plus a decode, and you have the complete privilege posture of any process on any machine — your shell, PID 1, or a container's process seen from the node. It works without the runtime, without kubectl, and without anything installed inside the container.

CapEff decoded: 0x0000000000000000= is the decode of an empty mask — the trailing = with nothing after it. That is correct output, not a broken command. getpcaps says the same thing more briefly: 2593: = is "this process holds nothing"; a root shell would print something like 2593: =ep.

LSM confinement is the one line that needs care. On a kernel with no active LSM the file contains the word kernel with no trailing newline, so without the echo the next command's output lands on the same line. On an AppArmor host it reads unconfined, and on SELinux it is a full label such as unconfined_u:unconfined_r:unconfined_t:s0.

Then the three refusals, which is the whole point. All three are exit status 1 from a program that could not do its job, and all three would appear in an application log as some variation of "permission problem". The wording is what separates them: Permission denied sent you to ls -l, Operation not permitted sent you to CapEff, and Read-only file system sent you to findmnt. Three completely different investigations, decided by reading the message rather than guessing.

Note that refusal 2 came from a UID-0 process, exactly as in Exercise A1.2 — so "run it as root and see if it works" would have confirmed the wrong theory here. That is why the posture dump comes first.

In a container, run the posture block against PID 1 from the node: P=$(host pid of the container); …. You get its capabilities, its seccomp state and its LSM profile without entering it, which matters when the image has no shell.

Do not argue about privileged. Find out which single operation they need.

Run it in a sandbox cluster with privileged: true and record what it actually does: strace -f -c on its main process, or simply read what it opens. Nearly always the real requirement is one of a very short list — reading /proc of other processes (needs hostPID, not privileged), reading /sys for hardware metrics (needs a read-only hostPath, not privileged), a raw socket for network metrics (CAP_NET_RAW), or perf counters (CAP_PERFMON, since 5.8, specifically so tools stop asking for CAP_SYS_ADMIN).

Then propose the minimum: capabilities: {drop: [ALL], add: [<the one>]}, allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, seccompProfile: RuntimeDefault, and a read-only hostPath for exactly the paths it reads. Test it; it usually works first time, because privileged: true was in the chart to avoid a support burden, not because anyone measured it.

If it genuinely needs CAP_SYS_ADMIN, that is a decision with a name on it, not a default. Confine it to a dedicated node pool with a taint, give it its own service account with no cluster-wide read, and set a review date. Record in the exception that CAP_SYS_ADMIN was 38% of all capability checks in Linux 3.2 and just over 45% by 5.2, and that this is not "one extra capability".

What not to do: grant privileged: true "temporarily". The deadline is real and the review never happens.

159 is 128 + 31, and signal 31 is SIGSYS. That is a seccomp kill.

The exit-code arithmetic is the whole diagnosis. A container killed by a signal exits 128 + N; you already know 137 as 128 + 9 (SIGKILL, usually the OOM killer, Module 09). 159 is SIGSYS, and inside a container that is essentially always seccomp — either a KILL_PROCESS or KILL_THREAD verdict, or a TRAP verdict for which the process installed no handler. There is no application log line because the process was killed between making a system call and returning from it — it never got the chance to write one.

Confirm in one step: re-run the pod with seccompProfile: {type: Unconfined}. If it lives, the diagnosis is certain.

Then find the call, without leaving it unconfined. Copy the node's profile, change defaultAction to SCMP_ACT_LOG, install it as a Localhost profile, and run the workload. Every call is recorded and none blocked; the audit log names the syscall. Add just that one to the allow-list and redeploy.

Expect the answer to be a JVM call. The usual culprits are membarrier, perf_event_open, process_vm_readv or a newer clone3 — the JVM feature-detects aggressively at startup, which is why this fails "a few seconds after start" rather than immediately.

And fix the class of bug, not the instance: the profile should use ERRNO rather than KILL as its default verdict wherever possible, and it should return ENOSYS for unknown calls so libraries fall back instead of giving up — the lesson of the clone3 breakage in 2021.

Different distributions, different LSM. Ubuntu is AppArmor; RHEL is SELinux, and the volume is unlabelled.

Confirm the split first: cat /sys/kernel/security/lsm on one node of each kind. The RHEL nodes will list selinux and getenforce will say Enforcing.

Then read the denial rather than guessing: ausearch -m AVC -ts recent on the failing node. It will show a process of type container_t denied read on a file whose type is var_log_t — a host label that container processes are not permitted to touch.

Note what the error is not. It is EACCES, not EPERM, and the container is root with capabilities intact. Being root is irrelevant here: mandatory access control is not something a capability overrides. That is the definition of mandatory.

The fix is a label, not a policy. For a hostPath, the supported route is the SELinux options in the pod's securityContext — seLinuxOptions with an appropriate type — or, for log collection specifically, using the node's container-log paths, which are already labelled for container access. In plain Docker the :z and :Z volume suffixes relabel for you.

What not to do: setenforce 0, or --security-opt label=disable. The first is silently temporary and reverts at reboot, at the worst possible moment. Both delete the report rather than the problem — an LSM never grants access, so it cannot be the reason something that should work does not.

The binary had a file capability, and NFS cannot store one.

cap_net_bind_service=ep was on /opt/collector/bin/collector, which is how an unprivileged user was binding port 443. File capabilities live in the security.capability extended attribute, and NFS does not carry the security.* xattr namespace — Red Hat documents this as by design for both NFSv3 and NFSv4. Copying the binary onto NFS silently dropped the attribute.

Confirm in two commands: getcap /opt/collector/bin/collector now prints nothing, and sudo setcap cap_net_bind_service=ep <path> fails with Operation not supported. findmnt -no FSTYPE -T /opt confirms nfs4.

Watch for the same failure with a different cause, because it will look identical: if /opt had been a local mount with nosuid, getcap would happily show the capability and it would still be ignored, since nosuid disables file capabilities as well as setuid bits. findmnt -no FSTYPE,OPTIONS -T /opt distinguishes the two in one line.

The durable fix is to stop depending on the filesystem. Put AmbientCapabilities=CAP_NET_BIND_SERVICE in the systemd unit alongside User=collector. The capability is then granted by the service manager at start, survives package upgrades that replace the binary, and works from any filesystem. Better still, if you control the port: net.ipv4.ip_unprivileged_port_start=443, or listen on 8443 behind the load balancer, and need no privilege at all.