Module 03 — File Descriptors, Inodes & Filesystems

Updated 21 August 2026

Module 03 · File descriptors, inodes and filesystems

In Module 02 you watched a shell open a file and turn it into "output number 1", and then ls wrote to it without ever knowing. This module explains what those numbers are, what a file really is underneath, and why a disk can be full with plenty of space left.

🧠 concept → 🧩 real-world analogy → 🧪 exercise → ✅ expected result (hidden) → 🎯 interview questions (hidden)

Before you start, you should already know:

From Module 01 — what a system call is, how to read strace output, what errno means, and how to read files under /proc.

From Module 02 — what a process is, fork and exec, and that a child keeps its parent's open files.

If any of that is fuzzy, go back first. This module builds straight on top of it.

🔢 Part A · File descriptors

A1 · What a file descriptor really is

Official docs: open(2) · proc(5) — /proc/PID/fd · intro(2)

When a program opens a file, it does not get the file back. It gets a small number.

That number is a file descriptor. It is nothing more than a position in a list the kernel keeps for your process. Entry 0, entry 1, entry 2, and so on.

So when your program says "write these bytes to 3", the kernel looks up entry 3 in your list and finds out what 3 actually means.

What you think you have	What you actually have
"My program has the file open"	Your program has a number. The kernel has the file.
"File descriptor 3 is my log file"	Entry 3 in your process's list points to your log file. In another process, 3 means something completely different.

Two consequences follow, and both matter later:

The numbers are per-process. Descriptor 3 in your shell and descriptor 3 in your web server have nothing to do with each other.
You always get the lowest free number. If your program has 0, 1, 2 and 5 open, the next open returns 3, not 6. Close a descriptor and the next open reuses that number.

Real-world analogy — the cloakroom ticket

You hand your coat in at a cloakroom and get a ticket with a number on it.

You do not have your coat. You have a number. The cloakroom has your coat.

The ticket does not describe the coat. It does not say what colour it is or where it is hanging. It is just a number that the cloakroom staff can look up in their list.

And the number only means something in that cloakroom. Ticket 42 at the theatre and ticket 42 at the museum are not related in any way. That is exactly why descriptor 3 in one process has nothing to do with descriptor 3 in another.

Where the analogy stops working — and it is the whole of Section B4. In a real cloakroom, if the coat is removed, your ticket becomes worthless.

In Linux the opposite happens. Someone can delete the file completely, and your ticket still works. The file stays alive, and keeps taking up disk space, for as long as anyone is still holding a ticket for it. That single difference causes one of the most common production incidents there is.

🧪 Exercise A1.1 — Look at your own list of open files

bash

# Your shell's file descriptor list. Each entry is a numbered link.
ls -l /proc/$$/fd

# Open a file as descriptor 3 and look again
exec 3> /tmp/demo.txt
ls -l /proc/$$/fd

# Close it, and watch entry 3 disappear
exec 3>&-
ls -l /proc/$$/fd

✅ Expected result — click to reveal

plain text

$ ls -l /proc/$$/fd
total 0
lrwx------ 1 zaeem zaeem 64 Aug 20 09:14 0 -> /dev/pts/0
lrwx------ 1 zaeem zaeem 64 Aug 20 09:14 1 -> /dev/pts/0
lrwx------ 1 zaeem zaeem 64 Aug 20 09:14 2 -> /dev/pts/0
lrwx------ 1 zaeem zaeem 64 Aug 20 09:14 255 -> /dev/pts/0

$ exec 3> /tmp/demo.txt
$ ls -l /proc/$$/fd
total 0
lrwx------ 1 zaeem zaeem 64 Aug 20 09:14 0 -> /dev/pts/0
lrwx------ 1 zaeem zaeem 64 Aug 20 09:14 1 -> /dev/pts/0
lrwx------ 1 zaeem zaeem 64 Aug 20 09:14 2 -> /dev/pts/0
l-wx------ 1 zaeem zaeem 64 Aug 20 09:14 3 -> /tmp/demo.txt
lrwx------ 1 zaeem zaeem 64 Aug 20 09:14 255 -> /dev/pts/0

$ exec 3>&-
$ ls -l /proc/$$/fd
total 0
lrwx------ 1 zaeem zaeem 64 Aug 20 09:14 0 -> /dev/pts/0
lrwx------ 1 zaeem zaeem 64 Aug 20 09:14 1 -> /dev/pts/0
lrwx------ 1 zaeem zaeem 64 Aug 20 09:14 2 -> /dev/pts/0
lrwx------ 1 zaeem zaeem 64 Aug 20 09:14 255 -> /dev/pts/0

What to read out of it.

The file descriptor list really is a numbered list, and here it is on screen. Each entry is a link from a number to a thing.

Your shell started with 0, 1 and 2 already open, all pointing at /dev/pts/0 — your terminal. Section A2 explains those three.

When you opened a file, you got 3. Not 4, not 256. Three was the lowest free number.

Look at the permission letters at the start of entry 3: l-wx. The w with no r means it was opened write-only, because you used >. The kernel remembers not just what you opened but how.

255 is bash's own bookkeeping. You can ignore it.

After closing, entry 3 is gone. If you opened another file now, you would get 3 again.

Now imagine this at 500 hosts. ls -l /proc/PID/fd is the fastest way to answer "what is this process actually touching?" without stopping it. Which config file did it really read? Which socket is it holding open? Which log file is it still writing to?

It is completely safe on a live process — unlike strace, it stops nothing. This is often the first command to run on a service that is behaving strangely.

A2 · The three you always get: 0, 1 and 2

Official docs: stdin(3) · open(2) · proc(5)

Every process starts with three descriptors already open. Nobody opened them — they were inherited, because of fork and exec from Module 02.

Number	Name	What it is for
0	standard input	Where the program reads from. Normally your keyboard.
1	standard output	Where normal results go. Normally your screen.
2	standard error	Where problems and warnings go. Also normally your screen.

The question worth asking is why 1 and 2 both go to the screen. If they end up in the same place, why have two?

Because you can send them to different places. Keeping them separate means you can save the results and still see the errors, or throw the errors away and keep the results. If they were one stream, your error messages would be mixed into your data forever.

Real-world analogy — three trays on a desk

An old-fashioned office desk has three trays.

The in-tray is where work arrives. That is descriptor 0.
The out-tray is where finished work goes. That is descriptor 1.
The problems tray is where anything that went wrong goes. That is descriptor 2.

By default, someone empties the out-tray and the problems tray into the same place, so it looks like there is no difference. There is.

The moment you want finished work to go to the archive and problems to go to your manager, you need them separated at the desk — not sorted out afterwards. Once the two piles are mixed you cannot reliably separate them again.

That is exactly why a program must decide, as it writes, whether something is a result or a problem.

Where the analogy stops working. A tray is a physical thing that belongs to the desk. Descriptors 0, 1 and 2 are just numbers, and anybody can point them anywhere before the program starts. The program cannot tell the difference and does not get a say.

🧪 Exercise A2.1 — Send results and errors to different places

bash

# ls one file that exists and one that does not.
# One line goes to descriptor 1, one goes to descriptor 2.
ls /etc/hostname /does/not/exist

echo "--- keep only the results ---"
ls /etc/hostname /does/not/exist 2>/dev/null

echo "--- keep only the errors ---"
ls /etc/hostname /does/not/exist 1>/dev/null

echo "--- send each to its own file ---"
ls /etc/hostname /does/not/exist > /tmp/results.txt 2> /tmp/errors.txt
echo "results file:"; cat /tmp/results.txt
echo "errors file:";  cat /tmp/errors.txt

✅ Expected result — click to reveal

plain text

$ ls /etc/hostname /does/not/exist
ls: cannot access '/does/not/exist': No such file or directory
/etc/hostname

--- keep only the results ---
/etc/hostname

--- keep only the errors ---
ls: cannot access '/does/not/exist': No such file or directory

--- send each to its own file ---
results file:
/etc/hostname
errors file:
ls: cannot access '/does/not/exist': No such file or directory

What to read out of it.

One command produced two separate streams, and you sorted them apart without changing ls in any way.

Notice the first output. The error line appeared before the result, even though /etc/hostname was listed first.

That is ls itself, not buffering. ls checks every argument up front and reports the ones it cannot reach straight away, then sorts and prints the rest afterwards.

Buffering can reorder the two streams as well, and that is worth knowing separately: descriptor 2 is unbuffered, while descriptor 1 switches to block buffering as soon as it is not going to a terminal. That is why results and errors often interleave strangely once you redirect them into a log file.

Now imagine this at 500 hosts. This is why 2>/dev/null is dangerous in production scripts. It is usually written to hide one noisy warning, and it silently hides every future error too, including the ones that matter.

The safer habit is to send errors somewhere you can still find them: 2>> /var/log/myjob.err rather than 2>/dev/null. A cron job that throws its errors away is a cron job nobody knows is broken.

A3 · Redirection — how > actually works

Official docs: dup(2) — dup and dup2 · open(2) · execve(2)

In Module 02 you saw this in a trace and it was left half-explained. Here is the whole thing.

When you type ls > out.txt, the shell does not hand the filename to ls. ls never learns that a file is involved. It writes to descriptor 1, exactly as it always does.

What happens is this, in the child process, in the gap between fork and exec:

Open out.txt. The kernel returns the lowest free number — 3.
Call dup2(3, 1). This means: make entry 1 point at whatever entry 3 points at. Entry 1 was the terminal; now it is the file. Whatever entry 1 used to point at is closed.
Close entry 3, which is no longer needed.
Call exec and become ls.

Now ls starts up. It has descriptors 0, 1 and 2 like every program. It writes results to 1. The bytes land in the file. It has no idea.

Real-world analogy — rewiring the phone extension

In an office, extension 1 always rings the sales desk. Everyone knows this. Staff are trained to dial 1 for sales and they never think about it again.

One morning, before anyone arrives, an engineer rewires extension 1 so it rings the warehouse instead.

Staff arrive and dial 1, exactly as always. Nobody was told anything. Nobody looks anything up. The calls go to the warehouse.

That is redirection. ls dials 1. The shell rewired 1 before ls arrived.

And notice when the rewiring happened: before anyone turned up for work. That is the gap between fork and exec from Module 02. The child was still the shell, so it could change its own wiring. Then it became ls.

Where the analogy stops working — and it is why this design is so good. An engineer has to rewire each phone for each new member of staff.

Here, the rewiring survives exec automatically, because descriptors carry across. So redirection works with every program on the system, including ones written decades before your shell existed, and none of them needed a single line of code to support it.

🧪 Exercise A3.1 — Do the rewiring by hand

bash

# Point descriptor 3 at a file, then copy it onto descriptor 1.
# From that moment, everything your shell prints goes to the file.
exec 3> /tmp/rewired.txt
exec 4>&1        # save a copy of the real terminal on descriptor 4
exec 1>&3        # rewire 1 to the file

echo "this line goes to the file"
ls /etc/hostname

exec 1>&4        # put descriptor 1 back to the terminal
exec 3>&- 4>&-   # tidy up

echo "back on screen. The file contains:"
cat /tmp/rewired.txt

✅ Expected result — click to reveal

plain text

$ exec 3> /tmp/rewired.txt
$ exec 4>&1
$ exec 1>&3
$ echo "this line goes to the file"
$ ls /etc/hostname
$ exec 1>&4
$ exec 3>&- 4>&-
$ echo "back on screen. The file contains:"
back on screen. The file contains:
$ cat /tmp/rewired.txt
this line goes to the file
/etc/hostname

What to read out of it.

The two commands in the middle printed nothing at all. Your shell was not broken and the commands were not silent — descriptor 1 was pointing at a file, so that is where the text went.

exec 4>&1 is the important trick. It saved a second copy of the terminal on descriptor 4 before overwriting descriptor 1. Without that copy there would have been no way back, and your shell would print to that file until you closed the window.

Notice that ls was also redirected, even though you never mentioned ls. It inherited descriptor 1 from the shell, exactly as Module 02 described. Change the shell's wiring and every command it starts afterwards is wired the same way.

🧪 Exercise A3.2 — The ordering trap (this one is meant to catch you out)

bash

# These two look like they mean the same thing. They do not.
# Predict what each file will contain BEFORE running them.

ls /etc/hostname /does/not/exist > /tmp/a.txt 2>&1
ls /etc/hostname /does/not/exist 2>&1 > /tmp/b.txt

echo "=== a.txt (redirect first, then copy) ==="
cat /tmp/a.txt
echo "=== b.txt (copy first, then redirect) ==="
cat /tmp/b.txt

✅ Expected result — click to reveal

plain text

$ ls /etc/hostname /does/not/exist 2>&1 > /tmp/b.txt
ls: cannot access '/does/not/exist': No such file or directory

=== a.txt (redirect first, then copy) ===
ls: cannot access '/does/not/exist': No such file or directory
/etc/hostname

=== b.txt (copy first, then redirect) ===
/etc/hostname

What to read out of it. The error line appeared on your screen for the second command, and b.txt does not contain it. Most people expect both files to be identical.

Read each one left to right, remembering that 2>&1 means "make 2 point wherever 1 points right now" — it copies the current wiring, it does not create a permanent link.

> /tmp/a.txt 2>&1

Point 1 at the file.
Point 2 wherever 1 points — which is now the file. ✅ Both go to the file.

2>&1 > /tmp/b.txt

Point 2 wherever 1 points — which is still the terminal.
Then point 1 at the file.
Descriptor 2 was never updated. It is still on the terminal. ❌

The order is the whole meaning. 2>&1 is a snapshot of where 1 goes at that instant, not a promise to follow it.

Now imagine this at 500 hosts. This is one of the most common bugs in cron jobs and container entrypoints. Someone writes mycommand 2>&1 > /var/log/out.log, believing everything is captured. The results are logged and every error message is lost.

The job then fails silently for months. The correct form is > /var/log/out.log 2>&1, or in bash simply &> /var/log/out.log.

A4 · Pipes — wiring one process into another

Official docs: pipe(7) — overview of pipes · dup(2) · proc(5)

Once you can point a descriptor anywhere, an obvious idea appears: point one program's output at another program's input.

That is a pipe. The kernel gives you a pair of descriptors joined together in memory. Whatever is written to one end can be read from the other. There is no file and nothing touches the disk.

So when you type ls | wc -l, the shell:

Creates a pipe — two descriptors, a write end and a read end.
Forks twice, once for each command.
In the ls child, points descriptor 1 at the pipe's write end.
In the wc child, points descriptor 0 at the pipe's read end.
Execs both.

Neither program knows the other exists. ls writes to 1 as always. wc reads from 0 as always.

Real-world analogy — the chute between two floors

A sorting office has a chute in the floor. Staff upstairs drop parcels into it. Staff downstairs take parcels out of it.

The person upstairs never meets the person downstairs. They do not agree a schedule, and neither needs to know the other's name. One drops things in, the other takes things out.

Two things follow from the shape of a chute, and both are real behaviours of pipes:

It only goes one way. You cannot send anything back up. A pipe is one-directional — that is why a two-way conversation between programs needs something else.
It holds only so much. If the person downstairs stops collecting, the chute fills, and the person upstairs cannot drop anything else in. They stand there, holding a parcel, waiting. They are not broken — they are blocked.

That is exactly why a slow command at the end of a pipeline quietly slows down every command before it.

Where the analogy stops working. A chute is part of the building. A pipe exists only while somebody is holding it. When the reader goes away, the writer is not left waiting forever — the kernel tells the writer the other end is gone. That is the Broken pipe message you have seen when you close less early.

🧪 Exercise A4.1 — Find the pipe in the file descriptor list

bash

# Start a pipeline that stays running, then look at how it is wired
sleep 60 | cat > /dev/null &
sleep 1

# Find the two processes
ps -eo pid,ppid,comm | grep -E 'sleep|cat' | grep -v grep

# Look at what each end holds. Substitute the PIDs you actually got.
SLEEP_PID=$(pgrep -n sleep)
CAT_PID=$(pgrep -n cat)
echo "--- sleep (the writer) ---"; ls -l /proc/$SLEEP_PID/fd
echo "--- cat (the reader) ---";   ls -l /proc/$CAT_PID/fd

kill $SLEEP_PID 2>/dev/null

✅ Expected result — click to reveal

plain text

--- sleep (the writer) ---
lrwx------ 1 zaeem zaeem 64 Aug 20 09:31 0 -> /dev/pts/0
l-wx------ 1 zaeem zaeem 64 Aug 20 09:31 1 -> 'pipe:[184023]'
lrwx------ 1 zaeem zaeem 64 Aug 20 09:31 2 -> /dev/pts/0

--- cat (the reader) ---
lr-x------ 1 zaeem zaeem 64 Aug 20 09:31 0 -> 'pipe:[184023]'
l-wx------ 1 zaeem zaeem 64 Aug 20 09:31 1 -> /dev/null
lrwx------ 1 zaeem zaeem 64 Aug 20 09:31 2 -> /dev/pts/0

What to read out of it — this is the clearest picture of a pipeline you will get.

Look at the number in brackets: pipe:[184023]. It appears in both lists. That same number is descriptor 1 for sleep and descriptor 0 for cat.

That one shared number is the entire pipeline. Two unrelated programs, joined by a single object in kernel memory.

Notice the permission letters. sleep has l-wx on its end — write only. cat has lr-x — read only. The chute goes one way, and the kernel enforces it.

Neither program was written to know about the other. sleep writes to 1. cat reads from 0. The shell did all the wiring before either of them started.

Also notice cat has descriptor 1 on /dev/null, because of the > /dev/null at the end — while descriptor 2 is still your terminal, because nothing redirected it. One command, three different destinations, all set up before either program started.

A5 · Running out of file descriptors

Official docs: getrlimit(2) — RLIMIT_NOFILE · errno(3) · proc(5)

The list of descriptors is not unlimited. There are two separate ceilings, and mixing them up wastes hours.

Error	Message	What ran out
EMFILE	Too many open files	This one process hit its own limit. Everything else on the machine is fine.
ENFILE	Too many open files in system	The whole machine hit its limit. Everything is about to break.

One letter apart, and completely different problems. EMFILE means fix one service. ENFILE means the host is in trouble.

The per-process limit also comes in two parts, which you met in Module 01's docs reading:

The soft limit is what is enforced right now. A process can raise its own soft limit at any time, up to the hard limit.
The hard limit is the ceiling on the soft limit. Only root can raise it.

Real-world analogy — hooks in the cloakroom

Back to the cloakroom. It has a fixed number of hooks.

Your own allowance is the soft limit: "any one guest may hang up 20 coats." Hit that and you are turned away while everyone else carries on perfectly happily. That is EMFILE — your problem, nobody else's.

The room's total capacity is the whole-machine limit. When every hook in the building is used, nobody can hang up anything. That is ENFILE, and it is everyone's problem at once.

Now the part that makes this a real incident rather than a curiosity. Imagine guests who take a ticket, wander off, and never collect their coat. Each one permanently uses a hook. Nothing looks wrong for hours. Then one afternoon the cloakroom is full of coats nobody is coming back for.

That is a file descriptor leak — a program that opens files or connections and never closes them. It runs fine for a week and then fails all at once.

Where the analogy stops working. A cloakroom attendant can see the coats piling up. Nothing warns you about descriptors. You have to look, which is why ls /proc/PID/fd | wc -l on a long-running service is a genuinely useful habit.

🧪 Exercise A5.1 — Hit the limit on purpose

bash

# What are your limits?
ulimit -Sn    # soft limit - enforced now
ulimit -Hn    # hard limit - the ceiling

# How many are open across the whole machine, and what is the machine's max?
cat /proc/sys/fs/file-nr
cat /proc/sys/fs/file-max

# Now hit the per-process limit deliberately, in a throwaway subshell
( ulimit -Sn 20
  echo "soft limit is now $(ulimit -Sn)"
  for i in $(seq 1 30); do
    exec {fd}> /tmp/fdtest_$i || { echo "FAILED opening number $i"; break; }
  done
)

rm -f /tmp/fdtest_*

✅ Expected result — click to reveal

plain text

$ ulimit -Sn
1024
$ ulimit -Hn
1048576

$ cat /proc/sys/fs/file-nr
1856	0	9223372036854775807
$ cat /proc/sys/fs/file-max
9223372036854775807

soft limit is now 20
bash: /tmp/fdtest_11: Too many open files
FAILED opening number 11

What to read out of it.

The soft limit is 1024 and the hard limit is over a million. That gap is deliberate: the low default protects the machine from a runaway program, and any process that genuinely needs more can raise its own soft limit without asking root.

This is exactly what a web server or database does at startup. It is why "just run ulimit -n 65535" often fixes nothing — the service was going to set its own value anyway, and the setting that matters is in its unit file (LimitNOFILE), not in your shell.

The failure came at 11, not 20, because the limit counts every descriptor the process holds — not just the ones you opened deliberately. Descriptors 0, 1 and 2 were already in use, and bash keeps a few of its own. Those come out of the same allowance.

One more detail worth knowing: RLIMIT_NOFILE is defined as one greater than the highest descriptor number allowed. With a soft limit of 20, descriptor 19 is the highest you can get.

Too many open files is EMFILE — one process, its own limit. The machine was never in any danger.

Now imagine this at 500 hosts. "Too many open files" is one of the most common production errors there is, and it is usually a leak rather than a limit that is too low.

Diagnose it in this order: ls /proc/PID/fd | wc -l for the current count, then ls -l /proc/PID/fd | awk '{print $NF}' | sort | uniq -c | sort -rn to see what kind of thing is piling up. Hundreds of sockets to the same address means unclosed connections. Hundreds of the same file means an unclosed file handle.

Raising the limit buys you time. It does not fix a leak — it just moves the outage to next month.

🎯 Interview questions — File descriptors

Q. What is a file descriptor?

A small non-negative integer that identifies an open file within one process. It is an index into a table the kernel keeps per process. The program holds the number; the kernel holds everything else — which file, the current position, and how it was opened.

Every process starts with three: 0 standard input, 1 standard output, 2 standard error.

Two rules worth stating without being asked:

open always returns the lowest unused number. Close 3 and the next open returns 3. This is not an implementation detail — the classic "close then open" idiom for redirection depends on it.
Descriptors are per process. Descriptor 3 in two different processes are unrelated.

The details that separate candidates:

Say that descriptors cover more than files. Sockets, pipes, terminals, devices, timers and even other processes are all descriptors. That uniformity is why select, poll and epoll can wait on any mixture of them, and it is the real reason "everything is a file" matters.
Name where to look: ls -l /proc/PID/fd shows exactly what a live process holds, and unlike strace it stops nothing.
Descriptors survive exec unless marked close-on-exec. That is precisely why shell redirection works for every program without any program supporting it.

Q. Explain what happens when you run command > file 2>&1. Why is the order important?

The shell forks, and in the child — before exec — it rewires descriptors. Read the redirections left to right:

> file opens the file and points descriptor 1 at it.
2>&1 points descriptor 2 at wherever descriptor 1 currently points, which is now the file.

Result: both streams go to the file.

Reversing them gives something completely different. 2>&1 > file first points 2 at wherever 1 is at that moment — still the terminal — and only then moves 1 to the file. Errors continue going to the terminal.

The key idea: 2>&1 copies the current destination. It does not create a permanent link between the two. This is dup2, and dup2 copies a value.

The details that separate candidates:

Name the mechanism: open, then dup2(3,1), then close(3), then exec. And say where it happens — in the gap between fork and exec, which is the reason Linux splits process creation into two calls at all.
Point out that the program is unaware. ls never learns a file is involved. That is why redirection works with every binary on the system.
Give the production consequence: mycommand 2>&1 > /var/log/out.log in a cron job or container entrypoint silently discards every error. It is an extremely common bug. The correct forms are > file 2>&1 or bash's &> file.

Q. A service is failing with "Too many open files". How do you investigate?

First, work out which limit was hit, because they lead to different places:

EMFILE — "Too many open files" — that one process hit its own limit.
ENFILE — "Too many open files in system" — the whole machine did. Check /proc/sys/fs/file-nr.

Then for the per-process case:

Count what it holds: ls /proc/PID/fd | wc -l.
Compare against its real limit: cat /proc/PID/limits, not ulimit -n in your own shell — your shell's limits are not the service's.
Find out what is piling up — this is the step that solves it: ls -l /proc/PID/fd | awk '{print $NF}' | sort | uniq -c | sort -rn. Many sockets to one address means unclosed connections. Many copies of one file means an unclosed handle.
Decide leak or genuine need. A count that only ever rises is a leak. A count that plateaus at a high level is a service that genuinely needs a higher limit.

The details that separate candidates:

Raising the limit is not a fix for a leak. Say so explicitly. It converts an outage today into an outage next month.
Know where the limit actually comes from. For a systemd service it is LimitNOFILE in the unit file. Editing /etc/security/limits.conf does nothing for a systemd-managed service, and ulimit in your interactive shell is irrelevant to it. This trips up a lot of people.
Mention that most modern daemons raise their own soft limit at startup, which is why the hard limit is the one that usually matters.

🗂️ Part B · What a file actually is

B1 · The inode — a file's real identity

Official docs: inode(7) · stat(1) · proc(5)

Here is the idea that makes the rest of this module make sense:

A file's name is not part of the file.

The actual file — its permissions, its owner, its size, its timestamps, and the list of disk blocks holding its contents — lives in a small record called an inode. Every inode has a number.

The name lives somewhere else entirely, which is Section B2.

An inode holds everything about a file except two things:

Inside the inode

File type (regular file, directory, link)

Permissions

Owner and group

Size in bytes

Timestamps

How many names point at it

Where the data blocks are

Not in the inode

The file's name. Not stored here at all.

The file's contents. The inode points at the blocks; it does not hold them.

Real-world analogy — the self-storage place

You rent a unit at a self-storage facility. The unit has a number on the door: 42.

Everything real about your storage is attached to that unit. What is inside it, how big it is, who is allowed in, when it was last opened.

The name is not on the unit. It is in a list in the office: "Zaeem → 42".

Now notice what follows from that, because all of Part B falls out of it:

Two names can point at the same unit. The office writes a second line, "Sarah → 42". One unit, two names. That is a hard link.
Renaming is a change in the office, not in the unit. Nothing inside moves.
Deleting a name is crossing out a line in the list. The unit is only emptied when no lines point at it any more.

Where the analogy stops working — and it is Section B4. A storage office can empty a unit whenever the last name is crossed out.

The kernel cannot, because somebody might still be standing inside the unit with a key. That is a process holding a file descriptor, and the unit stays exactly as it is until that person walks out.

🧪 Exercise B1.1 — Look at a file's real identity

bash

echo "hello inodes" > /tmp/thing.txt

# The inode number - the file's actual identity
ls -li /tmp/thing.txt

# Everything the inode holds
stat /tmp/thing.txt

# Rename it and check the inode number again. Predict first.
mv /tmp/thing.txt /tmp/renamed.txt
ls -li /tmp/renamed.txt

✅ Expected result — click to reveal

plain text

$ ls -li /tmp/thing.txt
1442919 -rw-rw-r-- 1 zaeem zaeem 13 Aug 20 10:02 /tmp/thing.txt

$ stat /tmp/thing.txt
  File: /tmp/thing.txt
  Size: 13         Blocks: 8          IO Block: 4096   regular file
Device: 259,1	Inode: 1442919     Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/   zaeem)   Gid: ( 1000/   zaeem)
Access: 2026-08-20 10:02:11.244000000 +0800
Modify: 2026-08-20 10:02:11.244000000 +0800
Change: 2026-08-20 10:02:11.244000000 +0800
 Birth: 2026-08-20 10:02:11.244000000 +0800

$ ls -li /tmp/renamed.txt
1442919 -rw-rw-r-- 1 zaeem zaeem 13 Aug 20 10:02 /tmp/renamed.txt

What to read out of it.

The inode number is 1442919 before the rename and 1442919 after it. The file did not move and nothing was copied. Only a line in a list changed.

That is why renaming a 50 GB file inside the same filesystem is instant, while moving it to a different filesystem takes minutes. The first is editing a list. The second is a real copy, because the destination has its own separate set of inodes.

Links: 1 means exactly one name currently points at this inode. Section B3 makes that number change.

Look at the three timestamps, because interviewers ask about them:

Modify — when the contents last changed.
Change — when the inode last changed. Renaming or running chmod updates this but not Modify.
Access — when it was last read. Often not updated in real time, because writing a timestamp every time anyone reads a file is expensive. Most systems mount with relatime to limit that.

A detail worth knowing. There is no timestamp for "created" in the classic Unix design, which is why ctime means change time, not creation time. Newer filesystems added a Birth field, as you can see here, but plenty of tools still ignore it. Calling ctime "creation time" in an interview is a common and easily avoided mistake.

B2 · Directories are just lists of names

Official docs: inode(7) · path_resolution(7) · stat(1)

If the name is not in the inode, where is it?

In the directory. And a directory is not a container. Nothing is "inside" it. A directory is a file whose contents are a simple list of pairs:

plain text

name                 inode number
------------------   ------------
"notes.txt"     ->   1442919
"photo.jpg"     ->   1442920
"backup"        ->   1442988

That is all. Each pair is called a directory entry, or a link.

So when you ask for /home/zaeem/notes.txt, the kernel does this:

Look up home in the directory /. Get an inode number.
Look up zaeem in that directory. Get an inode number.
Look up notes.txt in that directory. Get an inode number.
Open that inode.

Each step is a separate lookup. The path is not a location — it is a set of directions.

Real-world analogy — the office list, again

Back to the storage facility. The office list is the directory. It holds names and unit numbers, and nothing else.

Now the important bit: a folder does not contain your things. The office list does not contain your furniture. It contains a line of text pointing at a unit.

This explains something that confuses beginners. Moving a file between folders on the same disk is instant, no matter how big the file is, because you crossed a line out of one list and wrote it into another. Nothing was carried anywhere.

It also explains nested folders. One of the units at this facility can itself hold another office list, pointing at more units. That is a subdirectory: a file that happens to contain a list.

Where the analogy stops working. A storage office would happily let you write a list that points back at itself, in a loop.

Linux refuses. You cannot create an extra hard link to a directory, precisely because it would let you build a loop that programs walking the tree could never escape. .. is the one exception, and the kernel handles it specially.

🧪 Exercise B2.1 — Prove a directory is a file containing a list

bash

mkdir -p /tmp/demo && cd /tmp/demo
touch alpha.txt beta.txt

# A directory has an inode and a size, exactly like any file
ls -ldi /tmp/demo
stat -c 'type=%F  inode=%i  size=%s  links=%h' /tmp/demo

# The list it holds: inode numbers in the left column, names on the right
ls -li

# Add MANY more names. A directory only grows once a 4 KiB block fills up,
# so three extra files will not move it - five hundred will.
touch gamma.txt delta.txt epsilon.txt
stat -c 'size after 5 files   = %s' /tmp/demo
for i in $(seq 1 500); do touch f$i; done
stat -c 'size after 505 files = %s' /tmp/demo

cd /tmp

✅ Expected result — click to reveal

plain text

$ ls -ldi /tmp/demo
1442930 drwxrwxr-x 2 zaeem zaeem 4096 Aug 20 10:11 /tmp/demo

$ stat -c 'type=%F  inode=%i  size=%s  links=%h' /tmp/demo
type=directory  inode=1442930  size=4096  links=2

$ ls -li
total 0
1442931 -rw-r--r-- 1 zaeem zaeem 0 Aug 20 10:11 alpha.txt
1442932 -rw-r--r-- 1 zaeem zaeem 0 Aug 20 10:11 beta.txt

$ stat -c 'size after 5 files   = %s' /tmp/demo
size after 5 files   = 4096
$ stat -c 'size after 505 files = %s' /tmp/demo
size after 505 files = 20480

What to read out of it.

The directory has its own inode number (1442930) and its own size (4096 bytes). It is a file. The d at the start of the permissions is the only thing marking it as a directory.

Its size of 4096 has nothing to do with how big alpha.txt and beta.txt are — both are empty. 4096 is the space the list itself takes up.

And notice the list only grows in 4 KiB steps. Adding three files changed nothing; adding five hundred took it to 20480. A directory is allocated in whole blocks, like any other file.

In ls -li, the left column is the inode number of each file. The name next to it is the entry in this list. The names live here; the files live at those inode numbers.

links=2 on a brand-new empty directory looks wrong until you see why. Two names already point at it: demo in /tmp, and . inside itself. Add a subdirectory and it becomes 3, because the subdirectory's .. points back here. This is why a directory's link count tells you how many subdirectories it has, plus two.

Now imagine this at 500 hosts. A directory's size never shrinks on most filesystems, even after the files are deleted. A folder that once held a million log files can still report a large size and be genuinely slow to list, while appearing completely empty.

If ls on an empty-looking directory takes seconds, check stat on the directory itself. The fix is to recreate the directory rather than empty it.

B3 · Hard links and symbolic links

Official docs: link(2) · symlink(2) · inode(7)

Now that names and files are separate things, two kinds of link make sense.

	Hard link	Symbolic link (symlink)
What it is	Another name pointing at the same inode	A small separate file whose contents are a path
Own inode?	No. Same inode as the original.	Yes. It is its own file.
Delete the original	Nothing happens. Your name still works.	Breaks. It still points at a path that no longer exists.
Across filesystems?	No. Inode numbers are only meaningful within one filesystem.	Yes. It is just text.
To a directory?	No. Not allowed, to prevent loops.	Yes.
Which is "the original"?	Neither. The names are equal. There is no first one.	Clear — one is a link, one is a target.

Real-world analogy — a second name on the list, versus a sticky note

A hard link is the office adding a second line to the list: "Sarah → 42", next to the existing "Zaeem → 42".

Both lines are exactly as real as each other. There is no way to tell which was written first, and no reason to care. Cross out either one and the unit is still there, because the other line still points at it. The unit is only cleared when the last line is gone.

A symlink is a sticky note stuck to the door of unit 7 saying "go to unit 42".

The note is its own thing. It takes up its own space. It is obviously not the unit itself.

And if unit 42 is emptied and reassigned, the note still says 42. Follow it and you get whatever is in unit 42 now — or nothing at all. The note has no idea anything changed.

Two more consequences fall straight out of this:

A sticky note can point anywhere, including at another building. A symlink works across filesystems.
A line in the list only works in this office. Unit numbers mean nothing at a different facility. That is why a hard link cannot cross a filesystem boundary.

Where the analogy stops working. You could stick a note on a door pointing at that same door. Linux allows that too, and the kernel simply gives up after following about 40 links and returns an error.

🧪 Exercise B3.1 — Watch the link count change

bash

cd /tmp
echo "original content" > file_a.txt
ls -li file_a.txt

# Make a hard link. Watch the link count and the inode number.
ln file_a.txt file_b.txt
ls -li file_a.txt file_b.txt

# Make a symlink to the same file, for comparison
ln -s file_a.txt file_c.txt
ls -li file_a.txt file_b.txt file_c.txt

# Now DELETE the original name. Predict what happens to b and c first.
rm file_a.txt
echo "--- after deleting file_a.txt ---"
echo "hard link says:"; cat file_b.txt
echo "symlink says:";   cat file_c.txt
ls -li file_b.txt file_c.txt

✅ Expected result — click to reveal

plain text

$ ls -li file_a.txt
1442940 -rw-rw-r-- 1 zaeem zaeem 17 Aug 20 10:20 file_a.txt

$ ln file_a.txt file_b.txt
$ ls -li file_a.txt file_b.txt
1442940 -rw-rw-r-- 2 zaeem zaeem 17 Aug 20 10:20 file_a.txt
1442940 -rw-rw-r-- 2 zaeem zaeem 17 Aug 20 10:20 file_b.txt

$ ln -s file_a.txt file_c.txt
$ ls -li file_a.txt file_b.txt file_c.txt
1442940 -rw-rw-r-- 2 zaeem zaeem 17 Aug 20 10:20 file_a.txt
1442940 -rw-rw-r-- 2 zaeem zaeem 17 Aug 20 10:20 file_b.txt
1442941 lrwxrwxrwx 1 zaeem zaeem 10 Aug 20 10:21 file_c.txt -> file_a.txt

--- after deleting file_a.txt ---
hard link says:
original content
symlink says:
cat: file_c.txt: No such file or directory
1442940 -rw-rw-r-- 1 zaeem zaeem 17 Aug 20 10:20 file_b.txt
1442941 lrwxrwxrwx 1 zaeem zaeem 10 Aug 20 10:21 file_c.txt -> file_a.txt

What to read out of it — three separate lessons here.

One. After ln, file_a.txt and file_b.txt show the same inode number and a link count of 2. They are not two files. They are two names for one file. Ask which is "the real one" and the question has no answer.

Two. The symlink has a different inode (1442941) and its own size of 10 bytes — which is exactly the length of the text file_a.txt. That is literally what it contains: the path, as text.

Three, and this is the point of the exercise. After deleting file_a.txt:

The hard link still works perfectly, and the link count dropped from 2 to 1. Deleting a name does not delete a file; it removes one line and decreases the count. The file dies when the count reaches zero.
The symlink is now broken. It still says -> file_a.txt, because it never knew anything else. cat reports "No such file or directory" — which is confusing, because file_c.txt clearly exists. The error is about the target, not the link.

Now imagine this at 500 hosts. Broken symlinks are a classic deployment failure. A release directory is deleted while /opt/app/current still points at it, and the service fails with "No such file or directory" for a path that visibly exists.

Find them across a machine with find /opt -xtype l, which lists symlinks whose target is missing. Run it after every deploy that rotates release directories.

🧪 Exercise B3.2 — Two things hard links cannot do (both meant to fail)

bash

cd /tmp

# 1. A hard link to a directory
mkdir -p /tmp/somedir
ln /tmp/somedir /tmp/dirlink

# 2. A hard link across filesystems.
#    /dev/shm is a separate filesystem on nearly every machine.
echo "test" > /tmp/crossfs.txt
findmnt -no TARGET,SOURCE /tmp /dev/shm
ln /tmp/crossfs.txt /dev/shm/crossfs_link.txt

# The symlink version of the same thing works fine
ln -s /tmp/crossfs.txt /dev/shm/crossfs_link.txt && echo "symlink worked"
ls -l /dev/shm/crossfs_link.txt

rm -f /dev/shm/crossfs_link.txt /tmp/crossfs.txt; rmdir /tmp/somedir

✅ Expected result — click to reveal

plain text

$ ln /tmp/somedir /tmp/dirlink
ln: /tmp/somedir: hard link not allowed for directory

$ findmnt -no TARGET,SOURCE /tmp /dev/shm
/tmp      /dev/vda2[/tmp]
/dev/shm  tmpfs

$ ln /tmp/crossfs.txt /dev/shm/crossfs_link.txt
ln: failed to create hard link '/dev/shm/crossfs_link.txt' => '/tmp/crossfs.txt': Invalid cross-device link

$ ln -s /tmp/crossfs.txt /dev/shm/crossfs_link.txt && echo "symlink worked"
symlink worked
$ ls -l /dev/shm/crossfs_link.txt
lrwxrwxrwx 1 zaeem zaeem 16 Aug 20 10:28 /dev/shm/crossfs_link.txt -> /tmp/crossfs.txt

What to read out of it. Both failures follow directly from what an inode is.

"hard link not allowed for directory." A second name for a directory would let you create a loop — a folder that contains itself, somewhere up its own tree. Any program walking the tree would go round forever. Rather than detect loops everywhere, Unix simply forbids the thing that creates them.

"Invalid cross-device link" — that is the error EXDEV. A hard link is a name pointing at an inode number, and inode numbers only mean anything within one filesystem. Inode 1442940 on /tmp and inode 1442940 on /dev/shm are two completely unrelated files. There is no way to write a directory entry that means "inode 1442940, but on that other filesystem".

findmnt proves they really are separate: /tmp is on the disk, /dev/shm is tmpfs in memory.

The symlink works because it is just text. It does not need to understand anything about what it points at, which is exactly why it can also be broken.

Interview-grade detail. EXDEV is why mv behaves so differently depending on where you move things. Within one filesystem, mv is a rename — instant, atomic, no copying. Across filesystems, mv cannot rename, so it falls back to copy-then-delete.

That has a real consequence: a cross-filesystem mv is not atomic. Interrupt it and you can be left with a partial file at the destination. This is why tools that need safe writes always create the temporary file in the same directory as the target, then rename it.

B4 · Deleting a file that is still open

Official docs: lsof(8) — the +L1 option · inode(7) · proc(5) — /proc/PID/fd

Everything in this module now comes together into one very common production incident.

A file's data is freed when two counts reach zero:

The number of names pointing at the inode — the link count from Section B3.
The number of open file descriptors pointing at it — the cloakroom tickets from Section A1.

rm only affects the first one. It removes a name and decreases the link count. If a process still has the file open, the second count is not zero, and the data stays on disk.

The file now has no name. You cannot see it with ls. You cannot rm it again. It is still using every byte it was using before.

And df still counts that space as used, while du — which adds up files it can find by name — does not. So the two disagree, sometimes by hundreds of gigabytes.

Real-world analogy — the crossed-out line and the person still inside

The office crosses your name off the list. As far as the paperwork is concerned, unit 42 is not rented to anybody.

But you are standing inside unit 42 with the key in your hand.

The staff cannot clear it out. You are in there. So the unit stays exactly as it is, full of your things, taking up space in the building.

Now look at what the two members of staff would report:

Someone walking the office list adding up rented units says: "unit 42 is not rented." That is du. It adds up files that have names.
Someone looking at the actual building says: "unit 42 is full and I cannot use it." That is df. It reports space in use.

Both are telling the truth. The gap between them is you, standing in a unit with no name on it.

And the moment you walk out and hand back the key, the unit is cleared instantly. That is why restarting the process frees the space immediately.

Where the analogy stops working — and it is a useful feature, not a bug. In the storage place, this situation is a mistake.

In Linux it is used on purpose. A program can create a temporary file, immediately delete the name, and keep using it. Nobody else can find it, no other program can open it, and it is guaranteed to be cleaned up when the process exits — even if the process crashes. That is a genuinely elegant trick.

🧪 Exercise B4.1 — Make a disk lie to you

bash

cd /tmp

# Create a 200 MB file and hold it open from a background process
dd if=/dev/zero of=/tmp/bigfile bs=1M count=200 2>/dev/null
sleep 300 < /tmp/bigfile &
HOLDER=$!
sleep 1

echo "=== before deleting ==="
df -h /tmp | tail -1
du -sh /tmp/bigfile

# Delete it. It vanishes from ls.
rm /tmp/bigfile
ls -l /tmp/bigfile 2>&1

echo "=== after deleting - is the space back? ==="
df -h /tmp | tail -1

echo "=== who is still holding it? ==="
ls -l /proc/$HOLDER/fd | grep deleted
lsof +L1 2>/dev/null | grep -i bigfile

echo "=== now release it ==="
kill $HOLDER
sleep 2
df -h /tmp | tail -1

✅ Expected result — click to reveal

plain text

=== before deleting ===
/dev/vda2       9.8G  3.1G  6.2G  34% /
200M	/tmp/bigfile

$ rm /tmp/bigfile
$ ls -l /tmp/bigfile 2>&1
ls: cannot access '/tmp/bigfile': No such file or directory

=== after deleting - is the space back? ===
/dev/vda2       9.8G  3.1G  6.2G  34% /

=== who is still holding it? ===
lr-x------ 1 zaeem zaeem 64 Aug 20 10:41 0 -> '/tmp/bigfile (deleted)'
sleep  6142 zaeem 0r  REG 259,1 209715200 0 1442955 /tmp/bigfile (deleted)

=== now release it ===
/dev/vda2       9.8G  2.9G  6.4G  32% /

What to read out of it — read the numbers in order.

Before deleting, /tmp was at 34% used. After rm, the file is gone from ls — and df still says 34%. Nothing was freed. The 200 MB is still there.

The /proc listing shows why, and shows it in plain English. Descriptor 0 of the sleep process points at:

'/tmp/bigfile (deleted)'

The kernel is telling you exactly what has happened. That file has no name, one process is still holding it open, and it still occupies 200 MB.

lsof +L1 finds the same thing across the whole machine. The +L1 means "show me open files whose link count is below 1" — which is precisely the definition of a deleted-but-open file.

Then killing the process drops usage to 32% at once. Nobody deleted anything at that point. The last descriptor closed, the second count reached zero, and the kernel freed the blocks.

This is one of the most common "disk full" incidents there is. Someone deletes a huge log file to free space. df does not move. They delete more things. Still nothing.

The usual cause: the application still has that log file open, so deleting the name freed nothing at all. logrotate without a copytruncate or a proper reload does exactly this.

The two-command diagnosis:

lsof +L1 — lists every deleted-but-open file on the machine, with its size and the process holding it.

Then restart or reload that process to release it. You do not need a reboot.

The recovery trick worth knowing. If a process still holds it, the file is not gone. You can copy it back out through /proc:

cp /proc/PID/fd/0 /tmp/recovered.txt

Someone deletes the only copy of an important log while the service is still writing to it. As long as you get there before the process restarts, you can recover the whole thing. This works because /proc/PID/fd/N is a live handle to the actual inode, not to a name.

🎯 Interview questions — Files and links

Q. What is the difference between a hard link and a symbolic link?

A hard link is an additional name pointing at the same inode. Both names are equal — there is no "original". Deleting one just decreases the inode's link count; the file survives until the count reaches zero.

A symbolic link is a separate small file with its own inode, whose contents are a path as text. Deleting the target leaves the symlink pointing at nothing.

The two limits on hard links, and the reason for each:

Cannot cross filesystems. A directory entry stores an inode number, and inode numbers are only meaningful within one filesystem. Attempting it returns EXDEV, "Invalid cross-device link".
Cannot point at a directory. It would allow loops in the tree, which would trap any program walking it. Unix forbids the cause rather than detecting the symptom.

The details that separate candidates:

Say that neither hard link is the original. Most people describe a hard link as "a second name for the file", which is right, then talk about "the real file" as though one name is privileged. There is no such thing — there is an inode with a count of names.
Give the mv consequence. Within one filesystem mv is a rename: instant and atomic. Across filesystems it must copy then delete, so it is not atomic and an interruption can leave a partial file. This is why safe-write patterns always create the temp file in the same directory as the target.
Name a real use. Hard links are how deduplicating backup tools work — thousands of snapshots referencing the same inodes, so unchanged files cost nothing.

Q. A server reports "disk full", but df -h shows free space. What is going on?

There are three common causes, and you should name all three because they are checked differently:

1. Deleted files still held open. A process has the file open, so removing its name freed no space. df counts the blocks; du cannot see the file because it has no name. Find it with lsof +L1, then restart or reload the process holding it. This is by far the most common cause, and logrotate without copytruncate is the usual source.

2. Out of inodes, not blocks. The filesystem has plenty of space and no free inodes, so nothing new can be created. df -h looks fine; df -i shows 100%. Typical of directories holding millions of tiny files — mail queues, session files, cache directories.

3. A filesystem hidden under a mount point. Files were written to a directory before something was mounted over it. They still occupy space and are now unreachable. Check by mounting the parent elsewhere, or with du on the underlying path.

The details that separate candidates:

Say lsof +L1 specifically, and say what +L1 means: link count below 1, which is the definition of a deleted-but-open file. Anyone who says just "use lsof" has heard of the answer; naming the flag means you have used it.
Mention the recovery trick, because it turns an incident into a save: cp /proc/PID/fd/N /tmp/recovered gets the file back while the process still holds it.
Add the reserved-blocks detail. ext4 reserves 5% of the filesystem for root by default. A non-root process can get ENOSPC while df shows 5% free. tune2fs -m 1 reduces it, and on a large data volume that 5% is a lot of wasted space.

Q. What happens if you delete a file that a running process still has open?

The name is removed from its directory and the inode's link count drops. But the file's data is only freed when both counts reach zero: the number of names, and the number of open file descriptors.

So the process carries on reading and writing perfectly happily. The file has no name, does not appear in ls, cannot be reopened by anyone — and still occupies every byte it did before.

When the last descriptor closes, the kernel frees the blocks immediately.

The details that separate candidates:

Frame it as two reference counts, not one. That single sentence explains the entire behaviour, and everything else follows from it.
Say it is used deliberately. Programs create a temp file, unlink it straight away, and keep the descriptor. Nobody else can open it, and it is guaranteed to be cleaned up when the process exits — even on a crash. It is a feature, not just a quirk.
Name the observable: /proc/PID/fd shows the target as '/path (deleted)', and lsof +L1 lists them machine-wide.
Connect it to log rotation, which is where you will actually meet it. Renaming a log file while the daemon holds it open means the daemon keeps writing to the now-nameless inode. Either the daemon must reopen the file on a signal, or rotation must use copytruncate, which copies the contents and truncates in place rather than renaming.

Q. What is an inode, and what is stored in it?

An inode is the on-disk record that is the file. It holds the file type, permissions, owner and group, size, timestamps, the number of names pointing at it, and pointers to the data blocks.

It does not hold two things people expect: the filename and the file contents. The name lives in a directory entry; the contents live in the data blocks the inode points at.

Why that separation matters — everything else in this topic follows from it:

Multiple names can point at one inode. That is a hard link.
Renaming a file changes a directory entry, not the file. That is why it is instant regardless of size.
Deleting a name decreases a count. The file survives until the count reaches zero.
Inode numbers are only unique within one filesystem, which is why hard links cannot cross one.

The details that separate candidates:

Inodes are a fixed pool on ext4, allocated when the filesystem is created. You can exhaust them and get "No space left on device" with 90% of the disk free. df -i is the check. XFS and Btrfs allocate dynamically and mostly avoid this.
Get ctime right. It is change time — when the inode last changed — not creation time. chmod or a rename updates ctime and leaves mtime alone. Newer filesystems added a separate birth time, but many tools still do not show it. Calling ctime "creation time" is a very common slip.
Directories are files too, whose contents are a list of name-to-inode pairs. Being able to say that plainly shows you understand the model rather than the vocabulary.

🔒 Part C · Permissions

C1 · The nine bits, and how the kernel checks them

Official docs: chmod(1) · inode(7) — the mode field · credentials(7)

Permissions live in the inode, which you met in Section B1. There are nine bits, in three groups of three.

Group	Applies to	The three bits
user (u)	The file's owner	read (4), write (2), execute (1)
group (g)	Members of the file's group	read (4), write (2), execute (1)
other (o)	Everybody else	read (4), write (2), execute (1)

Add the numbers within each group and you get the familiar three digits. 644 means 6 for the owner (4+2, read and write), 4 for the group (read), 4 for everyone else (read).

Now the part that most explanations skip, and it is the part that actually matters.

The kernel picks exactly one group and stops. It does not add them up, and it does not fall through to the next one if the first refuses. The order is:

Are you the owner? Then only the user bits apply. Done.
Otherwise, are you in the file's group? Then only the group bits apply. Done.
Otherwise, the other bits apply.

This produces a result that looks impossible, and it is a favourite interview question.

A file with mode -r--rw---- — that is 0460 — gives the owner read-only and the group read and write.

If you are the owner and in the group, you cannot write to it. The kernel matched you as the owner, applied the user bits, and stopped. It never looked at the group bits.

So "I own this file and I still cannot write to it" is not a bug. Being the owner can give you less access than being an ordinary group member. The rule is first match wins, not best match wins.

Real-world analogy — the three doors, and the guard checks in order

A building has one entrance with a guard. The guard has three lists in a fixed order: the owner's list, the staff list, and the public rules.

You walk up. The guard looks at the owner's list first. If your name is on it, that is the list that applies to you — and the guard stops reading. Whatever it says about you is final, even if the staff list is more generous.

This is what people find unfair, and it is exactly how Unix permissions work. You do not collect the best of all three. You are matched to the first list you appear on, and that one decides everything.

Where the analogy stops working — and it is why a newer system exists. A real guard could sensibly say "well, she is also staff, let her in".

The kernel cannot, because these are just nine bits and there is nowhere to record exceptions. If you genuinely need "these three people can write, everyone else can read", nine bits cannot express it. That is what ACLs (getfacl and setfacl) are for — an extension bolted on precisely because the original model is too small for real teams.

🧪 Exercise C1.1 — Make a file you own but cannot write to

bash

cd /tmp
echo "some data" > perm_demo.txt

# Owner: read only.  Group: read AND write.
chmod 0460 perm_demo.txt
ls -l perm_demo.txt
id -nG | tr ' ' '\n' | head -3     # groups you belong to

# You are the owner AND in the group. Try to write. Predict first.
echo "more data" >> perm_demo.txt

echo "exit code: $?"
chmod 0644 perm_demo.txt; rm -f perm_demo.txt

✅ Expected result — click to reveal

plain text

$ chmod 0460 perm_demo.txt
$ ls -l perm_demo.txt
-r--rw---- 1 zaeem zaeem 10 Aug 20 11:02 perm_demo.txt

$ id -nG | tr ' ' '\n' | head -3
zaeem
sudo

$ echo "more data" >> perm_demo.txt
bash: perm_demo.txt: Permission denied
$ echo "exit code: $?"
exit code: 1

What to read out of it.

The file is owned by zaeem, its group is zaeem, and you are in the zaeem group. The permissions clearly say the group may write: rw in the middle position.

And you were refused.

The kernel checked "are you the owner?" first. You are. So it applied the user bits — r--, read only — and stopped. It never looked at the group bits at all.

This is the clearest possible demonstration that permission checking is first match wins. Being the owner is not a privilege here; it is a category that excluded you from a more generous one.

Now imagine this at 500 hosts. This is the shape of a real and confusing incident: a deployment sets ownership to the application user and mode 0640 on a shared file, and the application — which runs as that owner — can read but cannot write. The team looks at the rw for the group and concludes permissions are fine.

If a service reports permission denied on a file it appears to own, check the user bits, not the group bits. And check the directory too, which is the next section.

C2 · Directory permissions — the counter-intuitive one

Official docs: path_resolution(7) · chmod(1) · inode(7)

The same three letters mean completely different things on a directory. This catches people out constantly, so it is worth learning properly rather than by trial and error.

Bit	On a file	On a directory
r	Read the contents	List the names in it. Nothing else.
w	Change the contents	Create, delete and rename entries in it
x	Run it as a program	Go through it to reach what is inside

Section B2 explains why. A directory is a file containing a list of names. So r lets you read that list, and w lets you edit it.

And x is the one that surprises everybody. To reach /a/b/c.txt, the kernel has to step through /, then a, then b. Each step needs x on that directory. Without it, you cannot pass through — even if you know the exact path and have full permission on the file at the end.

The consequence people get wrong in interviews: deleting a file is controlled by the permissions on the directory, not on the file.

Deleting means removing a name from a list. That is editing the directory. So w on the directory is what you need — and the file itself can be read-only and owned by someone else entirely.

This is why anyone with write access to a directory can delete every file in it, no matter who owns them. Section C4 is about the fix for that.

Real-world analogy — the corridor and the rooms

A directory is a corridor. The files are rooms off it.

r on the corridor is being allowed to read the sign at the entrance listing which rooms are down there. That is all it gives you. You can read the list and go no further.
x on the corridor is being allowed to walk down it. You can reach a room — if you already know its name — but with no r you cannot read the sign to find out what rooms exist.
w on the corridor is being allowed to change the sign: add a room, remove one, rename one.

This makes the confusing cases obvious.

You can have a key to a room and still not reach it. The file is rw for you, and you cannot get to it, because you may not walk down the corridor.

You can walk to a room you cannot find. x but no r on the corridor: ls fails, but cat /path/to/exact/name works perfectly. This is a genuine security pattern — /home/user at mode 711 lets a web server reach ~/public_html without being able to see what else is in there.

And you can throw away someone else's furniture. Removing a room from the sign is editing the sign. It has nothing to do with owning what is in the room.

Where the analogy stops working. A real corridor could be walked in the dark to find rooms by touch. There is no equivalent here: without x the kernel refuses at that step, full stop, and returns EACCES — the same error as Module 01's Exercise A2.1.

🧪 Exercise C2.1 — Locked out of a file you fully own (meant to fail)

bash

cd /tmp
mkdir -p corridor
echo "the secret" > corridor/room.txt
chmod 644 corridor/room.txt      # you own it, you can read and write it

# Remove 'walk through' from the directory, keep 'read the list'
chmod 644 corridor
ls -ld corridor; ls -l corridor/room.txt 2>&1

echo "--- can we list the names? ---"
ls corridor
echo "--- can we read the file we own? ---"
cat corridor/room.txt

# Now the opposite: allow walking, forbid listing.
# Note 100, not 711 - you are the OWNER, so it is the owner bits that apply to you.
chmod 100 corridor
ls -ld corridor
echo "--- x but no r: can we list? ---"
ls corridor
echo "--- x but no r: can we read it by exact name? ---"
cat corridor/room.txt

chmod 755 corridor; rm -rf corridor

✅ Expected result — click to reveal

plain text

$ ls -ld corridor
drw-r--r-- 2 zaeem zaeem 4096 Aug 20 11:15 corridor
$ ls -l corridor/room.txt 2>&1
ls: cannot access 'corridor/room.txt': Permission denied

--- can we list the names? ---
room.txt
--- can we read the file we own? ---
cat: corridor/room.txt: Permission denied

$ chmod 100 corridor
$ ls -ld corridor
d--x------ 2 zaeem zaeem 4096 Aug 20 11:15 corridor

--- x but no r: can we list? ---
ls: cannot open directory 'corridor': Permission denied
--- x but no r: can we read it by exact name? ---
the secret

What to read out of it — the two halves are mirror images, and both are surprising.

First half: rw- on the directory, no x.

ls corridor worked and printed room.txt. You could read the list.

cat corridor/room.txt failed. You own that file. It is mode 644. You have full permission on it. And you cannot reach it, because you may not walk through the directory to get there.

Notice ls -l also failed while plain ls succeeded. ls -l needs to stat each file, and that means stepping into the directory. Reading the list is one permission; touching what is in it is another.

Second half: --x for you, no r.

ls corridor failed. You cannot read the list.

cat corridor/room.txt worked. You knew the exact name, and you were allowed to walk through.

Interview-grade detail. That second combination — x without r, usually written as mode 711 — is used on purpose.

Home directories are frequently 711 so a web server can reach /home/user/public_html while being unable to list what else is in the home directory. Everything is reachable by exact path and nothing is discoverable.

If you are asked "your app gets permission denied on a file it owns, what do you check?", the strong answer is: walk the whole path. namei -l /full/path/to/file prints the permissions of every directory along the way and shows exactly which step refuses. Most candidates only look at the file.

C3 · umask — why new files are not 666

Official docs: umask(2) · open(2) · chmod(1)

When a program creates a file it asks for permissions. Almost every program asks for 666 — read and write for everybody. Text editors, compilers, > in your shell: all of them ask for 666.

And yet the file you get is 644.

Something removed the write bits on the way through. That something is the umask.

The umask is a set of bits the kernel takes away from whatever a program asks for. It is a filter, not a setting. The default is 022, which means "remove write for group, remove write for other".

plain text

program asks for   666   (rw- rw- rw-)
umask removes      022   (--- -w- -w-)
file is created    644   (rw- r-- r--)

Directories are asked for as 777, so the same umask of 022 gives you 755. That is why new directories are executable and new files are not.

Real-world analogy — the standard security policy on every new room

A building manager has a standing rule: "whatever the tenant asks for, never hand out a master key to the cleaners or the public."

A new tenant fills in a form asking for full access for everyone. The manager processes it, applies the standing rule, and issues keys with those particular permissions crossed off. The tenant did ask. The policy removed them.

Two things follow, and both catch people out:

The policy can only take away, never add. If a tenant asks for read-only access, the standing rule cannot upgrade it to read-write. Which is why a program that deliberately asks for 600 gets 600, regardless of your umask. ssh keys work this way on purpose.
The policy is set per person, not per building. It applies to whoever is filling in the form. Your umask affects files you create; a service running as another user has its own.

Where the analogy stops working, and it is the bit people get wrong. A building manager's rule can be checked against a room at any time.

A umask is applied once, at the moment of creation, and then it is gone. Changing your umask does nothing to files that already exist. If permissions are wrong on existing files, chmod is the only fix — changing the umask fixes only the next ones.

🧪 Exercise C3.1 — Watch the umask subtract

bash

cd /tmp && mkdir -p umask_demo && cd umask_demo

umask                       # your current umask
touch default_file
mkdir default_dir
ls -l default_file; ls -ld default_dir

# Now a stricter umask: remove everything from group and other
umask 077
touch private_file
mkdir private_dir
ls -l private_file; ls -ld private_dir

# And a wide-open one - note what happens to the execute bit
umask 000
touch open_file
ls -l open_file

umask 022
cd /tmp && rm -rf umask_demo

✅ Expected result — click to reveal

plain text

$ umask
0022
$ ls -l default_file; ls -ld default_dir
-rw-r--r-- 1 zaeem zaeem    0 Aug 20 11:30 default_file
drwxr-xr-x 2 zaeem zaeem 4096 Aug 20 11:30 default_dir

$ umask 077
$ ls -l private_file; ls -ld private_dir
-rw------- 1 zaeem zaeem    0 Aug 20 11:30 private_file
drwx------ 2 zaeem zaeem 4096 Aug 20 11:30 private_dir

$ umask 000
$ ls -l open_file
-rw-rw-rw- 1 zaeem zaeem 0 Aug 20 11:30 open_file

What to read out of it.

With umask 022: the file is 644 and the directory is 755. Same umask, two different results, because the requests were different — 666 for the file, 777 for the directory.

With umask 077: 600 and 700. Everything for the owner, nothing for anyone else. This is the right umask for a machine handling anything sensitive, and it is what hardening guides such as the CIS benchmarks set.

It is not the shipped default for root, though — mainstream distributions still give root 022, the same as everyone else. That surprises people who assume root is stricter by default.

Now the last one, which is the important observation. With umask 000 the file is 666, not 777. The execute bit never appeared.

The umask can only subtract. Nothing was there to keep. A new file is never executable, no matter what your umask is, because no program asks for it — and that is a good thing. It means a file cannot accidentally become runnable just because it was created.

Now imagine this at 500 hosts. umask problems are almost always about which umask was in effect at creation time, and that is rarely your shell's.

A systemd service uses UMask= from its unit file, defaulting to 0022 — not your login umask, and not /etc/profile. A cron job runs with a very minimal environment and often a different umask again. A file created by a container uses the container's.

So when a shared directory keeps producing files that another service cannot read, do not change your own umask. Find the unit file, and either set UMask=0002 there or use the setgid bit from the next section.

C4 · The three special bits

Official docs: chmod(1) — setuid, setgid, sticky · inode(7) · credentials(7)

Beyond the nine bits there are three more. They sit in a fourth digit at the front, which is why you sometimes see four-digit modes like 4755.

Bit	Value	What it does
setuid	4	On a program: it runs as the file's owner, not as you. Ignored on directories and on shell scripts.
setgid	2	On a program: it runs as the file's group. On a directory: new files inside inherit that directory's group.
sticky	1	On a directory: an entry may only be deleted or renamed by the entry's owner, the directory's owner, or root — even by users who can write to the directory.

You already met setuid in Module 02 — that was passwd running as root so you can change your own password.

The other two are the ones you will actually configure.

setgid on a directory solves the shared-team-folder problem. Normally a new file gets your primary group, so files created by different people end up in different groups and nobody can read each other's work. With setgid, every new file inherits the directory's group instead.

The sticky bit solves the problem from Section C2 — that write access to a directory lets you delete anyone's files. /tmp has to be writable by everyone, which would otherwise mean anybody could delete anybody's temp files.

With the sticky bit set, an entry can only be removed or renamed by its own owner, the owner of the directory, or root.

Real-world analogy — the shared office fridge

/tmp is the shared fridge in an office kitchen. Everyone can put food in. Everyone can take their own food out.

Without the sticky bit, everyone could throw away anyone's lunch. The fridge is shared, so the door is unlocked, so nothing stops you.

The sticky bit is the rule written on the door: "only bin your own food." The fridge is still open to all. You still cannot touch what is not yours.

setgid on a directory is the shelf labelled "Marketing". Anything put on that shelf becomes marketing's, regardless of who carried it in. That is exactly what you want for a shared project folder — files belong to the project, not to whoever happened to create them.

Where the analogy stops working, and it is worth knowing. The kitchen rule relies on people being honest. The sticky bit is enforced by the kernel and cannot be argued with.

But it only protects deletion and renaming. If a file inside is world-writable, anyone can still open it and overwrite every byte. The rule says you cannot bin someone's lunch — it does not stop you eating it and putting the empty box back.

🧪 Exercise C4.1 — Find the special bits and see the sticky bit work

bash

# The sticky bit on /tmp - look for 't' at the very end
ls -ld /tmp

# setuid: look for 's' in the OWNER execute position
ls -l /usr/bin/passwd

# setgid on a directory, built by hand
cd /tmp && mkdir -p team && chmod 2775 team
ls -ld team
touch team/newfile.txt
ls -l team/newfile.txt

# And prove the sticky bit protects other people's files.
# Root creates a file in /tmp; you try to delete it.
sudo touch /tmp/roots_file.txt
ls -l /tmp/roots_file.txt
rm -f /tmp/roots_file.txt

sudo rm -f /tmp/roots_file.txt; rm -rf /tmp/team

✅ Expected result — click to reveal

plain text

$ ls -ld /tmp
drwxrwxrwt 10 root root 4096 Aug 20 11:45 /tmp

$ ls -l /usr/bin/passwd
-rwsr-xr-x 1 root root 68208 Mar 23  2026 /usr/bin/passwd

$ chmod 2775 team && ls -ld team
drwxrwsr-x 2 zaeem zaeem 4096 Aug 20 11:46 team

$ ls -l team/newfile.txt
-rw-r--r-- 1 zaeem zaeem 0 Aug 20 11:46 team/newfile.txt

$ ls -l /tmp/roots_file.txt
-rw-r--r-- 1 root root 0 Aug 20 11:47 /tmp/roots_file.txt
$ rm -f /tmp/roots_file.txt
rm: cannot remove '/tmp/roots_file.txt': Operation not permitted

What to read out of it.

/tmp is drwxrwxrwt. Look at the last character: t, where you would expect x. That is the sticky bit. And notice the permissions before it — rwxrwxrwx, writable by everyone. That combination is the whole point: a directory anyone can write to, where nobody can delete anyone else's files.

passwd is -rwsr-xr-x. The s in the owner's execute position is setuid, exactly as you saw in Module 02.

team is drwxrwsr-x. The s in the group execute position is setgid. Files created inside inherit the directory's group.

But look at the new file: -rw-r--r--, with no group write. The group was inherited; the permissions still came from your umask of 022 from Section C3. For a genuinely shared directory you need both — setgid on the directory and a umask of 002, which is why the production note below mentions UMask=0002 alongside chmod 2775.

And the last block is the sticky bit doing its job. Root's file is in /tmp. You have full write access to /tmp — it is rwxrwxrwx. And you were refused with Operation not permitted.

Note the errno. Section C2 said deleting is a directory operation, and by that rule you should have been allowed. The sticky bit is an extra check that runs afterwards, and it returns EPERM rather than EACCES. That distinction is exactly the one from Module 01: EACCES is a failed permission check, EPERM is an operation forbidden to you regardless.

Now imagine this at 500 hosts. Two habits come out of this section:

Audit setuid regularly. find / -perm -4000 -type f 2>/dev/null should return a short, familiar list. Anything unexpected, especially in /tmp or a home directory, is treated as a compromise indicator — it is a standard way for an attacker to keep root access.

Use setgid for shared directories. chmod 2775 on a team folder, plus UMask=0002 in the service unit, solves "everyone keeps creating files nobody else can write" permanently. Fixing it with a nightly chmod -R cron job — which is what many teams end up doing — treats the symptom forever.

🎯 Interview questions — Permissions

Q. Explain chmod 644. What do the numbers mean?

Three digits, one per group: owner, group, other. Each is the sum of read (4), write (2) and execute (1).

So 644 is: owner 6 = 4+2 = read and write; group 4 = read; other 4 = read. Written out, -rw-r--r--.

Then add the part that shows you understand the model rather than the arithmetic:

The kernel checks these groups in order and stops at the first match. Owner first, then group, then other. It does not combine them and it does not fall through.

That means mode 0460 — owner read-only, group read-write — leaves the owner unable to write even though they are also in the group. First match wins, not best match wins. Being able to state that is what separates understanding from memorising.

The details that separate candidates:

The bits mean different things on directories: r lists names, w creates and deletes entries, x allows you to traverse into it. Nine bits, two completely different meanings depending on file type.
Nine bits cannot express real team access. "These three people write, everyone else reads" is impossible. That is why POSIX ACLs (getfacl/setfacl) exist.
There is a fourth digit for setuid, setgid and sticky — so 4755 and 2775 and 1777 are all things you will meet.

Q. A process cannot read a file it owns and has read permission on. What do you check?

Work outward from the file, because the cause is usually not the file:

The whole path, not just the file. Every directory in the path needs execute permission to be traversed. namei -l /full/path/to/file prints the mode of each component and shows exactly which one refuses. This is the single most common cause and most candidates never mention it.
The user bits, not the group bits. If the process runs as the owner, only the owner's three bits apply. Group permissions are irrelevant to it. A file at 0460 denies its owner write access.
Which user is it really? Check /proc/PID/status for the effective UID. A systemd service with User= is not running as who you think, and DynamicUser=yes creates a different UID on every start.
SELinux or AppArmor. These deny independently of file permissions, and the denial does not appear in the file's mode at all. Check ausearch -m avc -ts recent or dmesg. Everything looks correct and access is still refused.
Is the path what you think it is? A symlink pointing somewhere else, or a filesystem mounted over the directory, or a different mount namespace if the process is containerised.

The details that separate candidates:

Naming namei -l immediately signals real experience.
Distinguishing EACCES from EPERM in the error. EACCES sends you to file modes and path traversal; EPERM sends you to capabilities, the sticky bit, or immutable attributes (chattr +i, which even root cannot write through until it is removed).
Remembering the container case: the file may exist on the host and not in the container's mount namespace at all.

Q. What are SUID, SGID and the sticky bit?

Three extra permission bits beyond the usual nine.

SUID (4) on an executable makes it run with the owner's identity rather than the caller's. /usr/bin/passwd is owned by root and SUID, which is how an ordinary user can update /etc/shadow. Shown as s in the owner's execute position.

SGID (2) on an executable does the same with the group. On a directory it does something different and far more useful: new files inside inherit the directory's group rather than the creator's primary group. That is the standard fix for shared team directories.

Sticky (1) on a directory restricts deleting and renaming to the entry's owner, the directory's owner, or root — even for users who can write to the directory. /tmp is 1777 for exactly this reason. Shown as t at the end.

The details that separate candidates:

Explain why the sticky bit is needed at all. Deletion is governed by the directory's permissions, not the file's — so without it, anyone who can write to /tmp could delete everybody's files. The sticky bit exists to patch that specific hole.
Know that SUID is ignored on shell scripts on Linux, deliberately, because of a race between checking the file and starting the interpreter. Claiming you can make a SUID shell script shows you have not tried it.
Name the modern alternative: capabilities. CAP_NET_BIND_SERVICE lets a program bind port 80 without being root at all. ping moved from SUID to CAP_NET_RAW for precisely this reason — much smaller blast radius.
Give the audit command: find / -perm -4000 -type f 2>/dev/null. An unexpected SUID binary is a standard indicator of compromise, because it is a common way for an attacker to retain root.

💽 Part D · Filesystems and mounts

D1 · The VFS — one set of verbs, many filesystems

Official docs: The /proc filesystem — kernel docs · open(2) · findmnt(8)

Everything so far has assumed one kind of filesystem. Your machine is running several at once, and they are not remotely alike.

ext4 and xfs store blocks on a disk. tmpfs lives entirely in memory. proc and sysfs make things up on demand, as you saw in Module 01. An NFS mount is on a different machine.

And yet cat works on all of them. So does open, read, write, ls and every program you have ever used.

That is the VFS — the Virtual File System. It is a layer inside the kernel that defines one set of operations: open, read, write, look up a name, list a directory. Every filesystem implements those operations its own way. Everything above the VFS only ever speaks the common set.

This is the same idea as the device driver in Module 01, applied one level up.

Real-world analogy — the front desk that takes every request the same way

A large organisation has one front desk. You go there for everything: a document, a room booking, a parking pass.

You fill in the same form every time. Name, what you want, sign here.

Behind the desk, those three requests go to completely different departments who work in completely different ways. One walks to an archive. One phones a building on another site. One types into a computer.

You never learn any of that. You filled in one form.

That is the VFS. open and read are the form. Behind them, ext4 reads a disk, tmpfs reads memory, nfs sends a network request, and proc runs a function and invents the answer.

Where the analogy stops working — and this is a real source of production bugs. At a front desk you can see whether your request will take a second or a week.

Through the VFS you cannot. read() on a file in page cache takes nanoseconds. The identical read() on an NFS mount whose server is down takes forever and puts your process in the unkillable D state from Module 02. Same call, same code, wildly different behaviour — and nothing in the interface warns you.

🧪 Exercise D1.1 — Count the filesystems you are already running

bash

# Every real (non-pseudo) filesystem, and what type it is
findmnt --real -o TARGET,SOURCE,FSTYPE | head -12

# How many different types are in use right now?
findmnt -no FSTYPE | sort -u

# The same operation on three completely different filesystems
echo "hello" > /tmp/vfs_disk.txt     # a real disk
echo "hello" > /dev/shm/vfs_mem.txt  # memory
cat /proc/uptime                     # invented on demand

wc -c /tmp/vfs_disk.txt /dev/shm/vfs_mem.txt
rm -f /tmp/vfs_disk.txt /dev/shm/vfs_mem.txt

✅ Expected result — click to reveal

plain text

$ findmnt --real -o TARGET,SOURCE,FSTYPE | head -12
TARGET                SOURCE      FSTYPE
/                     /dev/vda2   ext4
├─/proc               proc        proc
├─/sys                sysfs       sysfs
├─/dev                udev        devtmpfs
│ └─/dev/shm          tmpfs       tmpfs
├─/run                tmpfs       tmpfs
└─/boot/efi           /dev/vda1   vfat

$ findmnt -no FSTYPE | sort -u
cgroup2
devpts
devtmpfs
ext4
proc
securityfs
sysfs
tmpfs
vfat

$ cat /proc/uptime
5821.44 11302.87

$ wc -c /tmp/vfs_disk.txt /dev/shm/vfs_mem.txt
 6 /tmp/vfs_disk.txt
 6 /dev/shm/vfs_mem.txt
12 total

What to read out of it.

Nine different filesystem types are mounted on a plain VM that is doing nothing. They have almost nothing in common underneath:

ext4 writes to a virtual disk.
tmpfs is RAM. Reboot and it is empty.
proc and sysfs store nothing at all — Module 01 covered this.
vfat is a Microsoft format from the 1980s, still required by UEFI boot.
devtmpfs and devpts are device nodes managed by the kernel — devpts is where your terminal lives, and Module 04 comes back to it.
cgroup2 is the resource-control hierarchy, which Module 12 covers.

And echo > worked identically on the disk one and the memory one. wc -c reported 6 bytes for both. Nothing in the command, and nothing in the output, hints that one touched a disk and the other did not.

That is the VFS doing its job. It is also why the tree is a single tree rather than drive letters — everything gets grafted into one namespace, which is Section D2.

Interview-grade detail. The VFS is why "everything is a file" is a useful statement rather than a slogan. It means one set of tools works on everything: you can cat a device, grep kernel state, redirect into a socket, and use select/poll/epoll to wait on files, pipes, sockets and timers together.

The cost is that the interface hides how expensive an operation is. A read() that is normally instant can block forever on a dead NFS server, and the calling program has no way to tell the difference in advance.

D2 · Mounting — how the single tree is built

Official docs: mount(8) · findmnt(8) · proc(5)

Linux has no drive letters. There is exactly one tree, starting at /.

So how do several filesystems fit into one tree? You attach each one at a directory, and from that point on, that directory is that filesystem. That attachment is a mount, and the directory is the mount point.

Two consequences follow, and both cause real incidents:

Whatever was in the directory before is hidden. Not deleted. Hidden. Unmount and it reappears.
The mount point can be any directory. There is nothing special about /mnt or /media. Those are conventions.

Real-world analogy — the door into another building

Your house has a cupboard under the stairs. Ordinary cupboard, some boxes in it.

One day a builder connects the back of that cupboard to a completely different building. Now, when you open the cupboard door, you walk into that other building. Its rooms, its corridors, its rules.

Your boxes are still there. They are just behind the new doorway, and you cannot reach them while it exists. Disconnect the passage and there they are again, exactly as they were.

That is a mount. The directory is the doorway. The filesystem is the building on the other side.

It also explains the "disk full but the files are invisible" case from Section B4. If someone wrote 40 GB of logs into /var/log before the log volume was mounted over it, those 40 GB are still on the root filesystem, behind the doorway. df counts them and you cannot see or delete them until you unmount.

Where the analogy stops working. A builder cannot connect a doorway while you are standing in the cupboard.

Linux has the same rule, and it is the reason for the most common error in this whole area: umount: target is busy. If any process has its working directory inside, or any file open inside, the doorway cannot be removed. lsof +D /mount/point or fuser -vm /mount/point tells you who is standing in there.

🧪 Exercise D2.1 — Hide a directory under a mount, then get it back

bash

sudo mkdir -p /mnt/cupboard
echo "my important boxes" | sudo tee /mnt/cupboard/boxes.txt >/dev/null
ls -l /mnt/cupboard

# Mount a small memory filesystem OVER that directory
sudo mount -t tmpfs -o size=10M tmpfs /mnt/cupboard

echo "--- after mounting: where did boxes.txt go? ---"
ls -la /mnt/cupboard
findmnt /mnt/cupboard

# Put something in the new filesystem
echo "new stuff" | sudo tee /mnt/cupboard/newfile.txt >/dev/null
ls /mnt/cupboard

# Unmount and look again. Predict what you will see.
sudo umount /mnt/cupboard
echo "--- after unmounting ---"
ls -l /mnt/cupboard

sudo rm -rf /mnt/cupboard

✅ Expected result — click to reveal

plain text

$ ls -l /mnt/cupboard
-rw-r--r-- 1 root root 19 Aug 20 12:05 boxes.txt

--- after mounting: where did boxes.txt go? ---
total 0
drwxrwxrwt  2 root root   40 Aug 20 12:06 .
drwxr-xr-x  3 root root 4096 Aug 20 12:05 ..

$ findmnt /mnt/cupboard
TARGET        SOURCE FSTYPE OPTIONS
/mnt/cupboard tmpfs  tmpfs  rw,relatime,size=10240k

$ ls /mnt/cupboard
newfile.txt

--- after unmounting ---
-rw-r--r-- 1 root root 19 Aug 20 12:05 boxes.txt

What to read out of it.

boxes.txt disappeared the instant the mount happened, and came back the instant it was undone. Nothing deleted it and nothing restored it. It was behind the doorway the whole time, still using disk space on the root filesystem.

Notice that newfile.txt is gone after unmounting too. It was written into the tmpfs, which only existed while mounted. It was never on any disk.

findmnt on the mount point tells you three things at once: what is mounted there, what type it is, and the options it was mounted with. The size=10240k is the limit that Section D4 will use.

Now imagine this at 500 hosts. Two very common incidents come from exactly this behaviour.

Space used by hidden files. A service wrote to /var/lib/mysql before the data volume was mounted. df shows the root filesystem filling and du /var/lib/mysql shows almost nothing, because du can only see through the doorway. Diagnose it by bind-mounting the root elsewhere (mount --bind / /mnt/root) and looking at the path underneath.

A mount that failed silently at boot. The service starts, the mount is missing, and it happily writes to the root filesystem instead. Everything works — until the root disk fills a week later. This is why fstab entries for data volumes should not be nofail unless you genuinely mean it, and why a monitoring check for "is this path actually a mount point?" (mountpoint -q /data) is worth having.

D3 · df and du — two different questions

Official docs: df(1) · inode(7) · lsof(8)

These two commands are constantly compared and they are not measuring the same thing at all.

df — asks the filesystem

"How many blocks are allocated?"

One quick question to the filesystem's own accounting. Instant, whatever the size of the disk.

Counts everything, including files with no name left.

du — walks the tree

"Add up every file I can find."

Visits every directory and every file. Slow on large trees.

Can only count files that have a name and that you can reach.

So they disagree whenever a file is using blocks but cannot be found by walking the tree. Three cases cause that, and you have now seen all three:

Deleted but still open — Section B4. df counts it, du cannot see it.
Hidden under a mount point — Section D2. du walks the mounted filesystem, not what is underneath.
Directories you cannot enter — Section C2. du silently skips what it cannot read, so it under-reports.

Real-world analogy — the meter, and walking the building

You want to know how full a storage facility is. There are two ways to find out.

Ask the front office. They have one number, kept as units are rented and returned. Instant, and it covers the whole building.

Walk every corridor and add up what you find. Accurate for what you see. It takes all afternoon.

Normally these agree. They stop agreeing when something is occupying space but is not findable on the walk:

Someone is inside a unit whose name was crossed off the list. You walk past a door marked empty. The office still counts it. (deleted but open)
A corridor has been sealed off behind a new doorway. You never go down it. The office still counts everything down there. (hidden under a mount)
Some corridors are locked and you do not have a key, so you skip them and note nothing. (directories you cannot read)

In all three, the office number is right and your walk is short. df is the office. du is the walk.

Where the analogy stops working. Walking a building takes the same time whoever does it. du gets dramatically slower as the number of files grows, because it is doing one lookup per file — Module 01's Exercise C3.1 showed exactly this shape. On a filesystem with millions of small files, du -sh can take hours and generate real disk load. On a busy production host, that matters.

🧪 Exercise D3.1 — Make them disagree, and read the gap

bash

cd /tmp

# Normal case: du can account for the space df reports.
# Watch the USED column in df go up by about 50M.
df -h /tmp | tail -1
mkdir -p agree && dd if=/dev/zero of=agree/f1 bs=1M count=50 2>/dev/null
df -h /tmp | tail -1
du -sh /tmp/agree

# Now create the gap: a 300 MB file, deleted but held open
dd if=/dev/zero of=/tmp/ghost bs=1M count=300 2>/dev/null
sleep 300 < /tmp/ghost &
GHOST=$!
sleep 1
rm /tmp/ghost

echo "=== df says (asks the filesystem) ==="
df -h /tmp | tail -1
echo "=== du says (walks the tree) ==="
du -sh /tmp 2>/dev/null

echo "=== who is holding the difference? ==="
lsof +L1 2>/dev/null | grep ghost

kill $GHOST; sleep 2
echo "=== after releasing ==="
df -h /tmp | tail -1
rm -rf /tmp/agree

✅ Expected result — click to reveal

plain text

$ df -h /tmp | tail -1
/dev/vda2       9.8G  3.2G  6.1G  35% /
$ du -sh /tmp/agree
50M	/tmp/agree

=== df says (asks the filesystem) ===
/dev/vda2       9.8G  3.5G  5.8G  38% /
=== du says (walks the tree) ===
71M	/tmp

=== who is holding the difference? ===
sleep  7231 zaeem 0r REG 259,1 314572800 0 1443077 /tmp/ghost (deleted)

=== after releasing ===
/dev/vda2       9.8G  3.2G  6.1G  35% /

What to read out of it.

In the first block, df's used column rose from 3.1G to 3.2G and du accounts for it: 50 MB in, 50 MB visible. That is the normal case. Note that you compare the change in df, not its absolute figure — df reports the whole filesystem while du reports one directory, so the raw numbers were never going to match.

Then the interesting part. df says usage went up by 300 MB and stayed up after the rm. du -sh /tmp reports only 71 MB, because the 300 MB file has no name for it to find.

That gap — df high, du low, and neither of them wrong — is the signature of case 1. lsof +L1 names the culprit in one line, including the exact size (314572800 bytes) and the process holding it.

Killing the holder brings df straight back down to 35%. Nothing was deleted at that moment; the last descriptor simply closed.

The reasoning to give in an interview. Do not memorise "df counts deleted files". Give the reason:

"df asks the filesystem how many blocks are allocated. du walks the directory tree and adds up what it can find. So they differ exactly when something occupies blocks but has no reachable name — a deleted-but-open file, files hidden under a mount point, or directories du could not enter."

That single sentence covers all three causes and shows you understand the mechanism rather than the symptom.

D4 · Running out of inodes

Official docs: df(1) — the -i option · inode(7) · mount(8)

One last consequence of Section B1, and it produces the most confusing "disk full" message of all.

On ext4, the pool of inodes is fixed when the filesystem is created. A certain number of inodes, and that is that. You cannot add more later without rebuilding the filesystem.

Every file needs one inode, no matter how small. A million one-byte files use a million inodes and almost no disk space.

So you can run out of inodes while having plenty of blocks. Every attempt to create a file then fails with ENOSPC — "No space left on device" — while df -h cheerfully reports 90% free.

df -h shows blocks. df -i shows inodes. Anyone diagnosing a full disk should run both.

Real-world analogy — the office list has a fixed number of lines

The storage facility's office list is a printed ledger with 10,000 lines. That was decided when the building opened and nobody can add pages.

Now a customer rents 10,000 units and puts a single envelope in each.

The building is practically empty. Every unit has one envelope in it. There is enormous space.

And nobody can rent anything, because there is no line left in the ledger to write a name on.

Two staff members would report this completely differently. One measures floor space and says "we are 2% full." The other looks at the ledger and says "we are completely full." Both are correct. They are counting different things.

That is df -h versus df -i. Space and names are two separate resources, and either can run out on its own.

Where the analogy stops working, in a useful way. A printed ledger cannot be extended, and neither can an ext4 inode table. But XFS and Btrfs allocate inodes as they go, so they mostly do not have this failure mode at all. If you are choosing a filesystem for something that will hold millions of small files — a mail queue, a cache, a build directory — that difference is a real reason to pick one over the other.

🧪 Exercise D4.1 — Fill a disk with 99% of it free (meant to fail)

bash

# A small filesystem with a deliberately tiny inode allowance
sudo mkdir -p /mnt/tiny
sudo mount -t tmpfs -o size=50M,nr_inodes=100 tmpfs /mnt/tiny

echo "=== blocks: how much SPACE is free? ==="
df -h /mnt/tiny | tail -1
echo "=== inodes: how many NAMES are free? ==="
df -i /mnt/tiny | tail -1

# Now create small files until something breaks
sudo sh -c 'for i in $(seq 1 200); do
  echo x > /mnt/tiny/file_$i 2>/dev/null || { echo "FAILED at file number $i"; break; }
done'

echo "=== after the failure ==="
echo "space:"; df -h /mnt/tiny | tail -1
echo "names:"; df -i /mnt/tiny | tail -1

# The actual error message
sudo touch /mnt/tiny/one_more_file

sudo umount /mnt/tiny && sudo rmdir /mnt/tiny

✅ Expected result — click to reveal

plain text

=== blocks: how much SPACE is free? ===
tmpfs            50M     0   50M   0% /mnt/tiny
=== inodes: how many NAMES are free? ===
tmpfs           100     1    99    1% /mnt/tiny

FAILED at file number 100

=== after the failure ===
space:
tmpfs            50M  396K   50M   1% /mnt/tiny
names:
tmpfs           100   100     0  100% /mnt/tiny

$ sudo touch /mnt/tiny/one_more_file
touch: cannot touch '/mnt/tiny/one_more_file': No space left on device

What to read out of it — put the two df lines side by side.

df -h says 1% used. 50 MB free out of 50 MB.

df -i says 100% used. Zero names left.

And the error is "No space left on device" — the message that sends everyone straight to df -h, where they find nothing wrong and conclude the error is lying.

It is not lying. ENOSPC means "I cannot create this", and there are two separate ways for that to be true. The message does not distinguish them.

The 99 usable inodes went to 99 files (one is used by the directory itself), and file 100 failed.

Now imagine this at 500 hosts. Real inode exhaustion always comes from the same shape: something creating millions of small files with no cleanup.

The usual suspects are PHP session files, mail queues, unrotated per-request log files, and Docker layers on an ext4 root.

Find the offender: df -i to confirm, then

sudo find /var -xdev -type d -exec sh -c 'echo "$(ls -A "$1" | wc -l) $1"' _ {} \; 2>/dev/null | sort -rn | head

which lists the directories holding the most entries. -xdev keeps it on one filesystem.

And add it to monitoring. Almost every default disk alert watches blocks only. A host can be minutes from failing every write with df -h showing 20% used, and nothing will have paged. One extra check on df -i costs nothing.

🎯 Interview questions — Filesystems and mounts

Q. What is the difference between df and du, and why do they disagree?

They answer different questions. df asks the filesystem how many blocks are allocated — one quick lookup, instant regardless of size. du walks the directory tree and adds up the files it finds, which is slow and only sees files with names.

They disagree whenever something occupies blocks but cannot be reached by walking the tree. Three causes:

Deleted but still open. A process holds the file open, so rm freed nothing. df counts it, du cannot find it. Diagnose with lsof +L1, fix by restarting the process.
Files hidden under a mount point. Data written to a directory before a filesystem was mounted over it. Still occupying space, unreachable. Check by bind-mounting the parent elsewhere.
Directories du could not read. It skips them silently and under-reports. Run as root to rule this out.

The details that separate candidates:

Give the reason, not the rule. "df asks the filesystem, du walks the tree" explains all three cases in one sentence.
Mention that du is expensive. It is one lookup per file; on millions of files it takes hours and generates real I/O. It is not a free command on a production host.
Add the reserved-blocks case: ext4 keeps 5% for root by default, so a non-root process can get ENOSPC while df shows 5% free. tune2fs -m 1 reclaims most of it on a large data volume.

Q. df -h shows 20% used and writes are failing with "No space left on device". What is wrong?

Almost certainly inode exhaustion. Run df -i.

Every file consumes one inode regardless of size. On ext4 the inode pool is fixed when the filesystem is created, so millions of tiny files can exhaust the names while barely touching the blocks. Both conditions return the same ENOSPC, and the message does not say which one you hit.

Finding the cause: look for directories with huge entry counts — PHP session files, mail queues, unrotated per-request logs, cache directories, Docker layers on an ext4 root.

find /var -xdev -type d -exec sh -c 'echo "$(ls -A "$1" | wc -l) $1"' _ {} \; 2>/dev/null | sort -rn | head

Fixing it: delete the files, and fix whatever creates them. You cannot grow the inode table on ext4 without recreating the filesystem — this is genuinely a rebuild, which is why prevention matters.

The details that separate candidates:

Name the filesystem difference. XFS and Btrfs allocate inodes dynamically and largely avoid this. If a workload is known to be millions of small files, that is a real reason to choose XFS at build time.
Say it is almost never monitored. Default disk alerts watch blocks. Adding df -i to monitoring is a one-line change that catches an outage nothing else would.
Cover the other candidates too: deleted-but-open files (lsof +L1), and reserved blocks for root.

Q. What happens when you mount a filesystem over a directory that already has files in it?

The existing contents are hidden, not deleted. The directory becomes the entry point to the new filesystem, and the original files sit underneath, still occupying space on the original filesystem. Unmount and they reappear untouched.

Why this matters operationally — it causes two real incidents:

Invisible disk usage. A service wrote to /var/lib/mysql before the data volume was mounted. df shows the root filesystem filling up, du on that path shows almost nothing, because du only sees through the mount. Reach the hidden data with mount --bind / /mnt/root and look at the path underneath.
A mount that silently failed at boot. The service starts anyway and writes to the root filesystem. Everything works for a week, then root fills. A mountpoint -q /data check in monitoring catches this immediately.

The details that separate candidates:

Name the unmount failure and the fix. umount: target is busy means a process has a file open or a working directory inside. lsof +D /mount/point or fuser -vm /mount/point identifies it. Say why umount -l (lazy) is a last resort: it detaches the name immediately but the filesystem stays alive until the last user lets go, which can hide the real problem.
Mention mount namespaces. In a container, what is mounted is per-namespace. A path can exist on the host and not in the container, or be a completely different filesystem. Checking /proc/PID/mountinfo for the specific process is the reliable way to see what it sees.

🏁 Part E · Practice, capstone, reference and review

E1 · Production practice

Situation	What you now run	What it tells you
Disk full, df -h shows free space	lsof +L1, then df -i	Deleted-but-open files, or inodes exhausted. Two different fixes
Deleted a log by mistake, service still running	cp /proc/PID/fd/N /tmp/recovered	The file is still there. Recover it before the process restarts
"What is this process actually touching?"	ls -l /proc/PID/fd	Every file, socket and pipe it holds. Safe on a live process
"Too many open files"	ls -l /proc/PID/fd \\| awk '{print $NF}' \\| sort \\| uniq -c \\| sort -rn	What is piling up. Many sockets to one host means unclosed connections
Checking a service's real limit	cat /proc/PID/limits	The limit that applies to it — not your shell's ulimit -n
Cron job logs results but no errors	Look for 2>&1 before the >	Wrong order silently discards every error. Use > file 2>&1
Permission denied on a file it owns	namei -l /full/path/to/file	Permissions on every directory in the path. Usually a missing x
Service can read but not write its own file	Check the user bits, not the group bits	First match wins. Being the owner can give you less access
Shared directory keeps producing unreadable files	chmod 2775 dir, plus UMask=0002 in the unit file	setgid makes new files inherit the directory's group
Broken deploy: path exists, "No such file"	find /opt -xtype l	Symlinks whose target is gone. Classic after rotating release dirs
du shows nothing, df shows full	mount --bind / /mnt/root and look underneath	Files hidden under a mount point
umount: target is busy	lsof +D /mount/point or fuser -vm /mount/point	Who is standing inside. umount -l is a last resort, not a fix
Did the data volume actually mount?	mountpoint -q /data && echo yes	Catches a silently failed mount before the root disk fills
Security audit	find / -perm -4000 -type f 2>/dev/null	Every setuid binary. Anything unexpected is a compromise indicator

E2 · Capstone exercise

🧪 CAPSTONE — Three disk incidents, diagnosed from scratch

Create each fault, then diagnose it as if you had been paged at 3am with only "the disk is full".

bash

# ---------- INCIDENT 1: df and du disagree ----------
dd if=/dev/zero of=/tmp/incident1 bs=1M count=250 2>/dev/null
sleep 400 < /tmp/incident1 &
I1=$!
sleep 1; rm /tmp/incident1

df -h /tmp | tail -1
du -sh /tmp 2>/dev/null
#   a) df and du disagree. Explain WHY, from the mechanism.
#   b) One command finds the cause. Which, and what does its flag mean?
#   c) How do you free the space without rebooting?
#   d) Could you still recover the file's contents? How?

kill $I1 2>/dev/null; sleep 1

# ---------- INCIDENT 2: full disk, 99% free ----------
sudo mkdir -p /mnt/inc2
sudo mount -t tmpfs -o size=50M,nr_inodes=80 tmpfs /mnt/inc2
sudo sh -c 'for i in $(seq 1 200); do echo x > /mnt/inc2/f$i 2>/dev/null || break; done'
sudo touch /mnt/inc2/final

df -h /mnt/inc2 | tail -1
#   e) The error says "No space left on device" and df -h shows 1% used.
#      What is the second thing to check, and what will it say?
#   f) Why can't you just "add more" of whatever ran out, on ext4?
#   g) Which filesystem choice avoids this entirely, and why?

sudo umount /mnt/inc2; sudo rmdir /mnt/inc2

# ---------- INCIDENT 3: permission denied on a file you own ----------
mkdir -p /tmp/inc3/inner
echo "data" > /tmp/inc3/inner/target.txt
chmod 644 /tmp/inc3/inner/target.txt
chmod 600 /tmp/inc3/inner

cat /tmp/inc3/inner/target.txt
#   h) You own the file and it is mode 644. Why did that fail?
#   i) Which single command shows you exactly which step refuses?
#   j) What would change if the directory were 711 instead of 600?

chmod 755 /tmp/inc3/inner; rm -rf /tmp/inc3

✅ What a good answer looks like — click to reveal

You are marked on whether your reasoning explains the evidence, not on exact numbers.

a–d, the disagreement. df asks the filesystem how many blocks are allocated; du walks the tree and adds up files it can find by name. rm removed the name but a process still holds a descriptor, so the data is still allocated and du cannot see it. lsof +L1 finds it — +L1 means "link count below 1", which is exactly the definition of a deleted-but-open file. Free the space by restarting or killing the process holding it; no reboot needed. And yes, you can still recover the contents with cp /proc/PID/fd/N /tmp/recovered, because that descriptor is a live handle to the inode.

e–g, the full disk. Check df -i, which will show 100% inode usage. Every file needs one inode regardless of size, and on ext4 the inode table is fixed when the filesystem is created — so you cannot add more without recreating the filesystem. XFS and Btrfs allocate inodes dynamically and largely avoid this failure mode, which is a real reason to choose XFS for workloads with millions of small files. A complete answer also notes that ENOSPC is returned for both conditions, which is why the message misleads.

h–j, the permission denial. The file's own permissions were never the problem. x on a directory is what allows you to traverse it, and mode 600 removed it. The kernel refused at the directory step and never reached the file. namei -l /tmp/inc3/inner/target.txt prints the mode of every component and shows exactly which one refuses. At 711 the file would read fine by exact path, but ls on the directory would fail — you could walk the corridor without reading the sign.

Why this is the capstone. All three present as the same vague complaint, and all three are diagnosed by asking a different question rather than by trying harder at the same one. df vs du. Blocks vs inodes. The file vs the path to it.

That habit — noticing that two tools disagree and asking what each one actually measures — is the transferable skill. It is the same reasoning as df -h vs df -i, ps vs /proc, and load average vs CPU idle in Module 02.

E3 · Official documentation reference

Topic	Official page	Offline equivalent
Opening files, descriptors	open(2)	man 2 open
Standard streams	stdin(3)	man 3 stdin
Redirection mechanics	dup(2) — dup and dup2	man 2 dup
Pipes	pipe(7)	man 7 pipe
Descriptor limits	getrlimit(2)	man 2 getrlimit
Inodes — the whole model	inode(7)	man 7 inode
Hard links	link(2)	man 2 link
Symbolic links	symlink(2)	man 2 symlink
Reading file metadata	stat(1)	man 1 stat
How paths are resolved	path_resolution(7)	man 7 path_resolution
Permissions	chmod(1)	man 1 chmod
Default permissions	umask(2)	man 2 umask
Mounting	mount(8)	man 8 mount
Inspecting mounts	findmnt(8)	man 8 findmnt
Space and inode usage	df(1)	man 1 df
Open files across the system	lsof(8)	man 8 lsof
/proc/PID/fd	proc(5)	man 5 proc
The standard	POSIX.1-2024 Base Specifications Issue 8	man 7 standards

Where to start. Read man 7 inode first. It is the page this whole module is built on, and once the inode model is clear, hard links, df vs du, deleted-but-open files and cross-device errors all stop being separate facts to memorise.

Then man 7 path_resolution for the permissions half. It explains directory traversal properly, which is the thing most people learn by trial and error.

E4 · Self-assessment

Answer out loud, without scrolling up.

What is a file descriptor? Why is descriptor 3 in two different processes unrelated?
ls > out.txt — who opens out.txt, and how does ls end up writing into it without knowing?
Why does cmd 2>&1 > file behave differently from cmd > file 2>&1? Which one is the bug?
What is stored in an inode, and what two things are not?
Why can a hard link not cross a filesystem, and why can a symlink?
You rm a 10 GB log file and df does not change. Explain exactly why, and give the two commands you would run.
You own a file, it is mode 644, and you get "Permission denied". Give two different explanations and the command that distinguishes them.
A file is mode 0460 and you are both the owner and in its group. Can you write to it? Why?
df -h shows 15% used and every write fails with "No space left on device". What is your next command?
What happens to files in a directory when you mount a filesystem over it? Give one incident this causes.

E5 · Sources

Interview questions were taken from published 2026 question sets and then extended with operational detail beyond the published answers.

Technical content is sourced from the official documentation listed in E3.

Next: Module 04 — Signals, Sessions & Job Control. Module 02 gave you TERM and KILL as a working minimum. Module 04 covers the full set: what a signal handler is, why some signals cannot be caught, what Ctrl-C actually does, and why closing a terminal sometimes kills your job and sometimes does not.

Spotted a mistake or want something added? Send me a note.

🗂️Module 03 — File Descriptors, Inodes & Filesystems

🔢 Part A · File descriptors

A1 · What a file descriptor really is

A2 · The three you always get: 0, 1 and 2

A3 · Redirection — how > actually works

A4 · Pipes — wiring one process into another

A5 · Running out of file descriptors

🎯 Interview questions — File descriptors

🗂️ Part B · What a file actually is

B1 · The inode — a file's real identity

B2 · Directories are just lists of names

B3 · Hard links and symbolic links

B4 · Deleting a file that is still open

🎯 Interview questions — Files and links

🔒 Part C · Permissions

C1 · The nine bits, and how the kernel checks them

C2 · Directory permissions — the counter-intuitive one

C3 · umask — why new files are not 666

C4 · The three special bits

🎯 Interview questions — Permissions

💽 Part D · Filesystems and mounts

D1 · The VFS — one set of verbs, many filesystems

D2 · Mounting — how the single tree is built

D3 · df and du — two different questions

D4 · Running out of inodes

🎯 Interview questions — Filesystems and mounts

🏁 Part E · Practice, capstone, reference and review

E1 · Production practice

E2 · Capstone exercise

E3 · Official documentation reference

E4 · Self-assessment

E5 · Sources

Module 03 — File Descriptors, Inodes & Filesystems