Module 15 — Full Interview Simulation
Updated 22 August 2026
Fourteen modules of mechanism, and one question left: can you produce it out loud, under time pressure, in the order an interviewer expects? This module is four complete mock rounds — a rapid-fire screen, a "walk me through what happens when…" deep dive, a live broken-box session you can actually build and solve, and a design-and-incident round — each scored the way a real interviewer scores it.
🧠 round → 🧩 what is really being tested → 🧪 attempt it → ✅ model answer and rubric (hidden)
Work through it out loud. Reading a model answer and thinking "yes, I knew that" is the single most common way to fail an interview you were qualified for. Knowing and saying are different skills, and only one of them is being tested.
You will need a Linux machine you can break. Part D builds five faults from scratch with ordinary commands — a VM, a container, or a laptop you do not mind messing up. It touches /tmp, a loopback file, /mnt/lab, /srv/app and one directory under /sys/fs/cgroup. Check that /srv/app does not already exist on your machine before you start, because the clean-up in D7 removes it.
🗺️ Part A · How the loop actually works
A1 · The rounds, and what each one is really testing
The shape has been stable for a decade, and Google described it plainly in 2015: "a short technical pre-screen with a recruiter; next, an initial phone interview with an engineer, perhaps with a follow-up phone interview; and then a day at one of our sites, doing four or five interviews, each with an engineer."
The part that matters more than the shape is what comes next in the same paragraph: each interviewer is assigned a topic.
This single sentence should change how you prepare. You are not being assessed once by five people; you are being assessed on five separate axes, one per room. Being outstanding at troubleshooting does not compensate for being weak on internals, because the person who saw the troubleshooting is not in the room for the internals and will not write about it. Prepare each axis to a floor, rather than one to a ceiling.
And from the same paper, worth knowing before you walk in: "the interviewer provides a hire/no-hire recommendation along with detailed written feedback", and "the hire/no-hire decision is not made by a manager; instead, it's made by a hiring committee." Nobody in the room can hire you. They can only write down what you said.
A representative modern loop, assembled from what Google, Meta and mid-size companies publish:
| Round | Length | What it tests | Where in this track |
| Recruiter screen | 20–45 min | Vocabulary and motivation. SLI/SLO, error budgets, postmortems | Part E — vocabulary only; motivation and toil are out of scope here |
| Technical screen | 45–60 min | Rapid-fire fundamentals. Breadth, not depth — but a wrong answer here ends the process | Part B |
| Systems / OS internals | 45 min | "Walk me through what happens when…" One question, taken as deep as you can carry it | Part C |
| Live troubleshooting | 60–90 min | A deliberately broken machine, screen-shared, narrated. The round that most resembles the job | Part D |
| Design under failure | 45–60 min | Non-abstract design: real numbers, real failure modes, explicit trade-offs | Part E |
| Coding | 45 min | Still required. Usually SRE-flavoured — log parsing, bounded memory, rate limiting — not pure algorithms | Out of scope here |
| Incident / behavioural | 45 min | Ownership, blameless framing, and one concrete thing that changed afterwards | Part E |
Interviewers who run this format say so directly — "they don't care if you solve it in 10 minutes or 25 minutes." What is being watched is whether you scope before you dig, whether you say what you expect to see before you run the command, and whether you change your mind when the output disagrees with you.
Which means the highest-scoring possible move is one most candidates never make: narrate a hypothesis, run the check, and announce that you were wrong. That is the whole job, performed in miniature, and it is very hard to fake.
Everybody preparing for a driving test practises the manoeuvres: the parallel park, the hill start, the three-point turn. And every examiner will tell you the same thing — almost nobody fails on the manoeuvres. They fail on the ordinary driving in between.
What is actually being scored is a set of habits: mirror before signal, signal before manoeuvre, a glance over the shoulder at the right moment. The examiner cannot see inside your head, so the only evidence they have that you checked is that they saw you check. A candidate who looks in the mirror without moving their head has done the right thing and scored nothing for it.
A technical interview works identically. The interviewer cannot see you reason. They can only see what you say. The reasoning you do silently earns nothing — not because they are being unfair, but because they are writing feedback for a committee that was not in the room, and "seemed to know it" is not a sentence anyone can write down.
And the parallel holds at the other end too. The candidate who reverses into the space perfectly but never once looks over their shoulder is the one who fails, because the examiner is not scoring the parking. They are scoring whether you will be safe on a road they have not seen yet.
Where the analogy stops working. A driving test has one examiner and a published, uniform standard. An interview loop has five people, each covering a different topic, each writing separately for a committee — so an outstanding performance in one room genuinely does not travel to the next.
A2 · How you are actually scored
Very few companies publish their rubric. GitLab publishes its levelling ladder, which is more useful, because it tells you what the difference between grades actually is:
| Level | The published distinguishing line | What that sounds like in an interview |
| Associate | "Follow established processes and runbooks, and submit updates to improve them for others." | Knows the commands. Waits to be told which one |
| SRE | "Provides emergency response either by being on-call or by reacting to symptoms according to monitoring and escalation when needed" | Can drive an incident from a symptom to a cause |
| Senior | "Identifies significant projects that result in substantial improvements in reliability, cost savings and/or revenue" | Finishes the answer with "and here is how I stop it recurring across the fleet" |
| Staff | "Able to create innovative solutions that push GitLab's technical abilities ahead of the curve" | Reframes the question. Argues with the premise, correctly |
The gradient is worth saying out loud, because it is the thing you are being placed on: follows runbooks → responds to incidents → prevents classes of incident → sets direction. Almost every "what would you do?" question can be answered at any of those four levels, and the level you answer at is the level you get offered.
Given "the checkout service is erroring", two candidates:
"I'd check the database CPU and the query logs." — technically sensible, and scored as weak operational judgement, because users are still failing while you investigate.
"First I'd drain traffic from the failing region to stop user impact. Then I'd investigate the database." — same knowledge, strong signal, because stopping the bleeding came first.
This appears independently in Google's own troubleshooting guidance, in published interviewer notes, and in every incident-response framework in the industry. It costs one sentence and it is the cheapest points available in the entire loop. Say the mitigation out loud, every time, even when you are certain of the cause.
"I don't know, but here is how I would find out, and here is what I would expect each answer to rule out" is a good answer. It is not a fallback; on an unfamiliar system it is the correct answer, and senior interviewers score it as such.
A confident wrong explanation is worse than no explanation, because it tells the interviewer what you will do at 3 a.m. with an outage and no supervision. And Google's paper is explicit about what they are hunting for instead — it asks of a candidate whether they "can make reasonable assumptions and extrapolations from what they do know?"
Two phrases worth having ready: "I'm not certain — let me tell you how I'd check", and "I'd expect to see X; if I see Y instead, that rules out my theory and points at Z." The second one is the single highest-value sentence in a technical interview.
🧪 Exercise A1.1 — Score yourself before you start
Answer these five out loud, timed, before reading any of Parts B to E. Record yourself if you can bear it.
1. In 60 seconds: what does an SRE do that a systems administrator does not?
2. In 60 seconds: define SLI, SLO, SLA and error budget, and say which one
you would change first if reliability were declining.
3. In 90 seconds: a service is slow. No errors. Walk me through your first
five minutes.
4. In 90 seconds: what happens between pressing the power button and a
login prompt?
5. In 60 seconds: tell me about an outage you caused.✅ How to score what you just said — click to reveal
Do not grade the content yet. Grade the delivery, against the five things every interviewer in the loop is watching for:
1. Did you scope before you answered? Questions 3 and 5 are deliberately underspecified. "Slow for whom, since when, one host or all of them?" is not stalling; it is the first thing a strong candidate says, and its absence is the first thing a weak answer reveals.
2. Did you mitigate before investigating? In question 3, did the words "stop the user impact", "roll back", "drain", or "shed load" appear before the words "check the logs"? If not, that is the cheapest fix available to you and you should do it in every remaining exercise in this module.
3. Did you say what you expected to see? Not "I'd run vmstat" but "I'd run vmstat 1 and I'd expect r under nproc; if it is higher, it is CPU, and if b is non-zero instead, it is I/O." One is a list of commands, the other is a diagnosis. Interviewers write down the second.
4. Did you hedge honestly, or bluff? Anywhere you were unsure, did you say so and describe how you would check? False certainty is penalised harder than ignorance, and the penalty is invisible to you at the time.
5. Did you finish above your level? For question 5, did you stop at "we fixed it" — or did you get to "and here is the specific change that means it cannot happen again"? That last clause is the line between the SRE and Senior rows in the table above, and it is one sentence long.
Then, and only then, grade the content. Question 4 alone spans Modules 05, 02 and 04; if it came out as a vague sequence with no initramfs, no pivot_root, no PID 1, that is a reading list, not a failing.
Keep the recording. Do the same five at the end of this module and compare. The content will have moved a little. The delivery should have moved a lot, and the delivery is what is being scored.
⚡ Part B · Round 1 — the rapid-fire screen
B1 · Twenty-four questions, forty-five minutes
This round is breadth. Nothing here is hard if you have read Modules 01–14; the difficulty is answering in ninety seconds without waffling, and stopping when you are done.
A pub quiz and a written exam reward opposite behaviours. In the exam, everything you write is marks: pad it out, cover the alternatives, hedge. In the quiz, the answer is "1969" — and the team that says "well, it depends how you define the moon landing, because there's the Apollo programme and the question of whether you count…" gets nothing while the buzzer goes to someone else.
The technical screen is a quiz round. Its purpose is to establish, cheaply, that talking to you for another four hours is worth doing. Every question has a two-sentence answer, and the interviewer has twenty-four of them to get through.
This is the round where knowledgeable people over-perform themselves into a rejection: asked what a zombie process is, they give an eight-minute lecture on process reaping, cover four of the following questions in passing, and the interviewer runs out of time having tested three axes instead of ten. The transcript reads "couldn't get through the material."
Answer, stop, and let them ask the follow-up. The follow-up is where depth is rewarded — and it is their choice which direction to take it, which is information you want.
Where the analogy stops working. A quiz answer is right or wrong. Here the shape of a right answer still matters: "an orphan whose parent has not reaped it" scores higher than "a dead process", even though both are correct, because one of them shows you know the mechanism.
🧪 Exercise B1.1 — Set a timer for forty-five minutes
Answer all twenty-four out loud before opening a single toggle. Ninety seconds each, hard stop. Mark each one: got it / knew it but rambled / did not know.
Q1. What is a file descriptor? (Module 03)
A small non-negative integer that indexes a per-process table of open files. The kernel keeps the real state — the open file description, with its offset and status flags — and hands the process a number to refer to it by. 0, 1 and 2 are stdin, stdout and stderr by convention.
Weak: "It's a file that's open." Strong: adds that dup2 is how shell redirection works, that descriptors survive exec unless marked close-on-exec, and that "too many open files" is EMFILE against RLIMIT_NOFILE.
Q2. Hard link versus symbolic link — and what happens when you delete the original? (Module 03)
A hard link is a second directory entry pointing at the same inode. There is no "original": both names are equal, the inode has a link count, and the data is freed only when the count reaches zero and nothing has it open. Delete one name and the other works perfectly.
A symbolic link is a small file containing a path. Delete the target and the symlink still exists and is now dangling — every access fails with ENOENT.
Weak: "A hard link is a copy." Strong: notes that hard links cannot cross filesystems (an inode number is only meaningful within one) and usually cannot be made to directories. The link-count rule is also why rm on a file a process is writing frees no space until that process closes it: the name is gone, but the blocks are freed only once the count reaches zero and the last descriptor closes.
Q3. What is an inode? (Module 03)
The on-disk structure holding everything about a file except its name and its contents: type, permissions, owner, timestamps, size, link count, and the pointers to the data blocks. Names live in directories, which are just maps from name to inode number.
Weak: "Metadata about a file." Strong: draws the consequence — a filesystem can run out of inodes while still having free blocks, and df -h will cheerfully report 3% used while every write fails. df -i is the check, and you will build exactly that fault in Part D.
Q4. What is a zombie process, and how do you get rid of one? (Module 02)
A process that has exited but whose parent has not yet called wait(). The kernel keeps only the exit status and a slot in the process table, so it consumes no memory and no CPU — it is a receipt nobody has collected.
You cannot kill a zombie; it is already dead. You fix the parent, which is failing to reap. Kill or restart the parent and the zombies are reparented to PID 1, which reaps immediately.
Weak: "kill -9 it." Strong: says that a handful of zombies is harmless but thousands will exhaust the PID space, and connects it to containers — an application used as PID 1 that does not reap is exactly this bug, fixed with docker run --init or a real init in the image.
Q5. What are the process states in Linux? (Module 02, 07)
R runnable or running · S interruptible sleep, waiting for an event and killable · D uninterruptible sleep, almost always in a driver, not killable · T stopped by a signal · Z zombie. I for idle kernel threads on modern kernels.
Weak: lists them. Strong: points at D and says why it matters — D-state tasks count towards the load average on Linux, which is why a machine can show a load of 40 with 90% idle CPU. That is the highest-value sentence in this answer and it is a whole question of its own later.
Q6. fork versus exec? (Module 02)
fork duplicates the calling process — same memory (copy-on-write), same descriptors, new PID, returning 0 in the child and the child's PID in the parent. exec replaces the program running inside the current process, keeping the PID, the descriptors that are not close-on-exec, and the process's place in the tree.
Together they are how every process on the machine starts: the shell forks, the child execs.
Weak: "fork copies, exec runs." Strong: adds that the copy is copy-on-write so it is cheap, that clone is the underlying call and what varies is which resources are shared — which is how threads are made and, with CLONE_NEW* flags, how containers are made.
Q7. What signal does kill <pid> send, and how do SIGTERM, SIGKILL and SIGHUP differ? (Module 04)
Default is SIGTERM (15) — a polite request the process can catch and use to flush and shut down cleanly. SIGKILL (9) cannot be caught, blocked or ignored; the kernel destroys the process with no chance to clean up. SIGHUP (1) originally meant the terminal went away, and by convention now means "reload your configuration" for daemons.
Weak: "kill -9 is stronger." Strong: explains the operational consequence — reaching for -9 first means no flush, no clean unlock, no graceful connection drain, and it hides bugs. And that a process stuck in D state will not die from SIGKILL either, because it is not in a state where signals are delivered.
Q8. What happens when you press Ctrl-C? (Module 04)
The terminal driver recognises the INTR character and sends SIGINT to the foreground process group of that terminal's session — not to one process. That is why Ctrl-C kills a whole pipeline at once.
Weak: "It stops the program." Strong: distinguishes the three: Ctrl-C is SIGINT, Ctrl-Z is SIGTSTP and Ctrl-\ is SIGQUIT; notes that a process which ignores SIGINT simply carries on; and mentions that a background process reading from the terminal gets SIGTTIN instead — which is why a backgrounded interactive job mysteriously stops.
Q9. User space versus kernel space? (Module 01)
Two privilege levels enforced by the CPU. Kernel code can touch any memory and any hardware; user code cannot, and every attempt to reach hardware, files or the network must go through the kernel. Each process gets its own address space, so it cannot see any other process's memory.
Weak: "The kernel is privileged." Strong: says how the boundary is crossed — a system call, a deliberate trap that switches privilege level, and that this is why syscalls cost more than function calls, why strace can observe them, and why %sy in vmstat means work is happening on the kernel side of that line.
Q10. What is a system call, and how does it differ from a library call? (Module 01)
A system call is a request into the kernel — the only way to do anything the process cannot do itself. A library call is an ordinary function call inside your own address space; it may or may not make a system call underneath.
printf is a library call; the write it eventually performs is a system call. strlen never makes one.
Weak: treats them as the same thing. Strong: uses buffering as the example — printf writes into a userspace buffer and only calls write when it fills or is flushed, which is exactly why output can vanish when a program crashes, and why the same program's output interleaves differently through a pipe than to a terminal.
Q11. Difference between CPU load and CPU utilization? (Module 07)
Utilization is the fraction of time the CPU was busy — bounded at 100%. Load is a count of tasks wanting to run: on Linux, runnable plus uninterruptible-sleep tasks, unbounded, and averaged over 1, 5 and 15 minutes.
They can move in opposite directions. Load 40 with 90% idle CPU means forty tasks are blocked, not computing.
Weak: treats load as "CPU percentage". Strong: adds that load is meaningless without nproc — 8 is an emergency on a 2-core box and idle on a 64-core one — and that the modern replacement is PSI in /proc/pressure/, which measures stall directly instead of inferring it.
Q12. Load average reads 1.43, 2.34, 2.78. What does that tell you? (Module 07)
Averages over the last 1, 5 and 15 minutes. Read as a trend, this is falling — it was worse before and is recovering, so you are probably looking at the aftermath of an event rather than a live one.
Weak: "The load is about 2." Strong: says the trend first, then notes it must be divided by core count before it means anything, and that these are exponentially weighted moving averages with a lag of tens of seconds — so uptime answers "has this been going on a while?" and never "did something just start?". For that, vmstat 1 and the r column answer instantly.
Q13. VSZ versus RSS — and which one should alert? (Module 08)
VSZ is the size of the process's virtual address space: everything mapped, including memory it has reserved and never touched, files mapped in, and shared libraries. RSS is what is actually resident in physical memory right now.
VSZ can be enormous and mean nothing — a JVM or a Go runtime routinely reserves tens of gigabytes of address space it never uses.
Weak: alerts on VSZ. Strong: says neither is quite right for "how much memory is this process costing me", because RSS counts shared pages once per process — the honest answer is PSS from /proc/PID/smaps_rollup, and for "what would I get back if I killed it", Private_Dirty.
Q14. What is a page fault? Is it bad? (Module 08)
A trap taken when a process touches a virtual address the MMU cannot translate. The kernel then decides what to do.
A minor fault is resolved without disk — the page is already in memory, or it is a first touch of an anonymous page, or a copy-on-write. These happen constantly and are completely normal, tens to hundreds of thousands per second on a busy machine.
A major fault needs disk I/O. These are the expensive ones.
Weak: "Page faults are bad." Strong: inverts it — minor faults are how memory works and a process with zero of them is doing nothing; the metric worth alerting on is major faults, and the third kind, an invalid access, is what a segfault is.
Q15. free -m shows almost no free memory. Should you worry? (Module 09)
No. The kernel uses otherwise-idle memory as page cache, and free memory is wasted memory. The column that answers the question is available, which includes the cache the kernel can reclaim on demand.
Weak: alerts on free. Strong: notes that MemAvailable is an estimate the kernel computes for exactly this purpose, that on procps-ng 4 the used column is itself derived as total − available, and that the failure this hides is real: an alert on free pages someone about a perfectly healthy machine every single day until they mute it.
Q16. Explain the OOM killer. (Module 09)
When the kernel cannot satisfy an allocation and cannot reclaim enough, it picks a process and kills it rather than failing the allocation — because Linux overcommits by default and has already promised memory it does not have.
The victim is chosen by oom_score, driven mainly by resident memory, and tunable per process with oom_score_adj from −1000 to +1000.
Weak: "It kills the biggest process." Strong: distinguishes the global OOM killer from the far more common cgroup one — a container hitting memory.max gets killed by its own memcg while the host has plenty free — and identifies the fingerprint: exit code 137, 128 + 9, with a Killed process line in dmesg and an oom_kill counter in memory.events.
Q17. Does vm.swappiness=0 disable swap? (Module 09)
No. It makes the kernel strongly prefer reclaiming page cache over swapping anonymous pages, but it will still swap rather than invoke the OOM killer. Only swapoff disables swap.
Weak: "Yes, it turns swap off." Strong: argues the case both ways — a little swapping is healthy, because it evicts genuinely cold anonymous pages and buys you cache; what hurts is thrashing, which is a different symptom with a different fix. And notes that the range is 0–200 on modern kernels, not 0–100, with values above 100 permitted since 5.8.
Q18. What does %iowait actually measure? (Module 10, 14)
Time the CPU was idle with at least one I/O request outstanding. It is a subset of idle, not a fifth kind of busy.
Weak: "How much the disk is slowing us down." Strong: draws the two consequences. A busy CPU hides it completely — there is no idle time for it to be counted in — so a low %iowait never rules out a disk problem. And on virtualised or network-backed storage it can read zero even on an idle CPU, because the cost lands as system time instead. The numbers that tell the truth are r_await/w_await and aqu-sz from iostat -x, or /proc/pressure/io. Bonus: %util is meaningless on any device that serves requests in parallel.
Q19. Pipe versus Unix domain socket? (Module 11)
A pipe is a unidirectional byte stream between related processes (or, as a FIFO, between any two through a filesystem name). A Unix domain socket is bidirectional, supports datagram and sequential-packet modes as well as streams, allows many clients to one listener, and can pass file descriptors and peer credentials between processes.
Weak: "Sockets are for the network." Strong: notes that a UDS is faster than a loopback TCP socket because it skips the entire network stack, not because loopback skips checksums — loopback already skips those, and getting that reason right is a genuine differentiator. Also that PIPE_BUF (4096) is the atomicity limit for concurrent writers, and that writing to a pipe with no reader gets you SIGPIPE and exit 141.
Q20. What kernel features make a container? Namespaces versus cgroups? (Module 12)
Namespaces change what a process can see — its own PIDs, mounts, network stack, hostname, IPC objects, user IDs, cgroup view and clocks. cgroups limit what it can use — CPU, memory, I/O, PID count.
Neither is "containers". A container is an ordinary process a runtime has put into some namespaces, placed in a cgroup, stripped of capabilities, given a seccomp filter, and pointed at a different root filesystem.
Weak: "Docker uses namespaces and cgroups." Strong: states that there is no container object in the kernel — no container ID, no container_create() — which is why containers can be assembled by hand with unshare and why they leak in ways a VM does not. And that chroot is not a security boundary; runtimes use pivot_root.
Q21. Is a container a security boundary? (Module 12, 13)
It is several independent restrictions, not a boundary. Namespaces limit visibility; capabilities, seccomp, an LSM and the device cgroup limit authority. All of them share one kernel, so a kernel vulnerability is shared too.
Weak: "Yes, it's isolated." Strong: separates visibility from authority in exactly those words, then names what actually breaks out in practice — a mounted container socket, a writable hostPath, CAP_SYS_ADMIN, CAP_SYS_MODULE, hostPID plus CAP_SYS_PTRACE — and points out that most real escapes are misconfiguration, not vulnerability. Then offers the structural fix: user namespaces, so the container's root maps to an unprivileged host UID.
Q22. What does --privileged actually do? (Module 13)
Five things at once: grants all 41 capabilities; sets seccomp to unconfined; sets AppArmor or SELinux to unconfined; lifts the device cgroup so every host device is usable; and unmasks the hidden /proc paths while mounting /sys read-write.
Weak: "It gives the container root." Strong: says what it does not do — it does not put the container in the host's namespaces; --pid=host and friends are separate flags. So it is root on the host's kernel with a private view, and the framing is visibility is not authority. Then the concrete proof: /proc/kcore is unmasked, giving an ELF view of kernel memory, and /sys/kernel/uevent_helper becomes writable — which is a one-line escape.
Q23. What is strace, and would you run it in production? (Module 14)
It traces a process's system calls using ptrace, which stops the process twice per system call — once on entry, once on return. That mechanism is the whole answer: the measured slowdown is in the hundreds of times, so attaching it to a busy service can take the service down.
Weak: "Yes, it's a standard tool." Strong: two extra facts. -e trace= is a display filter, not a performance control — the process is still stopped on every call so strace can decide not to print it; tracing a syscall a process makes four times still costs 165×. And the production alternatives are perf trace (buffered) or bpftrace (aggregates in the kernel). strace is right for a process that is already stuck, idle, or in development.
Q24. Someone calls and says "my system is SLOW". What do you do? (Module 14)
Scope it first. Slow for whom, since when, every request or a fraction, this host or all of them? A single host in a pool behaving badly is a different investigation from the whole fleet, and it costs one question.
Then a fixed checklist, not a favourite command. uptime for the trend, dmesg | tail in case the kernel already said, vmstat 1 and mpstat -P ALL 1 for which resource, pidstat 1 for which process, iostat -xz 1, free -m, sar -n DEV 1, sar -n TCP,ETCP 1. Sixty seconds, and whole categories are eliminated.
Weak: "I'd run top." Strong: names the method — the USE method, one question each for utilization, saturation and errors across CPU, memory, disk and network — and says what a clean result means: not a dead end, but the finding that moves you from the host to the request path. Checking %steal early, because if the hypervisor is taking the CPU nothing you do on that box matters, is the detail that makes this answer sound like experience rather than revision.
✅ How to score Part B — click to reveal
Count only two things.
How many did you answer inside ninety seconds? Under 18 of 24 and the problem is not knowledge, it is compression — practise saying the two-sentence version and stopping. This is the most improvable skill in the entire module and it is worth more than any additional fact.
How many did you reach the Strong line on, unprompted? Those lines are not trivia. Each one is the sentence that tells an interviewer you have used the thing rather than read about it: D-state in the load average, minor faults being normal, %iowait being a subset of idle, -e trace= not being a performance control, visibility not being authority. Five or six of those, spread across the round, is what a hire looks like.
On the ones you missed, go back to the module, not to a summary. These twenty-four map to Modules 01–14 almost one per section, and a gap here is a section you skimmed. Re-reading the ✅ toggle of that section's exercise is usually enough, because the answer is in the output, and output is what you remember under pressure.
One habit to fix immediately if you saw it in yourself: answering the question you wish you had been asked. Asked what a zombie is, answer what a zombie is. The interviewer has twenty-three more and a page of topics to cover; running long does not demonstrate depth, it costs them coverage, and the feedback that gets written is "could not get through the material".
🧵 Part C · Round 2 — "walk me through what happens when…"
C1 · How to answer a traversal question at all
This round is one question that lasts forty-five minutes. "What happens when you type a URL and press enter." "What happens between the power button and a login prompt." "What happens when you run ls." They are the most-asked questions in systems interviewing and the worst-answered, for one structural reason: there is no natural place to stop, so most candidates stop in the wrong place.
The technique that fixes it is to answer in layers, out loud, and let the interviewer choose the depth:
| Layer | Length | What you do |
| 1 · The map | 30 seconds | Name the five or six stages, end to end, and stop. "Shall I go deeper on any of these?" |
| 2 · The walk | 3–4 minutes | A paragraph per stage, naming the mechanism and one artefact you can inspect |
| 3 · The dive | The rest | Whichever stage they pick. This is where depth is rewarded and only here |
It reads as instinct — every candidate feels that a longer answer demonstrates more knowledge. It does the opposite. A map first shows you have the whole shape in your head, and it hands the interviewer control of where to spend the time, which is the thing they actually want. Sprawling from the top down shows only that you know the beginning, because that is invariably where the clock runs out.
And it protects you. If your weakest stage is the one they pick, you find out at minute two with forty minutes left to reason honestly through it, rather than at minute forty with no time to recover.
Someone stops you in the street and asks how to get to the station.
The unhelpful answer is complete and precise and starts at your feet: "take the second left, no — the second left after the postbox, not the alley, that one's one-way now, then you'll see a bakery on the right, it used to be a butcher's…" Three minutes in, they still have no idea where the station is relative to where they stand, and you have not reached it.
The helpful answer is: "it's about ten minutes, straight down this road, left at the church, and it's opposite the big car park." Four landmarks, ten seconds, and they can now ask their own question — "which church?" — which is exactly the part they needed.
The second answer is not less knowledgeable. It comes from knowing the route so well you can compress it. That compression is the signal, and the interviewer is listening for it.
Where the analogy stops working. A stranger genuinely wants to reach the station. Your interviewer already knows the way, and is watching how you describe a journey you have both made — so the landmarks you choose, and the ones you skip, are the whole assessment.
C2 · "What happens between the power button and a login prompt?"
🧪 Exercise C2.1 — Give all three layers, timed, before opening the answer
Thirty seconds for the map. Four minutes for the walk. Then pick your own weakest stage and dive on it for five.
Diagram source
flowchart TD A["Power on"] --> B["Firmware<br>UEFI or BIOS"] B --> C["Bootloader<br>GRUB reads its config"] C --> D["Kernel unpacks<br>and initialises"] D --> E["initramfs<br>a temporary root in RAM"] E --> F["pivot_root<br>to the real root"] F --> G["PID 1 starts<br>systemd"] G --> H["Units reach<br>the default target"] H --> I["getty on a tty<br>login prompt"]
✅ Model answer — all three layers — click to reveal
Layer 1, the map, thirty seconds.
"Firmware initialises the hardware and finds something bootable. A bootloader loads the kernel and an initramfs into memory and jumps into the kernel. The kernel brings up its own subsystems, mounts the initramfs as a temporary root, and uses it to find and mount the real root filesystem. It switches to that root and executes PID 1 — on most distributions, systemd. systemd starts units until it reaches the default target, one of which puts a getty on a terminal, and that is your login prompt. Which of those would you like me to go into?"
That is the whole answer. Stop there and ask.
Layer 2, the walk.
Firmware. UEFI on anything modern; BIOS on older machines. It initialises CPU, memory and buses, runs its power-on self test, then looks for a bootloader — on UEFI, an .efi binary on the EFI System Partition, listed in NVRAM boot entries; on BIOS, code in the first 512 bytes of a disk. If Secure Boot is on, the firmware verifies the signature before executing anything, which is why an unsigned kernel module fails to load later.
Bootloader. GRUB reads its configuration, presents the menu, and loads two things into memory: the kernel image and the initramfs. It passes the kernel a command line — root=, ro, quiet, and anything else set in the config. You can read that command line back at any time from /proc/cmdline, which is the artefact to name here.
Kernel. Decompresses itself, sets up memory management and the scheduler, initialises built-in drivers, and mounts the initramfs as its root. Everything up to this point is recorded in the kernel ring buffer, readable with dmesg.
initramfs. A small compressed archive containing just enough userspace to find the real root — the drivers for your storage controller, LVM or RAID assembly, filesystem modules, disk decryption. It exists to solve a bootstrapping problem: the kernel cannot mount a root filesystem it has no driver for, and the driver is on the root filesystem.
Switch root. Once the real root is mounted, the system pivot_roots onto it and the initramfs is discarded. Not chroot — pivot_root, which actually changes the mount tree, the same distinction that matters for containers in Module 12.
PID 1. The kernel executes the init binary. It is PID 1 and it is special: it is the ancestor of every other process, it adopts orphans, and signals to it have no default action — a kernel refusal to let the machine be killed by an unhandled SIGTERM. On most systems it is systemd.
systemd. Resolves the dependency graph and activates units in parallel until it reaches default.target — usually multi-user.target or graphical.target. systemctl list-dependencies default.target shows the tree, and systemd-analyze blame shows what took the time.
Login prompt. getty opens a terminal device, sets it up as a new session with a controlling terminal, prints the issue file, and executes login. That session and process-group machinery is exactly what Module 04 covered, and it is why Ctrl-C later signals a whole process group.
Layer 3, the dive — where interviewers usually push.
"Why is there an initramfs at all?" The bootstrapping problem above. The good follow-through is that you can inspect it: lsinitramfs on Debian, lsinitrd on RHEL — and that "boots on one kernel but not the newly-installed one" is very often an initramfs missing a storage driver.
"What if PID 1 dies?" The kernel panics. There is nothing left to reparent orphans to. In a PID namespace, the equivalent is that the namespace is torn down and every process in it is killed — which is why a container whose entrypoint exits takes the whole container with it.
"Where would you look if the boot hung?" dmesg for the kernel phase, journalctl -b for the systemd phase, systemd-analyze blame and critical-chain for slow units, and the kernel command line in /proc/cmdline to confirm what was actually asked for versus what GRUB shows.
What is being scored. Naming pivot_root rather than chroot. Knowing why the initramfs exists rather than that it does. Naming an artefact you can read for each stage — /proc/cmdline, dmesg, journalctl -b — because that is the difference between a story and a diagnosis. And offering the map first.
C3 · "What happens when you run ./app?"
🧪 Exercise C3.1 — Answer it, then check yourself against the machine
Give the three layers first. Then run this and see how much of your answer is visible in the output.
# Every system call a trivial program makes, from exec to exit
strace -f -e trace=execve,openat,mmap,mprotect,brk,write,exit_group \
/bin/echo hello 2>&1 | head -25
# The address space of a running process, in address order.
# Note this is `awk`'s own map, not echo's - a process cannot show you
# the address space of one that has already exited.
awk '{print $1, $2, $6}' /proc/self/maps | head -12
# Which libraries would be loaded, and how they are found
ldd /bin/echo✅ Model answer — all three layers — click to reveal
Layer 1, the map, thirty seconds.
"The shell forks a child. The child execs the binary, which replaces its program image. The kernel parses the ELF header, sets up a fresh address space, and — for a dynamically linked binary — hands control to the dynamic linker rather than to the program. The linker maps the shared libraries in and resolves symbols, then jumps to the entry point. From there the program runs, faulting pages in on demand, until it exits and the parent reaps it. Where would you like me to go deeper?"
Layer 2, the walk.
The shell resolves the name. ./app is a path, so no lookup; a bare app is searched along PATH and possibly a shell hash table. execve on the wrong thing is where command not found versus Permission denied come from, and they are different failures.
fork. The shell duplicates itself. Same descriptors, same memory — but copy-on-write, so nothing is actually copied until one side writes. This is why forking a 10 GB process is fast, and why the child briefly shares every page with the parent.
Redirection happens in the gap. Between fork and exec, in the child, the shell rearranges file descriptors with dup2 — this is what > out.log and 2>&1 are. It has to be here: after exec the program is a different program, and before fork it would affect the shell itself.
execve. The kernel checks permissions — the execute bit, the mount's noexec, and any LSM — then discards the entire address space and builds a new one from the ELF program headers: text mapped read-execute, data read-write, BSS zeroed. Descriptors survive unless marked close-on-exec. The PID does not change, which is why exec in a shell script replaces the shell in place.
The dynamic linker runs first. For a dynamically linked binary, the ELF header names an interpreter — /lib64/ld-linux-x86-64.so.2 — and the kernel transfers control there, not to main. The linker maps in the libraries listed in DT_NEEDED, resolves symbols, applies relocations, and only then jumps to the program's entry point. ldd shows you what it will load; LD_DEBUG=libs shows you it happening.
Then the program runs, and almost nothing is in memory yet. Pages arrive on demand, as minor page faults, the first time each is touched. /proc/PID/maps is the address space; /proc/PID/smaps_rollup is what it actually costs.
Exit. The program calls exit_group. The kernel tears down the address space, closes descriptors, reparents any children to PID 1, and keeps the exit status until the parent waits — the brief window in which it is a zombie. The shell reaps it and puts the status in $?.
Layer 3, the dive — the follow-ups that come up most.
"What is #!/bin/sh doing?" The kernel recognises the two-byte magic and execs the interpreter instead, passing the script path as an argument. Which is why file capabilities and setuid bits do not work on scripts — the thing being executed is the interpreter, and giving it the privilege would give it to every script on the machine.
"Static versus dynamic?" Statically linked binaries skip the linker entirely — faster start, no ldd output, no shared-library security updates. It is the same trade-off as a scratch container image.
"Where does the memory go?" /proc/PID/maps shows the regions; RSS counts shared pages once per process, so PSS is the honest per-process number.
"Why is the first run slow and the second fast?" Page cache. The first run's library reads are major faults from disk; the second run finds them cached and takes minor faults instead.
What is being scored. Naming the fork/exec gap as where redirection lives. Knowing that the dynamic linker gets control before main. Saying "copy-on-write" and "demand paging" as mechanisms rather than as vocabulary. And, as always, an artefact per stage — strace, /proc/PID/maps, ldd, $?.
🔧 Part D · Round 3 — the broken box
D1 · How the round works, and how it is scored
You get SSH into a machine that has been deliberately broken, sixty to ninety minutes, and an interviewer watching your screen. The scenario is usually stated as a symptom, not a fault: "the web server returns 502", "the deploy fails with disk full", "this pod keeps restarting".
Everything about your score lives in the protocol, not the fix:
| Step | What to say out loud | Why it scores |
| 1 · Scope | "Is this one host or all of them? When did it start? Is anyone still being affected?" | Establishes blast radius before you touch anything |
| 2 · Mitigate | "If this were live I'd roll back / drain / fail over first, then investigate" | The single cheapest point in the loop. Say it even in a lab |
| 3 · Hypothesise | "I think it's X. If it is, <command> will show Y" | Turns a command list into a diagnosis |
| 4 · Test | Run it. Read it aloud. | The interviewer is watching whether you actually read the output |
| 5 · Update | "That's not what I expected, so X is ruled out. That points at Z" | The highest-scoring sentence available to you |
| 6 · Fix, then prevent | "Here's the fix. Here's the monitor that would have caught it, and here's the fleet-wide version" | The line between the SRE and Senior rows in A2 |
Silence. Ten minutes of confident typing with no narration scores near zero, because the interviewer has nothing to write down. If you need to think, say "give me a moment, I'm going to check X because Y."
Fixing without diagnosing. Restarting the service makes the symptom go away and ends your ability to learn anything. If you must, say "restarting would clear this, but I'd lose the evidence — let me capture X first." That sentence alone separates candidates.
Never revisiting the first theory. The commonest failure is picking a hypothesis in minute two and spending forty minutes confirming it. Say what would disprove you, and check that.
Two plumbers, one damp patch on a kitchen ceiling.
The first goes straight upstairs and starts cutting into the bathroom floor, because it is always the bath seal. Sometimes they are right and it is a very impressive twenty minutes. When they are wrong, there is now a hole in the floor, the evidence is disturbed, and they are working from a worse position than when they arrived.
The second stands in the kitchen and says what they are doing: "the stain's directly under the bathroom, so I'd start there — but it's dry to the touch, which for a live leak I wouldn't expect. Before I open anything, I'll run the bath for two minutes and watch. If it darkens, it's the seal. If it doesn't, it's the pipe run and the leak is somewhere along the wall." Then they run the bath and watch.
The homeowner learns more from the second plumber in four minutes than from the first in an hour — and crucially, learns it whether or not the theory turns out to be right. Being wrong out loud is informative. Being right silently is not.
And notice the second plumber's first move was to stop the water, not to find the leak.
Where the analogy stops working. The plumber is trying to fix the ceiling. You are being watched to find out how you will behave on a fault nobody has seen yet — so the fix is the part the interviewer cares about least.
D2 · Lab 1 — "The disk is full, but it isn't"
🧪 Exercise D2.1 — Build it, then solve it
Run the setup, then treat what follows as the interviewer's opening line. Do not read the answer first.
# --- SETUP (as root). Everything lives in a 64 MB loopback file. ---
dd if=/dev/zero of=/tmp/disk.img bs=1M count=64 status=none
mkfs.ext4 -q -F /tmp/disk.img
mkdir -p /mnt/lab && mount -o loop /tmp/disk.img /mnt/lab
dd if=/dev/zero of=/mnt/lab/app.log bs=1M count=40 status=none
python3 -c '
import time, os
f = open("/mnt/lab/app.log", "r") # the "application" holds it open
os.unlink("/mnt/lab/app.log") # "log rotation" deletes it
time.sleep(900)
' &
sleep 2The interviewer says: "The application on this box can't write. The team says they already cleaned up the logs, and du shows the directory is basically empty — but the disk is still full. Go."
df -h /mnt/lab
du -sh /mnt/lab✅ Worked answer and rubric — click to reveal
Filesystem Size Used Avail Use% Mounted on
/dev/loop0 56M 41M 12M 78% /mnt/lab
20K /mnt/labSay the contradiction out loud first. df says 41 MB used; du says 20 KB present. They disagree by 41 MB — essentially everything df believes is in use — and that specific disagreement has a short list of causes:
1. A deleted file still held open — by far the most common. unlink removes the name; the inode and its blocks survive until the last file descriptor is closed. du walks names and sees nothing; df reads the superblock's free-block count and sees the truth.
2. Something mounted over a populated directory, hiding files that still occupy blocks.
3. Reserved blocks — ext4 reserves 5% for root by default, which makes df and "space I can use" differ, though not by this much.
Then test the first theory, and say what you expect: "if it's a deleted-but-open file, lsof -a +L1 will list a file with a link count of zero. The -a matters: lsof ORs its selection criteria by default, so lsof +L1 /mnt/lab without it dumps every deleted-but-open file on the machine."
sudo lsof -a +L1 /mnt/labCOMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
python3 23875 root 3r REG 7,0 41943040 0 12 /mnt/lab/app.log (deleted)Read it aloud, field by field, because every field is the diagnosis. NLINK is 0 — no name points at this inode. SIZE/OFF is 41943040, exactly the 40 MB df cannot account for. FD is 3r, a specific descriptor. And the culprit is named: PID 23875.
The fix, in order of preference. Best: get the process to close and reopen its log — a SIGHUP the application handles, or logrotate's postrotate sending one. If the application cannot be told to reopen at all, copytruncate is the fallback: it copies the file and then truncates the same inode in place, so no descriptor is ever orphaned. Acceptable: restart it. Emergency, if you cannot restart: truncate through the descriptor without killing anything —
: > /proc/23875/fd/3— which frees the blocks immediately, because the descriptor still refers to the inode.
Then prevent it, which is where the round is actually won. The bug is a rotation scheme that deletes a file the writer still holds. Use copytruncate, or make the application reopen on SIGHUP and have logrotate send it. Fleet-wide, the monitor is not "disk usage" — it is a check that flags any filesystem where df used and du total diverge by more than a few percent, which catches this class before anyone is paged.
Rubric. Weak: runs du again in different directories. Hire: names deleted-but-open and finds it with lsof -a +L1, or with the no-lsof fallback — for p in /proc/[0-9]*/fd/*; do case "$(readlink "$p" 2>/dev/null)" in *"(deleted)") echo "$p -> $(readlink "$p")";; esac; done. (find /proc/*/fd -xtype l does not work: /proc/PID/fd/N are magic links the kernel resolves straight to the inode, so they are never dangling.) Then restarts the process. Strong hire: says the mitigation before investigating, reads NLINK 0 out loud as the proof, knows the /proc/PID/fd truncate trick for when a restart is not allowed, and finishes with the rotation fix and the fleet check.
D3 · Lab 2 — "No space left on device", with 99% free
🧪 Exercise D3.1 — Build it, then solve it
# --- SETUP (as root) ---
# Lab 1's python still holds a descriptor on that filesystem, so a plain
# `umount 2>/dev/null` fails silently and the next mount STACKS on top.
pkill -f 'time.sleep(900)'
while mountpoint -q /mnt/lab; do umount /mnt/lab; done
losetup -D
rm -f /tmp/disk.img
dd if=/dev/zero of=/tmp/disk.img bs=1M count=32 status=none
mkfs.ext4 -q -F -N 512 /tmp/disk.img # deliberately few inodes
mount -o loop /tmp/disk.img /mnt/lab
mkdir -p /mnt/lab/spool
for i in $(seq 1 600); do : > /mnt/lab/spool/msg-$i; done 2>/dev/nullThe interviewer says: "Writes to this volume are failing with 'No space left on device'. Ops swear there's plenty of room. Go."
touch /mnt/lab/newfile; echo "exit: $?"
df -h /mnt/lab✅ Worked answer and rubric — click to reveal
touch: cannot touch '/mnt/lab/newfile': No space left on device
exit: 1
Filesystem Size Used Avail Use% Mounted on
/dev/loop0 28M 40K 26M 1% /mnt/labAgain, say the contradiction. ENOSPC on a filesystem that is 1% full. Blocks are not the constraint, so something else the filesystem allocates has run out — and on ext4 there is one obvious candidate.
Hypothesise and say what you expect: "I think we're out of inodes. df -i will show IUse% at 100."
df -i /mnt/labFilesystem Inodes IUsed IFree IUse% Mounted on
/dev/loop0 512 512 0 100% /mnt/labThere it is. Every file consumes one inode regardless of size, and the count is fixed at mkfs time. Half a million empty files will exhaust a filesystem that is empty by every other measure. This is why df -h and df -i are two commands and not one.
Then find where they went, rather than guessing:
for d in /mnt/lab/*/; do echo "$(find "$d" | wc -l) $d"; done | sort -rn501 /mnt/lab/spool/
1 /mnt/lab/lost+found/The fix. Delete the files, if they are stale — but the real answer is that something is creating them and not cleaning up: an unrotated mail spool, a session directory, a cache with no eviction, a retry queue with no dead-letter path. Fixing the count without fixing the producer buys you a week.
And the uncomfortable part, which is a strong signal to raise unprompted: on ext4 the inode count is fixed relative to filesystem size. resize2fs adds inodes only when it adds whole block groups, and you can never raise the inode density — the bytes-per-inode ratio — in place. So on a volume you cannot grow, the count really is fixed, and the permanent fix is a rebuild with mkfs.ext4 -i or -N, or a filesystem that allocates inodes dynamically — XFS and Btrfs both do. If the interviewer asks what you would choose for a mail or cache volume next time, that is the answer.
Rubric. Weak: keeps running df -h and insists the disk is fine. Hire: reaches df -i quickly and finds the directory. Strong hire: says which other resource could produce ENOSPC before running anything, names the producer as the real bug, and knows that ext4's inode density is fixed at mkfs time while XFS and Btrfs allocate inodes dynamically.
D4 · Lab 3 — "I'm root and I can't edit this file"
🧪 Exercise D4.1 — Build it, then solve it
# --- SETUP (as root) ---
mkdir -p /srv/app && echo "port=8080" > /srv/app/app.conf
chattr +i /srv/app/app.confThe interviewer says: "Config change won't apply. The deploy is failing on this file and nobody can work out why. You have root. Go."
ls -l /srv/app/app.conf
id -u
echo "port=9090" > /srv/app/app.conf; echo "write exit: $?"
rm -f /srv/app/app.conf; echo "rm exit: $?"✅ Worked answer and rubric — click to reveal
-rw-r--r-- 1 root root 10 Aug 22 13:24 /srv/app/app.conf
0
-bash: /srv/app/app.conf: Operation not permitted
write exit: 1
rm: cannot remove '/srv/app/app.conf': Operation not permitted
rm exit: 1Read the two facts together and say them out loud: uid 0, mode rw- for the owner who is root — and Operation not permitted. By the permission bits this write must succeed. It did not, which means the refusal did not come from the permission bits at all.
And name the errno, because it narrows the search immediately. Operation not permitted is EPERM, which is what a privileged check returns. Permission denied — EACCES — is what mode bits return. Getting EPERM on a file whose mode plainly allows the write is the fingerprint of something above the DAC layer.
The candidate list, in the order worth checking:
1. A file attribute — chattr +i (immutable) or +a (append-only). Invisible in ls -l, which is exactly why it wastes hours.
2. A read-only mount — but that returns EROFS, "Read-only file system", not EPERM. Ruled out by the message.
3. An LSM — AppArmor or SELinux denying the write. Would show in ausearch -m AVC -ts recent or dmesg.
4. A missing capability — irrelevant here, since we hold all of them.
lsattr /srv/app/app.conf----i---------e------- /srv/app/app.confThe i is the answer. The immutable attribute means the file cannot be modified, deleted, renamed, or hard-linked to — by anyone, including root. Clearing it needs CAP_LINUX_IMMUTABLE:
chattr -i /srv/app/app.conf
echo "port=9090" > /srv/app/app.conf && echo "write OK"Then the question that gets you the offer: *why was it immutable? Nobody sets +i by accident. Either someone was stopping a misbehaving process from rewriting the file — in which case removing it re-enables that bug — or a hardening or compliance tool set it deliberately and will set it again on its next run. Find out before you clear it*, and say so.
Prevent it. lsattr belongs in your config-drift audit alongside ls -l, and the deploy tooling should fail with a comprehensible message instead of a bare Operation not permitted.
Rubric. Weak: checks ls -l, sees root can write, becomes stuck, tries chmod 777. Hire: separates EPERM from EACCES, works down the list, finds it with lsattr. Strong hire: names the errno distinction unprompted, rules out the read-only mount by the message rather than by checking, and refuses to clear the attribute before finding out who set it.
D5 · Lab 4 — "The service won't start"
🧪 Exercise D5.1 — Build it, then solve it
# --- SETUP: something is already holding the port ---
python3 -m http.server 8080 --bind 127.0.0.1 >/dev/null 2>&1 &
sleep 1The interviewer says: "This service used to come up fine. Since the last deploy it exits immediately. Logs are unhelpful. Go."
# "the service" starting up
python3 -m http.server 8080 --bind 127.0.0.1✅ Worked answer and rubric — click to reveal
Traceback (most recent call last):
...
File "/usr/lib/python3.11/socketserver.py", line 472, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in useRead the error rather than reacting to it. EADDRINUSE on bind, which has exactly three causes and they need different fixes:
1. Another process is genuinely listening on that port. Most likely: an old instance the deploy did not stop, or a second copy started by a stale unit.
2. A socket in TIME_WAIT from the previous instance, and the program did not set SO_REUSEADDR. Distinguishable because there is no LISTENer, only lingering connections.
3. Two things legitimately configured for the same port — a port collision introduced by the deploy itself.
Say which one you expect, then find the owner:
ss -tlnp 'sport = :8080'State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 5 127.0.0.1:8080 0.0.0.0:* users:(("python3",pid=24643,fd=3))That settles it: cause 1. There is a LISTENer, so it is not TIME_WAIT, and ss -tlnp names the process and even the descriptor. If ss is unavailable, lsof -i :8080, or the no-tools fallback from Module 11 — grep /proc/net/tcp for the port in hex (8080 is 1F90) and match the inode against /proc/*/fd.
Before killing it, establish what it is. ps -fp 24643, ls -l /proc/24643/exe, and cat /proc/24643/cgroup — the last one tells you whether it belongs to a systemd unit or a container, which changes the correct way to stop it. Killing a process that systemd will immediately restart is a good way to spend twenty minutes confused.
Then the prevention, which is the real finding. A deploy that starts a new instance without confirming the old one exited is the bug; the port conflict is the symptom. The fixes are a service manager that owns the lifecycle, a readiness check that fails loudly rather than a stack trace, and — if the process should tolerate its own restarts — SO_REUSEADDR so a TIME_WAIT socket never blocks a rebind.
Rubric. Weak: changes the port to 8081 and declares victory. Hire: finds the owner with ss -tlnp and stops it properly. Strong hire: lists the three causes before running anything, uses the presence of a LISTENer to eliminate TIME_WAIT, checks the cgroup before killing, and names the deploy lifecycle as the actual defect.
D6 · Lab 5 — "The container keeps restarting"
🧪 Exercise D6.1 — Build it, then solve it
# --- SETUP: a memory-limited process that grows without bound ---
# With Docker:
# docker run --rm -m 64m python:3-alpine python -c \
# 'a=[]
# while True: a.append(bytearray(4*1024*1024))'
#
# Without Docker, on cgroup v2 ONLY. Module 12's precondition applies:
# on a v1 or hybrid box /sys/fs/cgroup is plain tmpfs, mkdir and tee
# both succeed, no limit is applied, and this becomes a memory bomb.
[ "$(stat -fc %T /sys/fs/cgroup)" = cgroup2fs ] || {
echo "This box is $(stat -fc %T /sys/fs/cgroup), not cgroup2fs. Use the Docker"
echo "variant above - do NOT run the rest of this block."; exit 1; }
grep -qw memory /sys/fs/cgroup/cgroup.subtree_control \
|| echo +memory | sudo tee /sys/fs/cgroup/cgroup.subtree_control >/dev/null
CG=/sys/fs/cgroup/lab5
sudo mkdir -p $CG
echo 67108864 | sudo tee $CG/memory.max >/dev/null
[ "$(cat $CG/memory.max)" = 67108864 ] || { echo "limit not applied - abort"; exit 1; }
# $BASHPID must be captured BEFORE the pipeline: bash forks each pipeline
# element, so `echo $BASHPID | tee` writes the PID of a transient child.
( SELF=$BASHPID; echo $SELF | sudo tee $CG/cgroup.procs >/dev/null
exec python3 -c 'a=[]
while True: a.append(bytearray(4*1024*1024))' )
echo "exit code: $?"The interviewer says: "This container is in CrashLoopBackOff. No application logs at all — it just stops. Go."
✅ Worked answer and rubric — click to reveal
Killed
exit code: 137Start from the exit code, because it is a complete diagnosis on its own. A process killed by a signal exits 128 + N. 137 is 128 + 9, and 9 is SIGKILL — so nothing crashed and nothing threw an exception. Something outside the process killed it, and it was given no chance to log, which is exactly why there are no application logs.
Two other codes worth knowing in the same breath: 143 is 128 + 15, a clean SIGTERM — a normal shutdown, not a crash. 139 is 128 + 11, SIGSEGV. And 159 is 128 + 31, SIGSYS, which inside a container is essentially always seccomp.
On 137 there are only a few candidates, and they are separable:
1. The cgroup OOM killer — the container exceeded memory.max. Overwhelmingly the most common.
2. The global OOM killer — the node ran out. Rarer, and it usually kills the wrong thing.
3. A liveness probe failing, with the kubelet escalating to SIGKILL after the grace period.
4. Someone or something sent SIGKILL.
Say which you expect, then check:
# was it the OOM killer, and which cgroup?
dmesg | grep -i -E 'oom|killed process' | tail -3
# the cgroup's own counter - cgroup v2 only; there is no memory.events on v1
cat $CG/memory.eventsoom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,
oom_memcg=/lab5,task_memcg=/lab5,task=python3,pid=4313,uid=0
Memory cgroup out of memory: Killed process 4313 (python3) total-vm:79528kB,
anon-rss:65280kB, file-rss:4984kB, shmem-rss:0kB, UID:0 pgtables:196kB oom_score_adj:0constraint=CONSTRAINT_MEMCG is the word that matters. It says this was a cgroup OOM, not a node-level one — the host may have had gigabytes free. oom_memcg= names which cgroup hit its limit, and anon-rss tells you what the process was actually holding. memory.events has an oom_kill counter for the same event, and it survives the process, which makes it the better thing to alert on.
Then separate the two real causes, because the fixes are opposite. A limit set too low for legitimate working-set growth — traffic grew, the cache is sized by request volume, the heap was tuned for a different instance size. Or a genuine leak, in which case raising the limit buys you a slower crash. The way to tell them apart is the shape over time: a leak's memory rises monotonically and never plateaus across restarts; an undersized limit plateaus, just above the ceiling.
Mitigate first, and say so: raise the limit or scale out to stop the restart loop, then diagnose, because a CrashLoopBackOff is a live user-facing outage while you investigate.
Prevent it. Alert on memory.events oom_kill rather than on memory usage, because usage near the limit is normal and a kill never is. Set memory.high below memory.max so the cgroup is throttled and reclaimed before it is killed — a warning instead of an execution. And for a JVM or Node runtime, make sure the heap ceiling is derived from the cgroup limit rather than from the host's memory, which is one of the most common causes of this exact ticket.
Rubric. Weak: "the container crashed", starts reading application logs that do not exist. Hire: decodes 137 as SIGKILL, suspects the memory limit, confirms in dmesg. Strong hire: recites the 137/143/139/159 family, distinguishes CONSTRAINT_MEMCG from a node OOM, mitigates before diagnosing, separates undersized-limit from leak by the shape over time, and alerts on oom_kill rather than on usage.
D7 · Clean-up
# Release the descriptors first, or the unmount fails silently.
pkill -f 'time.sleep(900)'; pkill -f 'm http\.server'
while mountpoint -q /mnt/lab; do umount /mnt/lab; done
losetup -D # free the loop device, not just the file
rm -f /tmp/disk.img; rmdir /mnt/lab 2>/dev/null
chattr -i /srv/app/app.conf 2>/dev/null; rm -rf /srv/app
sudo rm -f /sys/fs/cgroup/lab5/* # a v1/hybrid box leaves plain files here
sudo rmdir /sys/fs/cgroup/lab5 2>/dev/null
# Verify rather than trust: both of these should print nothing.
losetup -a
mount | grep /mnt/lab
echo "labs removed"🏗️ Part E · Round 4 — design, incidents, and closing
E1 · Design under failure
The SRE design round is not the software-engineering one. Google calls its version non-abstract large system design, and the paper says why: you must "concretely design a large-scale system" with "realistic estimates of throughput, storage" while "considering various tradeoffs for reliability, cost, and difficulty."
| The SWE version asks | The SRE version asks | |
| Goal | Does it do the thing? | Does it survive? |
| Numbers | Often waved through | Required. Requests per second, bytes, machines |
| Failure | An error path | The subject of the question |
| Scaling | Shard the database | What breaks first, and what happens to latency at the knee |
| Truth | The schema | The SLIs and SLOs |
1. "What is the SLI, and what number is the SLO?" Not "it should be fast" — "the SLI is the fraction of requests served under 300 ms, measured at the load balancer; the SLO is 99.9% over 28 days." Then the follow-through: 28 days is 40,320 minutes, so that budget is just over 40 minutes of failure per window — and here is what we spend it on.
2. "What breaks first as this grows 10×?" Name one component and say why. It is almost never CPU; it is a connection limit, a lock, a single-threaded section, or a queue with no back-pressure.
3. "What happens when dependency X is slow — not down?" Slow is much worse than down, because down fails fast and slow consumes your threads. This is where timeouts, retry budgets, circuit breakers and load shedding belong, and it is the single most reliable place to demonstrate seniority.
4. "How do I know it is broken before a customer tells me?" The four Golden Signals for each component, and one page-worthy alert with a stated threshold and a reason for that threshold.
An architect designs a bridge that carries the traffic: spans, materials, elegance, cost. A structural engineer is handed the same drawing and asks a different set of questions. What is the rated load, and what happens at 1.5× it? Which member fails first, and does it fail gracefully or all at once? What if the wind is not down but gusting? How do we know a cable is fraying before it snaps?
Both people are competent and both are necessary. Only one of them is being interviewed in this round.
Notice what the engineer is really doing: they are not designing a stronger bridge. They are naming the first thing to fail, deciding whether that failure is survivable, and specifying the inspection that catches it early. A design that fails predictably at a known load, with an alarm on the way there, is better than one that is stronger but fails without warning.
And there is a budget in it. Nobody builds a bridge that cannot fall down; they build one whose probability of failing is agreed, priced, and monitored. That is an error budget, and it is why "we aim for 100% uptime" is the wrong answer to every reliability question in the loop.
Where the analogy stops working. A bridge's load is roughly predictable and its materials do not change under it. Your dependencies get redeployed on a Tuesday afternoon by a team you have not met.
🧪 Exercise E1.1 — Forty-five minutes, out loud, no notes
"Design the monitoring and alerting for a platform of 30 microservices across 3 regions. 50,000 requests per second at peak. Tell me what you collect, where it goes, what pages someone, and what happens when the monitoring itself fails."
✅ A model shape, and the rubric — click to reveal
Start with numbers, because non-abstract means arithmetic. 50,000 rps across 30 services is on the order of 150,000–500,000 internal spans per second once you count fan-out. If each service emits 200 metric series per instance and there are 20 instances, that is 30 × 20 × 200 = 120,000 active series, which is small; the volume problem is logs and traces, not metrics. Say those numbers. Being roughly right out loud beats being precisely silent.
Then what you collect, in three separable pipelines with different costs.
Metrics — numeric, aggregated, cheap, retained for a year. The four Golden Signals per service: latency as a histogram, never pre-computed percentiles. A percentile is a rank statistic, not a mean: the p99 of two hosts is not the average of their p99s and cannot be recovered from them. Histogram buckets are counters, so they add — ship those and compute the quantile at query time. Then traffic, errors split by cause, and saturation. Plus the USE triple per host: utilization, saturation and errors for CPU, memory, disk, network — with PSI where available, because it measures saturation directly.
Logs — expensive, high cardinality, retained for weeks. Structured, sampled at the source for high-volume paths, and never the primary alerting signal.
Traces — the most expensive, sampled at maybe 1%, with tail-based sampling so every error and every slow request is kept regardless of the rate. This is the piece that answers "which of the thirty services is the slow one", which is the question this platform will actually ask.
Then what pages. Very little. Symptom-based alerts on the SLO burn rate, not cause-based alerts on resources: page when the error budget is burning fast enough to be exhausted before the end of the window, using multi-window burn-rate alerts so a brief spike does not page and a slow bleed still does. Everything else is a ticket or a dashboard. If your answer produces more than a handful of pageable alerts for 30 services, say so and explain why you rejected the alternative — the published failure mode here is 200 alerts a week, of which none is read.
Then the question that is really being asked: what happens when the monitoring fails? This is the part most candidates never reach, and it is where the round is decided.
The collection path must not share fate with what it watches — a monitoring stack running on the same cluster it monitors goes dark exactly when you need it. Regional collectors with a global aggregation layer, so losing a region loses that region's data and not the alerting. A dead-man's-switch alert that fires when the monitoring stops reporting, because silence is indistinguishable from health. And an out-of-band path for paging that does not depend on your own infrastructure.
Close with the trade-offs, priced. Full-fidelity traces at 50,000 rps are an enormous bill for data nobody reads; 1% tail-based sampling costs a fraction and answers the same questions. A year of raw logs is a compliance decision, not an engineering one. Say what you would cut first and why.
Rubric. Weak: lists tools — "Prometheus, Grafana, ELK" — with no numbers, no SLOs, and no failure mode. Hire: separates the three pipelines by cost, uses the Golden Signals, alerts on symptoms rather than causes. Strong hire: does the arithmetic unprompted, insists on histograms and explains that percentiles do not aggregate, defines the burn-rate alert with its window, and treats monitoring the monitoring as a first-class part of the design rather than an afterthought.
E2 · The incident and behavioural round
This round is not a personality test. It has a rubric as specific as any of the technical ones: ownership, blameless framing, and one concrete thing that changed afterwards.
The strongest version of this answer is one where you were the cause. Interviewers ask "tell me about an outage you caused" precisely because it is the hardest to answer without either deflecting or self-flagellating, and both are bad signals. The shape that works: what broke, what the user impact was, what you did to stop it, what the system-level finding was, and what shipped as a result.
Q. Tell me about an incident you caused.
Five parts, in this order, and about ninety seconds.
Impact first, in user terms. "Checkout was failing for about 8% of users for 22 minutes." Not "a service went down." If you do not know the impact of your own outage, that is the finding.
What you did — mitigation before diagnosis. "I rolled back the deploy within four minutes; error rate recovered before we knew why."
What the cause turned out to be, briefly and technically.
The system-level finding, not the human one. "The migration was reversible in theory but nobody had tested the down path, and the deploy pipeline had no gate that required it."
What actually changed. One concrete thing, shipped. "Migrations now run in a separate step with an automated rollback test in CI."
The details that separate candidates: naming the impact in user terms and the time to mitigate as separate numbers from time to resolve; refusing to name a person even when the interviewer leaves space for it; and being specific about the change rather than saying "we improved our process". If the change was "we added it to the runbook", say why a runbook was the right level and not a guard rail — the honest answer is sometimes that it was not.
Q. Your team has burned 80% of its error budget in the first two weeks of the month. What do you do?
The budget is the mechanism, so use it as one rather than treating it as an alarm.
Find out where it went. One large incident and a healthy month otherwise is a different situation from a steady bleed, and the response differs: a single event means fix that event; a bleed means the service is chronically below its SLO and the SLO is either wrong or the system is.
Then apply the agreed policy, which should already exist — typically: feature releases pause, and reliability work takes priority, until the budget recovers. The value of having agreed this in advance is that it is not a negotiation during a bad month.
Then ask whether the SLO is right. If it is routinely burned by things users never notice or complain about, the SLI is measuring the wrong thing or the target was set by aspiration rather than by measurement.
The details that separate candidates: saying the quiet part — the purpose of an error budget is to make "how much reliability" a shared decision with a number, rather than an argument between two teams with different incentives. And handling the product-manager follow-up gracefully: the answer to "we need to ship anyway" is not "no", it is "that is a decision we can make together, and here is what it costs, and here is who signs it off."
Q. How do you run a blameless postmortem?
Write it while it is fresh, with a named owner and a deadline. Build a timeline first from evidence — logs, graphs, chat — before anyone writes narrative, because memory reorders events and the reordering is always in a flattering direction.
Separate detection, mitigation and resolution with their own timestamps. Time to detect is usually the most improvable of the three and the least discussed.
Write the contributing factors as conditions, not as decisions by named people: "the alert threshold was above the level users notice", "the rollback path had never been exercised", "the dashboard showed the mean".
Then a small number of action items with owners and dates, prioritised — and the discipline to close them, because a postmortem process whose actions are never done is worse than none, since it converts real findings into a backlog nobody reads.
The details that separate candidates: saying that blameless is not consequence-free at the system level; and knowing that the postmortem's audience is the person who joins next year and hits the same thing. If it is not readable by someone who was not there, it is a chat log with headings.
Q. Walk me through an incident for a service returning elevated errors that has not paged anyone.
That gap is itself the first finding, and I would say so at the start. Elevated errors with no page means either the alert threshold is above the level that hurts users, or the SLI is not measuring what users experience — and whichever it is goes in the postmortem regardless of what caused today's errors.
Then: declare it anyway. An incident is a coordination mode, not a severity. Open a channel, take the incident-commander role or name one, and post the impact as you understand it. Doing this early is cheap and doing it late is what turns a twenty-minute event into a two-hour one.
Then scope, mitigate, diagnose, in that order. Which users, which region, which endpoint, since when. Roll back the most recent change if there is one — most incidents are changes. Then the checklist from Module 14.
Then communicate on a fixed cadence even when there is nothing new: "still investigating, next update in 15 minutes" is a complete and useful message, and silence is what makes people start their own parallel investigations.
The details that separate candidates: treating "no alert fired" as a finding rather than as luck, separating the incident-commander role from the person debugging (one person cannot do both past a certain size), and knowing that the first question a customer-facing team will ask is not "what broke" but "what do we tell people, and when will you know more" — and having an answer to that ready.
E3 · The index — one differentiator per module
Not a summary. For each module, the one claim that most reliably separates a candidate who has used the thing from one who has read about it. If any row surprises you, that is the module to reread.
| Module | The sentence worth being able to say |
| 01 Syscalls | A library call and a system call are not the same thing — printf buffers in user space and only calls write when it flushes, which is why output vanishes on a crash |
| 02 Processes | fork is copy-on-write, so duplicating a 10 GB process is cheap — and shell redirection lives in the gap between fork and exec |
| 03 Files | unlink removes a name, not data; the blocks survive while any descriptor is open, which is why df and du disagree |
| 04 Signals | Ctrl-C signals a process group, not a process — and PID 1 has no default action for any signal |
| 05 Boot | The initramfs exists because the kernel cannot mount a root filesystem whose driver is on that filesystem; and the switch is pivot_root, not chroot |
| 06 Threads | A race that "never happens in testing" is a race with a narrow window, not an absent one; the fix is the invariant, never the sleep |
| 07 Scheduling | Linux load average counts uninterruptible-sleep tasks, so load 40 with 90% idle CPU means forty tasks are blocked in uninterruptible sleep, not computing — usually storage, but check before you say disk. And iowait is a subset of idle |
| 08 Virtual memory | Minor page faults are how memory works; the metric worth alerting on is major faults. And RSS counts shared pages once per process, so PSS is the honest number |
| 09 Page cache & OOM | Free memory is wasted memory — read available. Exit 137 is 128 + 9, and it is far more often a cgroup limit than the node |
| 10 Block I/O | %util is meaningless on any device that serves requests in parallel; judge by r_await/w_await and aqu-sz |
| 11 IPC | A Unix socket beats loopback TCP because it skips the network stack — not because loopback skips checksums; loopback already does |
| 12 Containers | There is no container object in the kernel. Namespaces limit visibility; cgroups limit consumption; both are optional and independent |
| 13 Capabilities & seccomp | UID 0 without CAP_CHOWN cannot chown. seccomp runs before the capability check, so it is the only control that still binds a root process |
| 14 Performance | Utilization is a lagging indicator and saturation is the leading one — and 80% → 90% utilization does not cost 10%, it doubles latency |
Day 1 — breadth. Part B of this module, timed, out loud, twice. Mark every question you could not answer in ninety seconds and reread only those sections.
Day 2 — the errno and exit-code table. EPERM vs EACCES vs EROFS vs ENOSPC vs ENOSYS; exit 137 vs 143 vs 139 vs 141 vs 159. Ten items, free diagnosis, and they route the first five minutes of most incidents.
Day 3 — traversals. Part C. Give all three layers for boot-to-login and for ./app, then invent one: "what happens when you run docker run", "what happens when you kill a process".
Day 4 — the broken box. Build all five Part D labs from scratch without looking at the setup blocks. Narrate the whole way through, to a wall if necessary.
Day 5 — the sixty-second checklist, from memory, on a real machine, plus the posture dump from Module 13. Then the USE table until you can produce a tool for every cell.
Day 6 — design and incidents. Part E out loud. Write your own outage story down in the five-part shape and time it.
Day 7 — nothing technical. Reread your own notes on the three questions you keep getting wrong, and stop. The failure mode of the last day is arriving tired having learned one more fact.
E4 · Self-assessment
The final one. Answer out loud, timed, and compare with the recording from Exercise A1.1.
- Name the rounds in a typical loop and say what each one scores. (A1)
- Why does it matter that each interviewer is assigned one topic? (A1)
- What is the sequencing rule that decides more outcomes than any fact in this track? (A2)
- What is penalised harder than not knowing something, and what do you say instead? (A2)
- Give the four levels of the GitLab ladder as behaviours, not titles. (A2)
- Answer any five Part B questions in under ninety seconds each. (B1)
- What is the three-layer technique, and why does the short answer sound more senior? (C1)
- Give the thirty-second map for power-on to login prompt. (C2)
- Why is there an initramfs, and what runs before main? (C2, C3)
- Where does shell redirection happen, and why can it not happen anywhere else? (C3)
- Name the six steps of the broken-box protocol. (D1)
- df says 78% used and du says 20 KB. Name three causes and the command that distinguishes them. (D2)
- ENOSPC on a filesystem that is 1% full. Explain, and say why you cannot simply fix it in place on ext4. (D3)
- Root cannot write a file whose mode allows it. What is your first check, and what does the errno tell you? (D4)
- EADDRINUSE. Name three causes and the one command that separates them. (D5)
- A container exits 137 with no logs. Explain the number, and say how you separate an undersized limit from a leak. (D6)
- Name four exit codes and four errnos, with meanings. (D6)
- Give the four questions that turn a design answer into an SRE design answer. (E1)
- Why can percentiles not be averaged across hosts, and what do you store instead? (E1)
- What does "blameless" actually mean, and what does a weak version of it sound like? (E2)
- You have burned 80% of the error budget in two weeks. What happens next? (E2)
- Tell your own outage story in ninety seconds, in the five-part shape. (E2)
E5 · Sources
Primary — published by the companies themselves
· Hiring Site Reliability Engineers — Jones, Underwood & Nukala, ;login: 40(3), 2015. Free full text (PDF). The source of the per-interviewer-topic structure, the hiring-committee decision, the "no brainteasers" stance, and the non-abstract design framing. Note there is no Google-hosted "how we hire SREs" page; this paper is the canonical public document.
· GitLab SRE job family — the only openly published SRE levelling ladder, and the source of the four-level gradient in A2.
· Google SRE Book and SRE Workbook — in particular Embracing Risk, Service Level Objectives, Monitoring Distributed Systems, Effective Troubleshooting and Postmortem Culture.
Open question banks — where most published questions actually come from
· bregman-arie/devops-exercises · michaelkkehoe/sre-interview · mxssl/sre-interview-prep-guide
· SadServers and its scenario list — the public bank of broken-box prompts, and the closest thing to a standard for that round. The Part D labs here are built from scratch so you can make and remake them offline.
· what-happens-when — the reference answer to the most-asked traversal question.
Manual pages and kernel documentation used in the answers
· section 1 · section 2 · man-pages(7) · execve(2) · fork(2) · ld.so(8) · bootup(7) · pivot_root(2) · kernel documentation · Brendan Gregg — Linux performance
A caution about everything else
· A large share of pages titled "DevOps interview questions 2026" are search-optimised or vendor-authored, often with something to sell. Their questions are usually plausible and worth practising; their claims about process and scoring should be treated as second-hand. Where this module states something about how you are scored, it is either from a primary source above or flagged as an interpretation. The strongest single claim — mitigate before root-cause — is the one that appears independently in Google's own troubleshooting guidance, in published interviewer notes, and in every incident framework in the industry, which is why it is the one thing here worth treating as a rule.
And the honest limitation of this module. No published question bank goes as deep as Modules 07–14. Nobody is going to ask you about EEVDF, %util on parallel devices, or perf_event_paranoid by name. That material is not there to be recited — it is there so that when you are asked something shallow, the answer you give has a floor underneath it, and so that the broken box in Part D does not surprise you.
The thing worth carrying out of it is not any single fact in the index above. It is the habit every module drilled the same way: say what you expect to see, run the command, and read the output — especially when it disagrees with you. That is what the exercises were for, it is what the ✅ toggles were hiding, it is what the broken-box round is scoring, and it is the job.
Good luck. Go and be wrong out loud.