Module 07 — CPU Scheduling, nice & Load Average
Updated 22 August 2026
Module 01 showed you the timer interrupt that lets the kernel take the CPU back. This module is about what it does with it — who runs next, why, and how to read the numbers everyone quotes and few people understand.
🧠 concept → 🧩 real-world analogy → 🧪 exercise → ✅ expected result (hidden) → 🎯 interview questions (hidden)
From Module 01 — the timer interrupt, preemption, and mode versus context switches.
From Module 02 — process states R, S, D, the voluntary and involuntary switch counters, and the basic idea of load average.
From Module 06 — that Linux schedules threads, not processes. This module depends on that completely.
⚖️ Part A · How the scheduler decides
A1 · What the scheduler is actually choosing between
The scheduler's job is one decision, made constantly: which runnable thread gets a CPU next, and for how long.
Three words in that sentence are doing real work.
Thread, not process. Module 06 established this. A process with eight threads is eight separate candidates, and the scheduler has never heard of the process they belong to.
Runnable, not all. Only threads in state R are candidates. On the machine from Module 01 that was 1 out of 243 — everything else was asleep and not competing for anything. The scheduler's job is much smaller than people assume.
A CPU, not the CPU. Each CPU has its own run queue. Work is balanced between them, which is Section B2.
A post office has four counters and a room full of people. But most of the people are not in the queue — they are filling in forms, waiting for someone, or reading. They are not competing for a counter and the staff correctly ignore them.
Only the people actually standing in the queue matter. That is state R.
Two things follow, and both correct common misconceptions.
First, a room with two hundred and forty-three people and one in the queue is not busy. That is the machine from Module 01, and it is why process count tells you almost nothing.
Second, the counters serve individuals, not families. Four people who arrived together still queue as four. That is threads, not processes, and it is why a process's CPU figure is a sum rather than a thing the scheduler ever considered.
Where the analogy stops working, and it is the whole of Section A2. A post office queue is first-come-first-served, and your place is decided when you arrive.
Linux does not work that way at all. It does not ask who arrived first — it asks who has had the least so far. Somebody who has just been served goes to the back regardless of when they arrived, and somebody who has been waiting patiently moves up.
🧪 Exercise A1.1 — See how few threads are actually competing
# How many threads exist, and how many are RUNNABLE right now?
echo "threads total: $(ps -eL --no-headers | wc -l)"
echo "threads runnable: $(ps -eLo stat --no-headers | grep -c '^R')"
echo "CPUs available: $(nproc)"
# The kernel's own view: runnable / total, from /proc/loadavg field 4
cat /proc/loadavg
# Now create real competition and look again
for i in $(seq 1 4); do ( end=$((SECONDS+6)); while [ $SECONDS -lt $end ]; do :; done ) & done
sleep 2
echo "--- with 4 busy loops running ---"
echo "threads runnable: $(ps -eLo stat --no-headers | grep -c '^R')"
cat /proc/loadavg
wait✅ Expected result — click to reveal
threads total: 412
threads runnable: 1
CPUs available: 2
$ cat /proc/loadavg
0.14 0.09 0.08 1/412 12043
--- with 4 busy loops running ---
threads runnable: 5
0.14 0.09 0.08 5/416 12089What to read out of it.
412 threads, 1 runnable. The scheduler had one candidate and two CPUs. It was not making a difficult decision; there was nothing to decide.
The fourth field of /proc/loadavg says it directly: 1/412 means one runnable out of 412 total. That single field is a better instant measure of "is this machine busy" than any average, because it is a live count rather than something smoothed over minutes.
With four busy loops, runnable goes to 5 — the four loops plus whatever ran the command. Now there is genuine competition: five threads, two CPUs, and three of them are waiting at any moment.
Notice the three load figures have not moved at all — they are byte-for-byte the numbers from before the loops started, while the runnable count is already 5. That is not a rounding artefact: the kernel recomputes load averages once every five seconds and then smooths them over a minute, so two seconds in they know nothing. 1/412 is instant; 0.14 is history.
A2 · Fair share, not priority
Older schedulers used fixed priority levels: run the highest-priority thread, and only look at lower ones when nothing above them is ready. That starves low-priority work completely.
Linux does something different for ordinary threads. It tracks how much CPU each thread has already had, and runs whichever has had the least.
That one rule produces fairness automatically:
- A thread that has been waiting has had little, so it goes next.
- A thread that has just run has had more, so it drops back.
- A thread that sleeps a lot has had almost none, so it gets served quickly when it does wake — which is exactly what you want for anything interactive.
For about fifteen years the answer was CFS, the Completely Fair Scheduler. Since Linux 6.6 the default is EEVDF — Earliest Eligible Virtual Deadline First.
EEVDF keeps the fair-share idea and adds a second question. CFS asked only "who has had the least?". EEVDF asks "who is eligible — has had less than their fair share — and of those, whose deadline is soonest?" That gives latency-sensitive threads a much better response time without abandoning fairness.
The lab kernel in this track is 6.6 or newer, so your machine is running EEVDF — check with uname -r. Saying "Linux uses CFS" in an interview is not wrong historically, but saying "CFS, replaced by EEVDF in 6.6" is noticeably better — and it is checkable in one command.
Back to the shared flat from Module 06. Six people, one kitchen, and somebody has to decide who cooks next.
The bad system is a ranking. Priya always goes first because she is most senior. Sam is last and, on a busy evening, never eats. That is fixed-priority scheduling, and starvation is built in.
The system Linux uses is a tally. Write down how much kitchen time each person has had. Whoever has had the least goes next.
Notice how well that behaves without anyone designing the behaviour:
- Someone who just cooked a three-course meal has a big tally and waits a while. Fair.
- Someone who has been out all week has almost nothing on their tally, so when they turn up they go straight in. That is a sleeping process waking up and getting served immediately — which is why your shell responds instantly even on a loaded machine.
- Nobody is ever permanently last, because as others cook their tallies rise and yours does not.
EEVDF adds one refinement: among everyone who is owed time, serve whoever needs it soonest — the person making toast goes ahead of the person planning a roast, because it costs the roast almost nothing and helps the toast enormously.
Where the analogy stops working. A rota is checked when someone finishes.
The scheduler re-evaluates hundreds of times a second, whether or not anyone has finished, because the timer interrupt from Module 01 forces it to. Nobody has to cooperate.
🧪 Exercise A2.1 — Watch fair share happen
# Which kernel, and therefore which scheduler?
uname -r
# Four identical busy loops on a machine with fewer CPUs than that.
# Fair share predicts they should each get an EQUAL slice. Check it.
for i in 1 2 3 4; do
( end=$((SECONDS+8)); while [ $SECONDS -lt $end ]; do :; done ) &
done
sleep 5
echo "=== CPU share per loop (nproc = $(nproc)) ==="
ps -o pid,pcpu,ni,comm -C bash --no-headers | sort -k2 -rn | head -4
wait✅ Expected result — click to reveal
$ uname -r
6.8.0-45-generic
=== CPU share per loop (nproc = 2) ===
12401 49.8 0 bash
12402 49.6 0 bash
12403 49.5 0 bash
12404 49.3 0 bashWhat to read out of it — this is fair share, measured.
Four threads, two CPUs, and each got almost exactly 49.5%. Not one at 100% and three starved. Not random. Four near-identical numbers.
The arithmetic checks: two CPUs is 200% of CPU capacity, split four ways is 50% each. The scheduler divided a scarce resource evenly without being told to, purely by always picking whoever had had the least.
The ni column is 0 for all of them — default nice, no bias. Section A3 changes that and watches the split change with it.
And note the small variation, 49.8 down to 49.3. Fair share is fair over time, not instant. At any given microsecond one thread is running and three are not; it is the tally that evens out.
On a 2-CPU box with four busy threads, 50% each is the correct and fair outcome. The scheduler is working. The machine is oversubscribed, which is a capacity question, not a scheduling one.
The useful follow-up is: what is the run-queue length relative to nproc? If it is persistently above 1.0 per CPU, you need more CPU or less work — no amount of nice, affinity or policy tuning creates capacity that does not exist.
A3 · nice — biasing the share
nice shifts the fair-share tally. The scale runs from −20 (greediest) to +19 (most generous), and the default is 0.
The name is the right way round if you read it as politeness: a high nice value means being nice to everyone else — taking less. It reads backwards if you think of it as priority, which is why people get it wrong.
The mechanism is a weight, not a queue position. Each step of 1 changes a thread's share by a factor of about 1.25. So:
- Two threads at nice 0 and nice 5 do not mean "the first always wins". They mean roughly a 3:1 split of CPU.
- A nice +19 thread still runs. It just gets a very small slice when there is competition, and all of the CPU when there is none.
It only matters under contention. A nice +19 process on an idle machine runs at full speed. nice divides a shortage; with no shortage there is nothing to divide.
Ordinary users can only be nicer, never greedier. You can raise your nice value, and you cannot lower it back — that needs root or CAP_SYS_NICE. Otherwise every program would simply set itself to −20 and the whole system would be pointless.
And nice does nothing at all for a process blocked on disk or network. If a backup job is slow because of I/O, renicing it changes nothing, because it was never waiting for CPU. That is Module 10's material, and reaching for nice is one of the most common wrong reflexes in production.
Same kitchen, same tally. nice is how insistent each person is about their turn.
Nice +19 is the flatmate who always says "no, you go first, I'm not in a rush." They still eat. On a quiet evening they cook whenever they like. On a busy one they end up last, repeatedly, because they keep giving way.
Nice −20 is the one who says "I need the hob now." Everyone else waits more.
Three things fall out of this, and they are exactly the three rules above:
- On an empty evening it makes no difference. Politeness only matters when there is a queue.
- You can always offer to go last. You cannot appoint yourself to go first — that takes the landlord. Hence root.
- It is a tendency, not a rule. The polite flatmate is not banned from the kitchen; they simply get less of it when it is busy. That is a weight, not a priority level.
Where the analogy stops working, and it is the common mistake. Being polite about the hob does not help if you are waiting for a delivery.
A process blocked on disk is waiting for a van, not a hob. Renicing it is offering to give way in a queue it was never standing in.
🧪 Exercise A3.1 — Change the split with nice
# Two identical loops, one at nice 0 and one at nice 10.
( end=$((SECONDS+10)); while [ $SECONDS -lt $end ]; do :; done ) &
NORMAL=$!
nice -n 10 bash -c 'end=$((SECONDS+10)); while [ $SECONDS -lt $end ]; do :; done' &
NICE=$!
# Pin both to ONE CPU so they genuinely compete
taskset -pc 0 $NORMAL >/dev/null 2>&1
taskset -pc 0 $NICE >/dev/null 2>&1
sleep 6
echo "=== both pinned to CPU 0, competing ==="
ps -o pid,pcpu,ni,comm -p $NORMAL,$NICE --no-headers
wait
# And prove nice does nothing when there is no competition
echo "=== nice +19 ALONE on an idle machine ==="
nice -n 19 bash -c 'end=$((SECONDS+4)); while [ $SECONDS -lt $end ]; do :; done' &
A=$!; sleep 3
ps -o pid,pcpu,ni,comm -p $A --no-headers
wait✅ Expected result — click to reveal
=== both pinned to CPU 0, competing ===
12610 89.4 0 bash
12613 10.2 10 bash
=== nice +19 ALONE on an idle machine ===
12688 99.1 19 bashWhat to read out of it — the two blocks make opposite points and you need both.
Under competition: roughly 89% versus 10%, an 8.7:1 split. Ten nice steps at about 1.25× each is 1.25¹⁰ ≈ 9.3, so the observed ratio matches the weighting almost exactly. This is not a rounding coincidence — it is the mechanism.
Crucially, the nice +10 process is not starved. It got 10%, steadily. Under a fixed-priority scheduler it would have got nothing at all. That is the difference between a weight and a priority level.
Alone: the nice +19 process ran at 99.1% — essentially the whole CPU. The nicest possible process on an idle machine runs at full speed, because there is no shortage to divide.
Note the taskset -pc 0. Without it the two loops land on different CPUs and never compete, and you see 99% and 99%. That is Section B2, and it is also the most common reason this experiment appears not to work.
It is the wrong tool for a slow database (usually I/O or locks), a slow web service (usually waiting on something else), or any process that is not actually in state R. Check the state first, exactly as in Modules 02 and 06 — if it is not competing for CPU, nice has nothing to change.
A4 · Voluntary and involuntary context switches
Module 01 introduced these two counters in passing. Now they have a scheduler to hang off, and they become one of the most useful diagnostics on Linux.
A thread stops running for one of exactly two reasons:
It asked for something and had to wait: disk, network, a lock, a timer.
It went to state S or D and left the run queue.
High count means: waiting on something.
More CPU will not help.
It still wanted the CPU. The timer fired and the scheduler picked someone else.
It stayed in state R, at the back of the queue.
High count means: competing for CPU.
That is a capacity problem.
The two point in opposite directions, and a plain "CPU %" figure cannot tell them apart. That is why this pair is worth knowing.
You are at the post office counter.
You leave voluntarily because they need a form you have not filled in. Nobody moved you on — you cannot proceed until something else happens, so you step away. That is waiting on I/O or a lock, and the counter was never the constraint. Opening more counters would not have helped you at all.
You are asked to step aside because you have had your five minutes and there is a queue. You were ready and able to carry on. That is preemption, and here more counters genuinely would help, because the constraint really is counter capacity.
Same outcome from a distance — you are no longer at the counter. Opposite causes, opposite fixes.
Where the analogy stops working, and it is why the counters matter. In a post office you can see which happened.
With a process you cannot. Both look like "not running". The two numbers in /proc/PID/status are the only easy way to tell, and almost nobody looks at them.
🧪 Exercise A4.1 — Two processes, opposite signatures
# A: pure CPU work - wants the CPU constantly, never waits for anything
( end=$((SECONDS+8)); while [ $SECONDS -lt $end ]; do :; done ) &
CPU=$!
# B: constant tiny waits - barely uses CPU at all
( end=$((SECONDS+8)); while [ $SECONDS -lt $end ]; do sleep 0.05; done ) &
IO=$!
# Create competition so the CPU-bound one gets preempted
for i in 1 2 3; do ( end=$((SECONDS+8)); while [ $SECONDS -lt $end ]; do :; done ) & done
sleep 6
for p in $CPU $IO; do
echo "--- PID $p ---"
grep -E 'voluntary' /proc/$p/status
ps -o stat=,pcpu= -p $p
done
wait✅ Expected result — click to reveal
--- PID 12801 --- (this is the CPU-bound one)
voluntary_ctxt_switches: 2
nonvoluntary_ctxt_switches: 1943
R 49.2
--- PID 12804 --- (this is the wait-bound one)
voluntary_ctxt_switches: 118
nonvoluntary_ctxt_switches: 4
S 0.3What to read out of it — the two signatures could not be more different.
CPU-bound: 2 voluntary, 1,943 involuntary. It essentially never chose to stop. It was moved on nearly two thousand times in six seconds, because four busy threads were sharing two CPUs. State R, CPU 49% — which is its fair share, from Section A2.
Wait-bound: 118 voluntary, 4 involuntary. It gave up the CPU 118 times — once per sleep — and was almost never preempted, because it was hardly ever in the queue to begin with. State S, CPU 0.3%.
Now the diagnostic value. Both processes "are not running" most of the time. If all you had was CPU percentage you would conclude the second one is fine and the first is busy, and you would learn nothing actionable.
The counters tell you what to do:
- High involuntary → it wants CPU and cannot get enough. Add CPU, reduce competition, or reduce its work.
- High voluntary → it is waiting on something else. Adding CPU changes nothing. Find what it waits on — WCHAN from Module 02, or futex from Module 06.
What matters is the ratio to work done, and the split. Rising involuntary switches with flat throughput means the machine is oversubscribed and time is going into switching rather than working. Rising voluntary switches means more waiting, not more contention.
Per-process, /proc/PID/status is free and safe on a live process. That is worth saying, because the alternative people reach for is strace, which Module 04 warned can slow the target tenfold.
🎯 Interview questions — The scheduler
Q. How does the Linux scheduler decide what runs next?
For ordinary threads it does not use fixed priorities. It tracks how much CPU each runnable thread has already received and runs whichever has had the least — fair share.
That one rule produces the behaviour you want without special cases: a thread that has just run drops back, a thread that has been waiting moves up, and a thread that sleeps most of the time gets served quickly when it wakes, which is why an interactive shell stays responsive on a loaded machine.
It schedules threads, not processes, and only threads in state R are candidates — usually a tiny fraction of what is on the machine.
The details that separate candidates:
- Get the name current. The answer was CFS for about fifteen years; since Linux 6.6 the default is EEVDF. EEVDF keeps fair share and adds a deadline, so it asks who is owed time and, of those, who needs it soonest — much better latency for interactive threads. Most published interview material still says CFS.
- Say why fair share beats fixed priority: fixed priority starves low-priority work completely, fair share never does.
- Name the observable: four busy threads on two CPUs settle at ~50% each. That is the scheduler working correctly, not a problem — it is a capacity question instead.
Q. What does nice actually do? When is it the wrong tool?
nice biases a thread's share of CPU under contention. The range is −20 (greediest) to +19 (most generous), default 0, and each step changes the share by a factor of about 1.25 — so a 10-step difference is roughly a 9:1 split, not a strict ordering.
It is a weight, not a priority level. A nice +19 thread is never starved; it simply gets a small slice when the machine is busy, and the whole CPU when the machine is idle.
Ordinary users can only raise their nice value. Lowering it needs root or CAP_SYS_NICE, otherwise everything would set itself to −20.
The details that separate candidates — the wrong-tool half is the real question:
- nice does nothing for a process that is not competing for CPU. If it is blocked on disk, network or a lock, it was never in the run queue, and renicing changes nothing. Check the state first: S or D means nice is irrelevant.
- It does nothing on an idle machine, because there is no shortage to divide.
- Name the one shape it is right for: a batch job that should yield to interactive work on the same host — backups, log compression, reindexing.
- Mention ionice as the I/O equivalent, which is what people usually meant when they reached for nice and it did not help.
Q. What is the difference between a voluntary and an involuntary context switch, and why does it matter?
A voluntary switch is a thread giving up the CPU because it must wait — for disk, network, a lock or a timer. It leaves the run queue and goes to S or D.
An involuntary switch is the scheduler taking the CPU away while the thread still wanted it. It stays in R at the back of the queue. This is the timer interrupt from Module 01 doing its job.
They point in opposite directions, which is exactly why the pair is useful:
- High involuntary → the thread wants CPU and cannot get enough. Adding CPU or reducing competition helps.
- High voluntary → the thread is waiting on something else. Adding CPU changes nothing.
Both are in /proc/PID/status, free to read and safe on a live process.
The details that separate candidates:
- A plain CPU percentage cannot distinguish them, and that is the whole point. Both look like "not running".
- Machine-wide, vmstat 1's cs column is not bad on its own. A busy web server legitimately switches tens of thousands of times a second. What matters is the ratio to work done and which kind is rising.
- Connect it to Module 06: high voluntary switches plus WCHAN showing futex_wait is lock contention, not I/O — the same counter, a different cause.
🎚️ Part B · Policies and placement
B1 · Scheduling policies
Fair share is one policy, and it is the one almost everything uses. Linux has several, in two families.
| Policy | Family | What it does |
|---|---|---|
| SCHED_OTHER | Normal | Fair share. The default for essentially everything. Obeys nice. |
| SCHED_BATCH | Normal | Like OTHER but assumes the thread is not interactive, so it is preempted less often. |
| SCHED_IDLE | Normal | Lower than nice +19. Runs only when nothing else wants the CPU at all. |
| SCHED_FIFO | Real-time | Runs until it blocks or yields. Not preempted by anything of lower priority. |
| SCHED_RR | Real-time | Like FIFO, but shares time slices with equal-priority real-time threads. |
| SCHED_DEADLINE | Real-time | You declare this much CPU, this often, by this deadline, and the kernel guarantees it or refuses. |
The two families are not variations on each other. Any real-time thread beats every normal thread, always. The scheduler does not consider a SCHED_OTHER thread while any real-time thread is runnable.
A FIFO thread runs until it chooses to stop. It is not preempted by anything below it. So a FIFO thread with a bug — an infinite loop with no blocking call — takes a CPU and never gives it back. Not slowly. Completely.
On a single-CPU machine that would be an unrecoverable hang — and would be, were it not for the safety valve below: you could not even get a shell, because your shell is SCHED_OTHER and will never be scheduled again.
Linux ships a safety valve for exactly this: kernel.sched_rt_runtime_us reserves a slice of every second for non-real-time work, so you can usually still log in and fix it. Some tuning guides tell you to disable that. Do not, unless you are building a genuine real-time system and know precisely what you are doing.
The post office from Section A1 serves people by the fair-share rota. Everyone gets a turn, nobody is permanently last.
Then an ambulance crew walks in. They do not join the queue at all. They go straight to a counter, and everyone else waits — regardless of how long they have been standing there or how little service they have had. That is SCHED_FIFO.
This is right when it is genuinely an emergency. Audio that must not glitch, a robot arm that must react in a fixed number of milliseconds, an industrial controller. Those jobs are useless if they are merely usually on time.
And it is catastrophic when misused. A crew that walks in and then stands at the counter doing nothing blocks the entire post office forever. Nobody can move them, because the rule is that they take priority. That is a buggy FIFO thread, and there is no polite mechanism to recover.
The reserved slice is the manager's rule that one counter always stays open for ordinary customers, no matter what. It is the only reason you can still get in and sort the mess out.
Where the analogy stops working, and it is the honest caveat. A real ambulance crew is genuinely urgent.
Almost no application that asks for real-time priority actually needs it. Being important is not the same as being useless if late. Standard Linux is not a real-time operating system, and marking an ordinary web service SCHED_FIFO makes it less reliable, not more.
🧪 Exercise B1.1 — Look at policies, and see the safety valve
# What policy is everything running under?
ps -eo pid,cls,rtprio,ni,comm --no-headers | awk '{print $2}' | sort | uniq -c | sort -rn
# Anything real-time on this machine?
ps -eo pid,cls,rtprio,comm --no-headers | awk '$2 != "TS" && $2 != "-"' | head -8
# The safety valve: how much CPU is reserved for NON-real-time work?
echo "rt period: $(cat /proc/sys/kernel/sched_rt_period_us) us"
echo "rt runtime: $(cat /proc/sys/kernel/sched_rt_runtime_us) us"
# Your own shell's policy
chrt -p $$✅ Expected result — click to reveal
$ ps -eo pid,cls,rtprio,ni,comm --no-headers | awk '{print $2}' | sort | uniq -c | sort -rn
137 TS
5 FF
$ ps -eo pid,cls,rtprio,comm --no-headers | awk '$2 != "TS" && $2 != "-"' | head -8
18 FF 99 migration/0
21 FF 99 migration/1
40 FF 50 watchdogd
71 FF 50 irq/24-ACPI:Ged
72 FF 50 irq/25-ACPI:Ged
$ echo "rt period: $(cat /proc/sys/kernel/sched_rt_period_us) us"
rt period: 1000000 us
rt runtime: 950000 us
$ chrt -p $$
pid 3901's current scheduling policy: SCHED_OTHER
pid 3901's current scheduling priority: 0What to read out of it.
137 processes on TS — time sharing, which is SCHED_OTHER, fair share. That is essentially everything you will ever run.
The five real-time threads are all kernel threads — migration, watchdogd, the threaded IRQ handlers. Note the two priority tiers: migration at 99, the highest anything gets, and the rest at 50, which is what the kernel's sched_set_fifo() helper assigns. On this machine there are no RR threads at all — SCHED_RR is rare in practice. Module 01 taught you to recognise kernel threads by their bracketed names in ps -ef; here they are again, and they are the only things on this machine with real-time priority. No application has it, and that is normal and correct.
Now the safety valve, and read the arithmetic: period 1,000,000 µs (one second), runtime 950,000 µs. So real-time threads may use at most 95% of each second, leaving 5% guaranteed for everything else. That 5% is what lets you log in and run chrt to fix a runaway real-time thread.
Your shell reports SCHED_OTHER at priority 0 — an ordinary citizen, exactly as it should be.
Auditing: ps -eo cls,comm | grep -v TS should return only kernel threads. An application thread on FF or RR is worth a conversation — someone marked it real-time, and it is far more likely to cause an outage than prevent one.
Never set sched_rt_runtime_us to −1 (unlimited) because a tuning blog said so. That removes the 5% reservation, and the next buggy real-time thread takes the machine with no way back in.
B2 · CPU affinity
Section A1 said each CPU has its own run queue. CPU affinity is the set of CPUs a thread is allowed to run on, and by default it is all of them.
The kernel moves threads between CPUs to balance the load, but it prefers not to. Moving a thread to a different CPU throws away everything that CPU had cached for it, and the thread starts slowly on the new one — the same cold-cache cost Module 01 described for context switches.
So the scheduler tries to keep a thread where it was. taskset is how you force it.
Pinning removes the kernel's ability to balance. If you pin a thread to CPU 0 and CPU 0 is busy, that thread waits — even with three idle CPUs sitting next to it. You have traded flexibility for locality, and flexibility is usually worth more.
It genuinely helps in a few narrow cases: latency-critical threads that must never migrate, isolating a noisy workload away from everything else, and NUMA machines where being on the wrong socket means every memory access crosses an interconnect.
For an ordinary service on an ordinary machine, the default — let the kernel decide — beats hand-pinning nearly every time.
A barbershop has four chairs. You have been coming for years and one barber knows exactly how you like it — no explaining, straight to work. That familiarity is the CPU's cache, warm with your data.
Left alone, the shop will send you to your usual barber when they are free, because it is faster for everyone. That is the scheduler's natural preference to keep a thread where it was.
Pinning is insisting on that barber and no other. When they are free, you are served fast. When they are busy, you sit and wait while three empty chairs face you.
That is the whole trade in one image, and it is why pinning so often makes things slower. You optimised for the good case and made the bad case much worse.
Where the analogy stops working, and it is the case where pinning wins. In a barbershop, any chair is roughly as good.
On a NUMA machine they are not. Memory is attached to specific sockets, and a thread running on the wrong socket pays a real penalty on every single memory access — as if your barber had to walk to another building for the scissors each time. There, keeping a thread near its memory is worth giving up flexibility for, and that is what numactl exists to do.
🧪 Exercise B2.1 — Pin a thread and watch it lose
# Default: two busy loops on a 2-CPU box, free to use either CPU
for i in 1 2; do ( end=$((SECONDS+6)); while [ $SECONDS -lt $end ]; do :; done ) & done
sleep 4
echo "=== unpinned: both should be near 100% ==="
ps -o pid,pcpu,psr,comm -C bash --no-headers | sort -k2 -rn | head -3
wait
# Now pin BOTH to the same single CPU
for i in 1 2; do
( end=$((SECONDS+6)); while [ $SECONDS -lt $end ]; do :; done ) &
taskset -pc 0 $! >/dev/null 2>&1
done
sleep 4
echo "=== both pinned to CPU 0: now they must share one ==="
ps -o pid,pcpu,psr,comm -C bash --no-headers | head -3
wait
# What is your shell allowed to run on?
taskset -pc $$
grep Cpus_allowed_list /proc/$$/status✅ Expected result — click to reveal
=== unpinned: both should be near 100% ===
13101 99.2 1 bash
13102 99.0 0 bash
=== both pinned to CPU 0: now they must share one ===
13140 49.6 0 bash
13141 49.4 0 bash
$ taskset -pc $$
pid 3901's current affinity list: 0,1
$ grep Cpus_allowed_list /proc/$$/status
Cpus_allowed_list: 0-1What to read out of it — the psr column is the one to watch.
Unpinned: two loops, psr values 1 and 0 — the kernel put them on different CPUs by itself, and each got 99%. Two threads, two CPUs, full speed, no configuration required.
Pinned to CPU 0: both show psr 0, and each dropped to 49.5%. They are now sharing one CPU while CPU 1 sits completely idle.
That is the cost of pinning, measured. You halved the throughput of this workload by taking a decision away from the kernel, and the idle CPU next door is the evidence.
This also explains Exercise A3.1. The nice demonstration only worked because both loops were pinned to CPU 0 — without that they land on separate CPUs, never compete, and nice appears to do nothing.
Cpus_allowed_list: 0-1 is where the kernel records this per-thread. In a container it is often a subset, and that is one of the numbers nproc does not tell you — Module 01's Exercise B4.2 problem, from a different angle.
Then name the related lever: isolcpus on the kernel command line from Module 05 removes CPUs from the general scheduler entirely, reserving them for explicitly pinned work. That is the real-world version of this, and it is set at boot rather than at runtime.
🎯 Interview questions — Policies and placement
Q. What are the Linux scheduling policies, and when would you use a real-time one?
Two families. Normal: SCHED_OTHER (fair share, the default for everything), SCHED_BATCH (assumes non-interactive, preempted less), SCHED_IDLE (runs only when nothing else wants the CPU). Real-time: SCHED_FIFO (runs until it blocks or yields), SCHED_RR (FIFO with time slicing among equals), SCHED_DEADLINE (declare a budget and a deadline; the kernel admits or refuses).
Any real-time thread beats every normal thread, always. The families are not variations on each other.
You use real-time when late output is worthless, not merely undesirable: audio processing, motor control, industrial systems. Not because a service is important.
The details that separate candidates:
- Explain the hazard concretely. A SCHED_FIFO thread that never blocks takes a CPU and never gives it back — on a single-CPU box that is an unrecoverable hang, because your shell is SCHED_OTHER and will never run again.
- Name the safety valve and defend it: sched_rt_runtime_us reserves 5% of each second for non-real-time work by default. Tuning guides that tell you to set it to −1 are removing your only way back in.
- Give the audit: ps -eo cls,comm | grep -v TS should return only kernel threads.
- Say plainly that Linux is not an RTOS. PREEMPT_RT narrows the gap, but standard Linux gives no hard guarantees.
Q. What is CPU affinity? When would you pin a process, and when would you not?
Affinity is the set of CPUs a thread may run on. By default it is all of them, and the kernel balances work across CPUs while preferring to keep a thread where it was — because migrating throws away that CPU's warm cache.
taskset forces a restriction. The trade is locality against flexibility, and flexibility usually wins: a pinned thread waits when its CPU is busy even if others are idle.
Worth pinning for: latency-critical threads that must not migrate, isolating a noisy workload, and NUMA machines where a thread on the wrong socket pays on every memory access.
Not worth pinning for: ordinary services on ordinary machines. The default is better.
The details that separate candidates:
- Give the measurable failure: two busy threads pinned to one CPU get 50% each while another CPU sits idle. Pinning halved the throughput.
- Name the boot-time version: isolcpus= on the kernel command line removes CPUs from the general scheduler so only explicitly pinned work runs there.
- Connect it to containers: Cpus_allowed_list in /proc/PID/status is the truth about which CPUs a process may use. nproc inside a container does not reflect a CPU quota.
- Mention numactl for the NUMA case, since taskset controls CPU placement but not memory placement.
📊 Part C · Load average and CPU accounting
C1 · Load average, properly
Almost everyone reads load average wrong. Here is what it actually is.
Load average is the average number of threads that either want the CPU right now or are stuck waiting on disk. That is it. It is a count of scheduling entities — threads, as Section A1 established, not processes — and it is not a percentage. A Java service with 200 runnable threads contributes 200, not 1.
Three numbers, three time windows:
| Number | Window | What it tells you |
| First | Last 1 minute | What is happening now |
| Second | Last 5 minutes | The recent trend |
| Third | Last 15 minutes | The background level |
Three things trip people up.
It is not a percentage. A load of 4.0 does not mean 400% busy. It means four processes wanted to run on average. On a machine with 8 CPUs, four processes is half capacity — quiet. On a machine with 2 CPUs, four processes means two of them are always waiting — busy.
On Linux it counts disk waiting too. Most other Unix systems only count processes waiting for CPU. Linux also counts processes in the D state — stuck in an uninterruptible disk or network read. So a machine with an idle CPU and one very slow disk can show a high load. The CPU is doing nothing; the load number is still climbing.
It lags. The three numbers are smoothed averages. They do not jump. If a machine goes from idle to fully hammered, the 1-minute number takes about a minute to reflect it — and when the load stops, the number keeps falling slowly for minutes afterwards.
Load average is the average number of people either being served or standing in line at a bank.
"Twelve people" tells you nothing until you know how many tellers there are. Twelve people with twelve tellers is a calm morning — everyone walks straight up. Twelve people with two tellers is a queue out the door.
The Linux twist: it also counts people who are standing at the counter but frozen, waiting for head office to fax back a document. The teller is free. The customer is not moving. Linux still counts them.
And the number is an average over the last while, not a snapshot. If a coach party of forty walks in right now, the "average" only creeps up over the following minutes. If they all leave, the number stays high for a while afterwards.
Where the analogy stops working. Bank customers can see each other and get discouraged and leave. Processes cannot. They simply wait until the kernel picks them.
🧪 Exercise C1.1 — watch load average lag behind reality
# How many CPUs do we have? Every load number is judged against this.
nproc
# Baseline: an idle machine
cat /proc/loadavg
# Now hammer every CPU for 30 seconds
for i in $(seq 1 $(nproc)); do
timeout 30 bash -c 'while :; do :; done' &
done
# Sample the load once a second while they run, and for 10 seconds after
for i in $(seq 1 40); do
echo "$i s: $(cut -d' ' -f1-3 /proc/loadavg)"
sleep 1
done
wait✅ Expected result — click to reveal
On a 2-CPU virtual machine:
2
0.08 0.11 0.09 1/312 4187
1 s: 0.10 0.23 0.31
2 s: 0.10 0.23 0.31
5 s: 0.26 0.26 0.32
10 s: 0.40 0.29 0.33
15 s: 0.52 0.31 0.34
20 s: 0.64 0.34 0.35
25 s: 0.83 0.39 0.36
29 s: 0.93 0.41 0.37
30 s: 0.93 0.41 0.37
31 s: 0.93 0.41 0.37
35 s: 0.85 0.41 0.37
40 s: 0.78 0.40 0.37What to read out of this.
At second 1 the machine was already fully loaded — two busy loops on two CPUs — but the 1-minute load was 0.15. The number had not caught up. If you had run uptime at that moment you would have concluded the machine was idle. It was not.
Two things jump out, and the second is the lesson. First, the number only moves every five seconds — the kernel recomputes it on a 5-second tick, so sampling once a second gives you steps, not a curve. Second, look how little it moves: thirty seconds of a fully saturated 2-CPU machine took the 1-minute figure from 0.10 to 0.93, under half the true answer of 2.0. The exponential average reaches only 1 − e^(−0.5) ≈ 39% of its target in 30 seconds; it needs about a minute to reach 63%, and several to converge.
Now look at second 31. The loops stopped at second 30. The machine is completely idle. The load reads 1.90 — and at second 40, ten seconds into an idle machine, it still reads 1.64. A high load average does not mean the machine is busy right now. It means it was busy recently.
The 5-minute number only reached 0.75 and the 15-minute number 0.29. A 30-second burst barely registers on the longer windows. That is the point of having three numbers: if the 1-minute figure is much higher than the 15-minute figure, something started recently. If all three are equal and high, it has been going on for a while.
Divide by nproc: 1.92 / 2 = 0.96. Just about saturated, no queue building. If that had been 4.00 / 2 = 2.0, half the runnable processes would be sitting in the queue at any moment.
C2 · Where the CPU time actually went
Load average tells you how many were waiting. It does not tell you what they were doing. For that, Linux splits every tick of CPU time into categories. You see them on the %Cpu(s) line in top, or as columns in vmstat and mpstat.
| Field | Name | What it means | What high values suggest |
| us | user | Running normal program code | The app is doing real work — or spinning in a loop |
| sy | system | Running kernel code on behalf of a program | Lots of syscalls: small reads/writes, many connections |
| ni | nice | User code from processes with a positive nice value | Background/batch jobs are the ones burning CPU |
| id | idle | Nothing to run | Spare capacity |
| wa | iowait | Idle, but at least one task is blocked on disk I/O | Storage is slow — not a CPU problem |
| st | steal | The hypervisor gave your CPU to someone else | Noisy neighbour, or you are being throttled |
| si / hi | soft/hard IRQ | Handling interrupts | Heavy network or disk interrupt load |
The corollary catches people out: wa goes down when you add more work. If you start a busy loop on a machine showing wa=40, the CPU now has something to run, so that time is counted as us instead. The disk is exactly as slow as before. The number just moved.
At the end of the day the shop owner writes down where the hours went:
- us — serving customers. The actual point of the shop.
- sy — paperwork the till forces on you: card authorisations, receipts, stock lookups. Necessary, but it is not selling.
- ni — restocking shelves, which you only do when nobody is at the counter.
- id — standing behind an empty counter.
- wa — standing behind an empty counter while waiting for the delivery van that has not shown up. You are just as idle as id. You simply know why nothing is moving.
- st — the landlord borrowed you for an hour to help in the shop next door. Your shop was open. You were not in it.
- si — the phone kept ringing and you kept answering it.
The wa trap in shop terms: if a customer walks in while you are waiting for the van, you serve them, and that hour goes in the "serving" column instead. The van is not one minute earlier. Your log just looks different.
Where the analogy stops working. A shop owner can only be in one place; a machine has many CPUs, each keeping its own log, and the summary you usually see is the average of all of them.
🧪 Exercise C2.1 — produce three different CPU profiles on purpose
# Tools we need. Install if missing.
which mpstat || sudo apt-get install -y sysstat
# --- Profile 1: pure user time ---
# A shell loop that does nothing at all in userspace and never calls the kernel.
for i in $(seq 1 $(nproc)); do timeout 10 bash -c 'while :; do :; done' & done
mpstat 1 5
wait
echo "--- profile 2 ---"
# --- Profile 2: heavy system time ---
# Millions of tiny syscalls. Each one crosses into the kernel and back.
for i in $(seq 1 $(nproc)); do
timeout 10 bash -c 'while :; do read -r x < /proc/uptime; done' &
done
mpstat 1 5
wait
echo "--- profile 3 ---"
# --- Profile 3: iowait ---
# Write a file, drop the cache so reads must hit the real disk, read it back.
dd if=/dev/zero of=/tmp/iotest bs=1M count=1024 conv=fsync 2>/dev/null
sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
( dd if=/tmp/iotest of=/dev/null bs=4k 2>/dev/null ) &
mpstat 1 5
wait
rm -f /tmp/iotest✅ Expected result — click to reveal
On a 2-CPU VM with a normal cloud disk (trimmed to the average lines):
Profile 1:
Average: CPU %usr %nice %sys %iowait %irq %soft %steal %idle
Average: all 98.71 0.00 1.09 0.00 0.00 0.20 0.00 0.00
--- profile 2 ---
Average: CPU %usr %nice %sys %iowait %irq %soft %steal %idle
Average: all 51.32 0.00 47.85 0.00 0.00 0.63 0.20 0.00
--- profile 3 ---
Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
Average: all 1.41 0.00 7.22 43.66 0.00 1.05 0.11 0.00 0.00 46.55What to read out of this.
If your Profile 3 shows %iowait near zero, your storage is simply too fast for this demo, and that is worth knowing rather than debugging. On an NVMe or a virtio disk backed by host page cache, a 1 GiB sequential read finishes in under two seconds and lands as %sys, not %iowait — measured on one lab VM: 1 GiB in 1.9 s at 556 MB/s, %iowait 1.32 and %idle 89.94. Add iflag=direct bs=4k to force small uncached requests, and confirm with iostat -x 1 that the device is really saturated. The lesson survives either way: whatever %iowait you do get is idle time, not busy time.
Profile 1 is what "the application is busy" looks like: %usr near 99, %sys near zero, %idle zero. The shell loop never asks the kernel for anything, so almost no time is spent in kernel code. If you see this on a real server, the fix is in the application, not the machine.
Profile 2 is the same amount of CPU burned, but roughly half of it is now %sys. The loop body is no more expensive than Profile 1's; the only change is that each iteration opens and reads /proc/uptime, and every one of those is a trip into the kernel and back. This is the signature of a chatty program: thousands of small reads instead of a few big ones, or a new connection per request instead of a pooled one. When you see %sys this high, reach for strace -c -p <pid> and count which syscall dominates.
Profile 3 is the one people misdiagnose. %iowait is 43.66 — but look at %idle: 46.55. Together that is 90% of the machine doing nothing. There is no CPU shortage here at all. %usr is 1.41. Adding CPUs to this machine would change nothing; the disk is the constraint.
And notice %sys at 7.22 in profile 3 — that is the kernel doing the block-layer and filesystem work around each read. Real I/O is never entirely free of CPU cost.
C3 · Steal time, throttling and pressure
What to use instead, per task: /proc/PID/schedstat. Field 2 is the nanoseconds that task spent on the run queue wanting a CPU it could not get — the same harm PSI measures, with no config dependency:
for i in $(seq 1 4); do ( end=$((SECONDS+5)); while [ $SECONDS -lt $end ]; do :; done ) & done
P=$!; sleep 4; echo "exec_ns run_delay_ns timeslices: $(cat /proc/$P/schedstat)"On a 2-CPU box with four loops that returns something like 2033076015 976965552 275 — 0.98 s of pure queueing against 2.03 s of running. (The system-wide /proc/schedstat needs CONFIG_SCHEDSTATS, which is often off; the per-PID file works regardless.)
Everything so far assumed the machine's CPUs are yours. On a cloud VM or inside a container, they are not. Two separate mechanisms can take CPU away from you, and they look different in the numbers.
Steal time (st) — the hypervisor took the CPU. Your VM has, say, 2 virtual CPUs. The physical host has 64 real cores shared among many VMs. When your virtual CPU wants to run and the hypervisor is running someone else on that core, the time is recorded as stolen. Your process was runnable. It just was not scheduled — and not by a decision your kernel made.
CFS throttling — your cgroup ran out of quota. (Yes, still "CFS", even though Section A2 told you EEVDF replaced it in 6.6. EEVDF replaced the picking logic; the bandwidth-control code that enforces cpu.max was never part of that change and keeps the old name everywhere — CONFIG_CFS_BANDWIDTH, cpu.cfs_quota_us, container_cpu_cfs_throttled_periods_total. Do not "correct" it to EEVDF in an interview.) Containers are usually given a CPU limit, expressed as "you may use X CPU-seconds in every 100 ms period". When the container uses its whole quota early in a period, every thread in it is frozen until the next period starts. This does not show up as steal. Your CPUs look idle. Your application looks stalled.
You rent a slot in a shared commercial kitchen.
Steal time is the landlord letting another tenant use your booked hour. You turned up on time, you had prep to do, and the hob was in use by someone else. Nothing you can change from inside your own operation — you either complain to the landlord or move to a kitchen with guaranteed slots.
Throttling is the prepaid electricity meter. You bought 100 units per hour. If you switch on every oven at once you can burn the whole hour's allowance in ten minutes, and then the power cuts out until the top of the next hour. The kitchen is empty, the hobs are free, and you are standing there unable to cook. Your average usage for the hour is exactly what you paid for. Your bread is still ruined.
The fix for the meter is not more power — it is switching the ovens on in sequence, or buying a bigger allowance.
Where the analogy stops working. A power cut is total; CFS throttling only freezes threads in that one cgroup. Other containers on the same host keep running perfectly.
🧪 Exercise C3.1 — read steal, pressure and throttling
# 1. Steal time. On bare metal this is always 0.00.
grep -E '^cpu ' /proc/stat
vmstat 1 3
# vmstat's first column, `r`, is the live count of runnable tasks - the
# un-smoothed version of the load average from C1. Compare it with `nproc`:
# `r` persistently above `nproc` is a real queue, and it says so immediately.
# 2. Pressure Stall Information: real time lost to waiting for CPU.
# NOTE: /proc/pressure may not exist - see the callout below. If it does not,
# use per-task run-queue wait instead: field 2 of /proc/PID/schedstat is the
# nanoseconds that task spent wanting a CPU it could not get.
cat /proc/pressure/cpu 2>/dev/null || echo "(no PSI on this kernel - use /proc/PID/schedstat)"
# 3. Throttling. Which cgroup version has the CPU controller here?
if grep -qw cpu /sys/fs/cgroup/cgroup.controllers 2>/dev/null; then
CG=/sys/fs/cgroup/throttletest; sudo mkdir -p $CG
echo "20000 100000" | sudo tee $CG/cpu.max >/dev/null # v2
else
echo "cgroup v1 or hybrid - the cpu controller lives under /sys/fs/cgroup/cpu"
CG=/sys/fs/cgroup/cpu/throttletest; sudo mkdir -p $CG
echo 20000 | sudo tee $CG/cpu.cfs_quota_us >/dev/null # v1
echo 100000 | sudo tee $CG/cpu.cfs_period_us >/dev/null
fi
# Put the shell in the cgroup BEFORE it starts spinning. Writing to
# cgroup.procs moves only the named PID - existing children stay put, which
# is why `timeout ... & echo $! > cgroup.procs` silently measures nothing.
sudo bash -c "echo \$\$ > $CG/cgroup.procs
end=\$((SECONDS+10)); while [ \$SECONDS -lt \$end ]; do :; done" &
sleep 11
cat $CG/cpu.stat
sudo rmdir $CG
# --- the original v2-only form, kept for reference ---
sudo mkdir -p /sys/fs/cgroup/throttletest
# "20000 100000" = 20 ms of CPU allowed per 100 ms period
echo "20000 100000" | sudo tee /sys/fs/cgroup/throttletest/cpu.max
# Put a busy loop into the cgroup
timeout 10 bash -c 'while :; do :; done' &
LOOP=$!
echo $LOOP | sudo tee /sys/fs/cgroup/throttletest/cgroup.procs
sleep 10
# 4. Read the throttling counters
cat /sys/fs/cgroup/throttletest/cpu.stat
cat /sys/fs/cgroup/throttletest/cpu.pressure
# Clean up
wait 2>/dev/null
sudo rmdir /sys/fs/cgroup/throttletest✅ Expected result — click to reveal
On a small shared cloud VM:
cpu 1049231 812 402117 28714455 19204 0 8841 12093 0 0
procs -----------cpu-----------
r b us sy id wa st
0 0 3 1 95 0 1
0 0 2 1 96 0 1
some avg10=0.42 avg60=0.31 avg300=0.19 total=48210331
full avg10=0.00 avg60=0.00 avg300=0.00 total=0The 8th number on the cpu line of /proc/stat is steal: 12093 ticks. At 100 ticks per second that is about 121 seconds of CPU taken by the hypervisor since boot. vmstat shows it as a steady st of 1 — small, but not zero. On a dedicated machine this column is always 0. Anything consistently above about 5 means you are sharing a host with someone hungry.
/proc/pressure/cpu is the more honest metric. some avg10=0.42 means: over the last 10 seconds, 0.42% of the time at least one runnable task was waiting for a CPU it could not get. That is a direct measure of harm. Load average tells you how many were waiting; PSI tells you how much time that waiting actually cost. (In /proc/pressure/cpu the full line is always zero — CPU full is undefined at the system level and is reported as zero for backward compatibility. In a cgroup's cpu.pressure it is not: a throttled cgroup has every task stalled at once, so full is exactly the line that lights up.)
Now the throttling part:
usage_usec 2013994
user_usec 1998110
system_usec 15884
nr_periods 100
nr_throttled 99
throttled_usec 7891204Read these carefully.
nr_periods 100 — the cgroup was scheduled across 100 periods of 100 ms, which is the 10 seconds it ran.
nr_throttled 99 — in 99 of those 100 periods it was frozen. It used its 20 ms allowance almost immediately and then sat stopped.
throttled_usec 7891204 — roughly 7.9 seconds out of 10 spent frozen. usage_usec 2013994 is about 2 seconds of actual CPU, which is exactly the 20% limit working as configured.
Here is the thing to take away: from inside that container, top would show the process at roughly 20% CPU and the host looking mostly idle. Nothing says "problem". Only nr_throttled and cpu.pressure reveal that the process spent 79% of its life stopped dead.
Which cgroup version you are on changes the field names. The block above is cgroup v2: usage_usec, user_usec, system_usec, throttled_usec. cgroup v1 reports nr_periods, nr_throttled and throttled_time — and throttled_time is in nanoseconds, with no usage_usec at all. A real v1 run of the same 20%-of-a-CPU limit for ten seconds gives nr_periods 96 / nr_throttled 94 / throttled_time 7359536636, which is the same 7.4 seconds of being frozen.
If you still see nr_throttled 0 — the loop's PID did not land in the cgroup. Writing to cgroup.procs moves only that one PID; if the shell forked again the child may be elsewhere. cat $CG/cgroup.procs should list it while it runs.
🎯 Interview questions — Load and accounting
Q. What does a load average of 5 mean on a Linux server?
On its own, nothing — you have to know the CPU count. It means that on average five threads were either running, waiting for a CPU, or blocked in uninterruptible disk or network I/O over that window. On a 16-CPU host that is a quiet machine. On a 2-CPU host, three are queuing at any moment.
The details that separate candidates: most people say "5 means the CPU is 500% busy" or forget the CPU count entirely. The two details that mark out a strong answer are (1) Linux includes D-state tasks, unlike most other Unix systems — so a storage outage can push load to 50 on a machine whose CPUs are completely idle, and (2) the numbers are exponentially smoothed and recomputed only every five seconds, so they lag both the start and the end of an incident by minutes. If you want to know whether the CPU is busy now, load average is the wrong tool — read vmstat 1's r column, or /proc/pressure/cpu for how much time was actually lost to waiting.
Q. top shows 40% iowait. Do you need faster CPUs?
No. iowait is a subset of idle time — the CPU has nothing to run and something on it is blocked on disk. Forty percent iowait means the CPU was free 40% of the time and the storage was the thing holding work up. More or faster CPUs would change nothing.
The details that separate candidates: the follow-up that catches people is "what happens to iowait if I start a CPU-heavy job on that machine?" The answer is that it drops, often to near zero, because the CPU now has something to run while it waits — that time is reclassified from wa to us. The disk has not got one millisecond faster. This is why iowait is a poor alerting metric: it is diluted by unrelated load and inflated by having spare capacity. Measure the storage directly instead — iostat -x 1 for await and queue depth, or /proc/pressure/io, which reports the time actually lost to I/O regardless of what else the CPU was doing.
Q. A container is reported as slow, but its CPU usage sits at 50% of its limit. What is your first hypothesis?
CFS throttling with bursty threads. A CPU limit is a quota per 100 ms period, not a smooth rate. If the workload wakes many threads at once it can exhaust the whole period's quota in a few milliseconds, and every thread is then frozen until the next period. Averaged over a minute the usage looks like half the limit; in reality the app is stopping dead many times a second. I would check cpu.stat in the container's cgroup for nr_throttled and throttled_usec.
The details that separate candidates: two things mark out a strong answer. First, naming the mechanism precisely — quota per period, all threads in the cgroup frozen together, not slowed. Second, knowing the fix is usually not "raise the limit": reducing the thread pool or worker count so the concurrency matches the quota often removes the throttling entirely, because the same total work is then spread across the whole period instead of crammed into the start of it. Mentioning cpu.pressure as the metric that shows the cost in real time is a further step up.
Q. What is steal time and what would you do about it?
Steal time is CPU time your virtual machine wanted but the hypervisor gave to another guest. Your kernel had a runnable task and no physical core to put it on. It appears as the st column in top/vmstat and the 8th field of the cpu line in /proc/stat. Nothing inside the guest can fix it — the options are moving to a dedicated or CPU-pinned instance type, or moving the workload to a quieter host.
The details that separate candidates: recognising that steal time makes every in-guest timing measurement unreliable, not just CPU numbers. A request that took 200 ms of wall clock may have had 80 ms stolen; your application's own latency metrics will blame the database when the CPU was simply absent. Also worth saying: burstable instance types (the ones with CPU credits) show the same symptom for a different reason — once credits are exhausted the hypervisor caps you, which registers as steal. Checking whether the credit balance hit zero before blaming a noisy neighbour is the practical first move.
🔍 Part D · Diagnosing "the server is slow"
D1 · Four questions, in order
"The server is slow" is not a diagnosis. Everything in Parts A to C exists so you can turn it into one. The trick is to ask the questions in an order where each answer eliminates whole categories of cause.
Ask them in this order, always:
- Is the CPU actually busy? — vmstat 1 5. If id is high, stop looking at CPU.
- Busy doing what? — us vs sy vs wa vs st. This picks which direction to go.
- Is anything waiting that should not be? — load average against nproc, and /proc/pressure/cpu.
- Who is responsible? — pidstat, per-thread, per-cgroup.
Most people skip straight to step 4, find the process using the most CPU, and blame it — even when the machine was 90% idle and the real problem was a slow disk.
Diagram source
flowchart TD
A["Server is slow"] --> B{"vmstat 1<br>Is id low?"}
B -->|"No, id is high"| C{"Is wa high?"}
C -->|"Yes"| D["Storage is the limit<br>iostat -x 1"]
C -->|"No"| E["Not a CPU problem<br>Check network, locks, DB"]
B -->|"Yes, id is low"| F{"Which field dominates?"}
F -->|"us"| G["App code burning CPU<br>Profile it: perf top"]
F -->|"sy"| H["Too many syscalls<br>strace -c -p PID"]
F -->|"st"| I["Hypervisor taking CPU<br>Move host or instance type"]
F -->|"si"| J["Interrupt load<br>Check /proc/interrupts"]
G --> K{"Inside a container?"}
H --> K
K -->|"Yes"| L["Check cpu.stat<br>nr_throttled"]
K -->|"No"| M["Attribute per thread<br>pidstat -t 1"]Someone says "I feel terrible". A good doctor does not open with a scan of the most expensive organ. They take temperature, pulse, blood pressure — cheap, fast measurements that rule out whole families of illness before any expensive test is ordered.
vmstat 1 is the thermometer. It takes one second, costs nothing, and its answer tells you which specialist to call. Going straight to pidstat and picking the biggest process is like scheduling heart surgery because the patient mentioned chest discomfort — sometimes right, often expensive and wrong.
Where the analogy stops working. A doctor cannot re-run yesterday's vitals; you can, if you kept the metrics. Historical data is the one advantage you have over a physician, and most people waste it by storing a single merged "CPU %" number.
🧪 Exercise D1.1 — Run the four questions against a machine you have broken on purpose
# Question 1 + 2: is the CPU busy, and doing what?
echo "=== baseline ==="; vmstat 1 3
# Manufacture a us-dominated fault and re-ask
for i in $(seq 1 $(nproc)); do timeout 8 bash -c 'while :; do :; done' & done
echo "=== us-dominated ==="; vmstat 1 5
wait
# Manufacture a sy-dominated fault and re-ask
for i in $(seq 1 $(nproc)); do timeout 8 bash -c 'while :; do read -r x < /proc/uptime; done' & done
echo "=== sy-dominated ==="; vmstat 1 5
wait
# Question 3: how many were waiting, relative to capacity?
echo "loadavg: $(cat /proc/loadavg) nproc: $(nproc)"
# /proc/pressure/cpu is the right tool, but see the C3 callout - it may not exist.
# Per-task run-queue wait works everywhere: field 2 = ns spent waiting to run.
for i in $(seq 1 4); do ( end=$((SECONDS+5)); while [ $SECONDS -lt $end ]; do :; done ) & done
P=$!; sleep 4
echo "PID $P schedstat (exec_ns run_delay_ns timeslices): $(cat /proc/$P/schedstat)"
wait
# Question 4: who is responsible?
for i in $(seq 1 3); do ( end=$((SECONDS+6)); while [ $SECONDS -lt $end ]; do :; done ) & done
pidstat -t 1 3 | head -15
wait✅ Expected result — click to reveal
=== baseline ===
r b swpd free buff cache si so bi bo in cs us sy id wa st gu
0 0 0 5796840 20376 1754600 0 0 0 0 289 401 1 4 95 0 0 0
=== us-dominated ===
2 0 0 5796780 20376 1754600 0 0 0 0 922 955 96 3 0 0 0 0
=== sy-dominated ===
2 0 0 5796780 20376 1754600 0 0 0 0 8891 4210 52 47 0 0 0 0
loadavg: 3.41 1.12 0.44 5/191 28066 nproc: 2
PID 28112 schedstat (exec_ns run_delay_ns timeslices): 2033076015 976965552 275
Average: UID TGID TID %usr %system %guest %wait %CPU CPU Command
Average: 0 28150 - 65.67 0.33 0.00 33.00 66.00 - bash
Average: 0 - 28150 65.67 0.33 0.00 33.00 66.00 - |__bashWhat to read out of this — the four questions, answered in order.
Q1, is the CPU busy? id goes from 95 to 0 in both fault profiles. That is the whole of question one, and it costs one command.
Q2, doing what? The two faults have identical id 0 and opposite compositions: us 96 / sy 3 versus us 52 / sy 47. Same saturation, completely different investigation — one sends you to a profiler, the other to a syscall tracer. Notice also in and cs jumping from a few hundred to nearly nine thousand in the second profile: syscall-heavy work is visible in the interrupt and context-switch columns too.
Q3, how many were waiting? The load average is history, so read r instead: 2 with nproc 2 means the queue is exactly full. And /proc/PID/schedstat gives you the number PSI would have, per task: 0.98 s of run-queue wait against 2.03 s of running. That task spent a third of its life wanting a CPU it could not get — which is the harm, quantified, on a kernel with no /proc/pressure.
Q4, who? Only now do you name a process. pidstat -t also prints %wait, which is the per-task version of the same queueing number: 33% here, matching the schedstat reading.
Your numbers will differ, and the ratios are what matter. If sy dominates on a machine you did not deliberately break, that is a real finding — healthy application servers are us-dominated.
D2 · Finding the guilty thread
Once you know the CPU is genuinely busy with user or system time, you need to name the culprit. Two details make the difference between finding it and guessing.
Look per thread, not per process. From Module 06 you know a process can hold many threads, each with its own TID. top and ps show the process total by default. A Java service showing 300% CPU across 200 threads tells you nothing; the same service with one thread at 99% and the rest idle tells you exactly where to look. pidstat -t and top -H break it out.
The %CPU column in ps is a lifetime average. ps aux computes CPU percent over the entire life of the process. A process that pegged a core for six hours last night and has been idle since will still show a high number. For "what is happening right now", use top, pidstat 1, or two reads of /proc/PID/stat a second apart. This trips people up constantly.
The building manager gets a noise complaint about flat 3B. Standing outside the front door you can hear it is loud, but "flat 3B is loud" is not actionable — six people live there. You have to walk in and find which room the noise is coming from. That is top -H: the flat is the process, the rooms are the threads.
And the lifetime-average trap: asking the neighbours "how noisy is 3B?" gets you an opinion formed over two years of living next door. Asking "is 3B noisy right now?" is a different question with a different answer. ps gives you the two-year opinion; pidstat 1 puts an ear to the door.
Where the analogy stops working. Rooms in a flat stay put. Threads move between CPUs constantly, so a thread's TID is stable but the core it runs on is not — which is why per-CPU views and per-thread views answer different questions.
🧪 Exercise D2.1 — one hot thread hiding inside a busy process
# NOTE: `( ... ) &` forks a SUBSHELL - a process, not a thread. The script below
# builds a process group, which is enough to show the averaging problem. For a
# genuinely multi-threaded target - one PID, one hot thread - use this instead,
# and `pidstat -t` will show one TGID at ~100% with one |__ TID at ~99%:
#
# python3 - <<'PY' &
# import threading, time
# def idle():
# while True: time.sleep(5)
# def hot():
# while True: pass
# for _ in range(3): threading.Thread(target=idle, daemon=True).start()
# threading.Thread(target=hot, daemon=True).start()
# time.sleep(25)
# PY
cat > /tmp/mixed.sh <<'EOF'
#!/bin/bash
# three quiet workers
for i in 1 2 3; do ( while :; do sleep 5; done ) & done
# one hot worker
( while :; do :; done ) &
wait
EOF
chmod +x /tmp/mixed.sh
timeout 25 /tmp/mixed.sh &
sleep 2
# The process-level view: which process group is burning CPU?
ps -eo pid,ppid,%cpu,comm --sort=-%cpu | head -6
# The per-thread / per-child view, sampled live over 1-second intervals
which pidstat || sudo apt-get install -y sysstat
pidstat -t 1 3 | head -20
# Now compare against the lifetime average for the same PIDs.
# Sleep past the `timeout 25` so this really is a post-mortem reading.
sleep 24
ps -eo pid,%cpu,etime,time,comm --sort=-%cpu | head -6
wait 2>/dev/null
rm -f /tmp/mixed.sh✅ Expected result — click to reveal
PID PPID %CPU COMMAND
5121 5104 96.4 bash
5104 4870 0.1 bash
5118 5104 0.0 bash
5119 5104 0.0 bash
5120 5104 0.0 bash
Linux 6.8.0-40-generic (vm-01) 08/21/26 _x86_64_ (2 CPU)
10:14:22 UID TGID TID %usr %system %CPU CPU Command
10:14:23 1000 5104 - 0.00 0.00 0.00 1 bash
10:14:23 1000 - 5104 0.00 0.00 0.00 1 |__bash
10:14:23 1000 5121 - 98.02 1.98 100.00 0 bash
10:14:23 1000 - 5121 98.02 1.98 100.00 0 |__bash
10:14:23 1000 5118 - 0.00 0.00 0.00 1 bash
10:14:23 1000 5119 - 0.00 0.00 0.00 0 bashWhat to read out of this.
Five bash processes, all with the same command name, all children of the same parent. ps sorted by %cpu puts PID 5121 on top at 96.4 — but if you had only looked at the parent (5104, at 0.1%) you would have concluded the script was idle. The parent's CPU number says nothing about its children.
pidstat -t splits TGID (the thread group, i.e. the process) from TID (the individual thread). Lines with a number in TGID are the process; lines with |__ are its threads. Here each process happens to be single-threaded, so they mirror each other — but on a real multi-threaded service this is where you would see one TID at 99 and forty others at 0.
Also note the CPU column: 5121 is on CPU 0, the quiet ones are spread across both. If you ran top and pressed 1, CPU 0 would read near 100% while CPU 1 sat idle — and the summary line would say 50%. Half-busy in the average, completely saturated on the core that matters.
Now the second ps, taken after the loop was killed by timeout (note the sleep 24 — with a shorter sleep the loop is still running and the reading means something different):
PID %CPU ELAPSED TIME COMMAND
5121 96.1 00:25 00:00:24 bash
5104 0.1 00:25 00:00:00 bashThe process is gone or idle, yet %CPU still reads 96.1. That is the lifetime average: 24 seconds of CPU over 25 seconds of elapsed time. ps is telling you about the past. TIME (total CPU consumed) and ELAPSED (wall clock) are the two numbers it divides — seeing them side by side makes the trap obvious.
🎯 Interview questions — Diagnosis
Q. Walk me through what you run, in order, when someone says a server is slow.
uptime and nproc first, to see the load relative to capacity and how long it has been going. Then vmstat 1 5 to find out whether the CPU is actually busy and, if so, whether the time is us, sy, wa or st. That single line picks the direction: wa sends me to iostat -x 1, sy to strace -c, us to a profiler, st to the hypervisor or instance type. Only then do I attribute it — pidstat -t 1 for the thread, systemd-cgtop or cpu.stat for the cgroup. And if the CPU turns out to be mostly idle, I stop looking at CPU entirely and go to the network, the database, or lock contention in the app.
The details that separate candidates: the ordering itself is the answer — cheap measurements that eliminate categories before expensive ones that identify individuals. Two additions raise it further. First, capture before you mitigate: vmstat 1 30 redirected to a file, because a restart destroys the evidence. Second, mentioning PSI (/proc/pressure/cpu and /proc/pressure/io) as the metric that quantifies harm rather than activity — utilisation tells you the machine is busy, pressure tells you how much time was actually lost to waiting, which is what the user is complaining about.
Q. ps aux shows a process at 180% CPU. What does that mean, and what does it not tell you?
It means the process consumed 1.8 CPU-seconds per second of its life on average, so it is multi-threaded and used roughly two cores' worth. What it does not tell me is anything about right now: ps computes %CPU as total CPU time divided by elapsed time since the process started. A process that saturated two cores for an hour this morning and has slept since will still report a high number.
The details that separate candidates: naming the lifetime-average behaviour is the main thing — most people assume it is instantaneous like top. Beyond that: the number is a sum across threads, so 180% could be two threads at 90% or ninety threads at 2%, and those have completely different fixes. top -H or pidstat -t splits it out. And on a machine with fewer cores than the number suggests, or inside a cgroup with a CPU limit, the value can be capped in ways that hide the real demand — cpu.stat's nr_throttled shows the demand that was refused.
🏁 Part E · Practice, docs and self-check
E1 · Production practice
Things that actually go wrong, and what the earlier parts tell you to do about them.
| Symptom in production | What is really happening | What to run | The fix |
| Load average 40 on a 4-CPU box, CPUs idle | Processes stuck in D state on a hung NFS mount or a failing disk — Linux counts them in load | ps -eo state,pid,wchan,comm | grep '^D' | Fix the storage. No amount of CPU helps. |
| Pod latency spikes, CPU usage ~50% of limit | CFS throttling: quota burned early in each 100 ms period, all threads frozen for the rest | cat /sys/fs/cgroup/.../cpu.stat | Raise the limit or reduce worker threads so work spreads across the period |
| A batch job starves the web service on the same host | nice alone is a weak hint; with enough batch threads it still wins | ps -eo pid,ni,cls,comm --sort=-ni | nice plus a cgroup cpu.weight, or move the batch job off the host |
| Whole machine freezes when a "high priority" job runs | Someone set SCHED_FIFO and the thread never yields — it can starve the kernel itself | chrt -p PID, sysctl kernel.sched_rt_runtime_us | Use SCHED_OTHER with negative nice unless it is genuinely real-time work |
| App "slow" but every metric looks normal | Steal time on a shared or burstable instance; timings inside the guest are unreliable | vmstat 1, look at st; check credit balance | Dedicated/pinned instance type, or a quieter host |
| One core pinned at 100%, dashboard says 12% CPU | Single-threaded bottleneck averaged across 8 cores | top then press 1; pidstat -t 1 | Parallelise the hot path, or accept the ceiling and shard |
| Context switches in the hundreds of thousands per second | Too many runnable threads for the core count, or a lock being contended | vmstat 1 (cs column), pidstat -w -t 1 | Reduce thread pool size to roughly match nproc |
| CPU alert fires a minute after users complain, clears late | Alerting on load average, which is a smoothed lagging average | Compare /proc/loadavg with /proc/pressure/cpu | Alert on PSI or on us+sy from /proc/stat deltas |
E2 · Capstone — four "the server is slow" tickets
Four tickets land in your queue. For each one: decide what you would run in order, what you expect to see, and what the fix is. Work through them before opening the answers.
Ticket 2. A Kubernetes pod with limits.cpu: 1. Latency p99 has tripled. The dashboard shows the container at 0.52 CPU — barely half its limit. Memory is fine. What is your hypothesis and how do you confirm it?
Ticket 3. A build server. top shows %Cpu(s): 4.1 us, 6.8 sy, 0.0 ni, 45.0 id, 43.8 wa, 0.0 hi, 0.3 si, 0.0 st. A colleague has already opened a ticket to double the CPU count. What do you tell them?
Ticket 4. A data-processing box, 8 CPUs. A nightly batch job was given nice 19 so it "cannot affect anything else". The API on the same host still degrades every night. Why did nice not save you?
✅ Ticket 1 — worked answer
Run, in order: nproc (confirm 4) → vmstat 1 5 → if id is high, ps -eo state,pid,wchan,comm | grep '^D' → otherwise pidstat -t 1.
The reasoning. Load 12.44 on 4 CPUs is 3.1 runnable-or-blocked tasks per CPU, which sounds like a CPU shortage. But load average on Linux includes D-state processes, so the number alone cannot distinguish "12 processes fighting for CPU" from "3 processes computing and 9 stuck on a dead mount".
vmstat settles it in one second. If us + sy is near 100 and id near 0, it is genuinely CPU-bound and more cores will help. If id is high and wa is high, adding CPUs changes nothing.
Also note the trend: 12.44 / 11.90 / 9.02 — the 1-minute figure is the highest, so it is still getting worse, not recovering.
Do you approve the resize? Not yet. Approving a 4× cost increase on the strength of a number you have not divided by nproc, and whose composition you have not checked, is exactly the mistake this module exists to prevent. One vmstat run justifies the decision either way.
✅ Ticket 2 — worked answer
Hypothesis: CFS throttling with bursty concurrency.
Why the metric lies. limits.cpu: 1 is not "one CPU continuously". It is 100 ms of CPU per 100 ms period. If the app has, say, 16 worker threads that wake together on each request batch, they can consume the whole 100 ms allowance in about 6 ms of wall clock — and then every thread in the cgroup is frozen for the remaining 94 ms. Averaged over a minute, usage reads 0.52 CPU. In reality the process is stopping dead ten times a second, which is precisely what destroys p99 while leaving p50 fine.
Confirm it:
# From the node, find the pod's cgroup, then:
cat /sys/fs/cgroup/.../cpu.stat
# nr_periods, nr_throttled, throttled_usec
cat /sys/fs/cgroup/.../cpu.pressureIf nr_throttled / nr_periods is meaningfully above zero — anything past a few percent — the hypothesis is confirmed. In Prometheus the same signal is container_cpu_cfs_throttled_periods_total over container_cpu_cfs_periods_total.
The fix. Two options, and the second is the one people miss. Raise the limit, or reduce the worker/thread count so the same work is spread across the period instead of crammed into its start. A service with 16 threads and a 1-CPU limit is asking to be throttled; the same service with 2 threads often shows the same throughput and no throttling at all.
✅ Ticket 3 — worked answer
What you tell them: the CPU is 88.8% idle. Do not buy more of it.
Read the line properly. 45.0 id plus 43.8 wa is 88.8% of CPU time doing nothing. iowait is a kind of idle — it means the CPU had nothing runnable and something on it was blocked on disk. Only 10.9% of the time was spent doing actual work (4.1 us + 6.8 sy).
Where to look instead:
iostat -x 1 5 # await, %util, queue depth per device
cat /proc/pressure/io # time actually lost to I/O
ps -eo state,pid,wchan,comm | grep '^D'The trap to warn them about. If they run a CPU-heavy job on this box, wa will drop toward zero and us will rise — and they will conclude the problem is fixed. The disk will be exactly as slow as before. iowait is diluted by unrelated load, which is why it is a poor thing to alert on. /proc/pressure/io measures the lost time directly regardless of what else the CPU is doing.
The 6.8 sy is also a hint: that is the kernel's block-layer and filesystem work around each I/O. Real disk traffic always carries some system time with it.
✅ Ticket 4 — worked answer
Why nice did not save you. nice biases the share, it does not impose a cap. Each nice level multiplies a process's weight by roughly 1.25. At nice 19 a single thread gets a very small slice against a nice-0 competitor — but the batch job does not have one thread. If it runs 64 parallel workers on 8 CPUs, sixty-four small shares add up to a large share, and the API's few threads are squeezed.
There are two further reasons it under-delivers:
- nice only governs CPU. The batch job's disk and page-cache pressure is completely unaffected. If the API slows because the batch job evicted its data from the page cache or saturated the disk queue, nice 19 was never going to help.
- nice is per process, and autogrouping can change the picture. On desktop-oriented configurations kernel.sched_autogroup_enabled groups by session, which changes how nice values compose. Check sysctl kernel.sched_autogroup_enabled before reasoning about it on a server.
What to do instead:
# Cap the batch job's share at the cgroup level, not the process level
systemd-run --unit=nightly-batch --slice=batch.slice \
-p CPUWeight=10 -p CPUQuota=200% -p IOWeight=10 \
/usr/local/bin/nightly.shIf you are not on a systemd host — containers, minimal images, many lab VMs — do the same thing directly with cgroups. On v2: mkdir /sys/fs/cgroup/batch; echo 10 > .../cpu.weight; echo "200000 100000" > .../cpu.max. On v1 the knobs are named differently: cpu.shares (relative weight, default 1024) and cpu.cfs_quota_us / cpu.cfs_period_us. v2's cpu.weight (range 1–10000, default 100) is the successor to v1's cpu.shares — same idea, different scale, so a cpu.weight of 10 is roughly a cpu.shares of 100. And systemd-cgtop shows nothing in its CPU column without CPUAccounting=yes; without it, read cpu.stat yourself.
CPUWeight sets the relative share for the whole group regardless of how many threads it spawns; CPUQuota=200% is a hard ceiling of two cores' worth; IOWeight covers the disk contention that nice never touched. The general lesson: process-level knobs do not contain multi-process workloads. Use cgroups for that.
E3 · Documentation reference
| Topic | Where to read it | Why this one |
| Scheduler concepts and all policies | sched(7) | The single best overview of SCHED_* policies, nice, and RT limits |
| EEVDF, the current default scheduler | Kernel docs — EEVDF | Default since 6.6; most interview material still describes CFS |
| CFS, for context and older kernels | Kernel docs — CFS design | What EEVDF replaced; still what you will meet on older LTS kernels |
| All scheduler documentation | Kernel docs — Scheduler index | Includes RT throttling, domains, and capacity-aware placement |
| Load average fields | proc_loadavg(5) | Defines the 4th field (running/total) that nobody reads |
| The /proc/stat CPU columns | proc_stat(5) | Where us/sy/wa/st actually come from, in order |
| Pressure Stall Information | Kernel docs — PSI | The modern replacement for guessing from load average |
| cgroup v2 CPU controller | Kernel docs — cgroup v2 | cpu.max, cpu.weight, cpu.stat — the throttling story |
| Changing policy and priority | chrt(1) | Syntax for policy and priority. The warning about runaway SCHED_FIFO is in sched(7), under "Limiting the CPU usage of real-time and deadline processes" — read that first |
| Setting nice values | nice(1) · renice(1) | renice(1) is the one that states the rule — "an unprivileged user can only increase the nice value"; sched(7) names the capability, CAP_SYS_NICE |
| CPU pinning | taskset(1) | Masks are hex by default — the -c list form is easier to get right |
| Per-process and per-thread stats | pidstat(1) | -t for threads, -w for context switches |
| Per-CPU stats | mpstat(1) | Breaks the misleading average into per-core reality |
| Live process view | top(1) | Press 1 for per-CPU, H for threads — both are in the man page |
E4 · Self-assessment
Answer these out loud before moving to Module 08. If you cannot, the section to reread is named.
- What is the scheduler actually choosing between at any instant? (A1)
- Why is "fair share" not the same as "equal priority"? (A2)
- Which scheduler is the Linux default today, and what did it replace? (A2)
- Roughly what does one nice level do to a process's CPU share, and why does nice 19 fail to protect a service from a 64-thread batch job? (A3, E2 ticket 4)
- What is the difference between a voluntary and an involuntary context switch, and what does a high count of each tell you? (A4)
- Why is SCHED_FIFO dangerous, and what stops one FIFO thread from freezing the whole machine? (B1)
- When is pinning a process to a CPU a good idea, and what does it cost you? (B2)
- A load average of 6 — is that bad? What do you need to know before answering? (C1)
- Why can load average be high on a machine whose CPUs are completely idle? (C1)
- Why does iowait go down when you add CPU-heavy work to a machine? (C2)
- What is steal time, and why does it make your application's own latency metrics untrustworthy? (C3)
- A container is at 50% of its CPU limit and its latency is terrible. What is your first hypothesis and which file confirms it? (C3, E2 ticket 2)
- What four questions do you ask, in order, when told "the server is slow"? (D1)
- Why is %CPU in ps aux a poor answer to "what is using the CPU right now"? (D2)
E5 · Sources
Kernel documentation
Manual pages
· sched(7) · cgroups(7)
· nice(1) · renice(1) · chrt(1) · taskset(1)
· proc_loadavg(5) · proc_stat(5) · proc_pid_stat(5)
· top(1) · vmstat(8) · mpstat(1) · pidstat(1)
Standards
· POSIX.1-2024 System Interfaces — index — see sched_setscheduler, sched_setparam, sched_get_priority_max