Module 14 — Logs, Troubleshooting, and Performance
Updated 2 September 2026
This is where the whole track converges: the module about incidents. "The server is slow," "something's broken," "it worked yesterday" — this teaches the systematic method that turns panic into diagnosis. You'll learn to read a machine's vital signs, find the story in its logs, and follow a repeatable troubleshooting method instead of guessing. Nearly every senior-level Linux interview is, at heart, "here's a broken server — what do you do?"
Legend used throughout: 🧠 concept → 🧩 analogy → 🧪 exercise → ✅ expected result (hidden) → 🎯 interview questions (hidden)
You need the whole track so far, Modules Module 1 — What Linux Is–Module 13 — Networking — this module composes everything: processes (8), services and journalctl (11), storage (12), networking (13). Load-bearing especially: top/ps/states (Module 8), the journal (Module 11), df/du/free (Module 12). Tools: an Ubuntu 24.04 machine; we install sysstat (for iostat) in Part C. Everything here is read-only inspection — safe to run anywhere, which is the point: diagnosis observes before it acts.
Goes deeper: the kernel mechanics behind these signals — scheduling and load, virtual memory and the OOM killer, the I/O path and iowait, and a formal performance methodology — are four companion modules in the Operating Systems track (OS 07–10 and 14). This module is the hands-on operator's view; those are the theory.
Part A — Reading the vital signs
A1. uptime and load average
🧠 uptime gives a one-line health summary: how long the machine's been up, how many users, and the load average — three numbers that are the single most-quoted, most-misunderstood metric in Linux. Load average is the average number of processes wanting to run (running, or waiting to run, plus on Linux those in uninterruptible I/O wait — Module 8's D state) over the last 1, 5, and 15 minutes. The three timescales show trend: 2.0, 1.0, 0.5 means load is rising (recent higher than older); 0.5, 1.0, 2.0 means it's falling. Reading the trend, not just the instant, is the skill.
Load average is the number of shoppers wanting to check out. But "8 people waiting" only means something once you know how many tills are open: 8 shoppers across 8 tills is one each — smooth. 8 shoppers at 1 till is a long, angry queue. The three numbers (1/5/15 min) are whether the queue is growing (a rush starting) or clearing (winding down).
Where the analogy stops working. Shoppers only want the till — they're all "runnable." Linux load also counts shoppers stuck waiting for a price check (uninterruptible I/O wait, D state) who aren't using a till at all. So a high Linux load can mean "CPU overloaded" or "everything's stuck waiting on a slow disk" — two completely different problems with the same load number. Distinguishing them (is it us/sy CPU, or wa iowait?) is what top and vmstat are for, and why load alone never finishes a diagnosis.
🧪 Exercise A1.1 — Read the load, in context (read-only, safe anywhere)
uptime # the one-line summary with the three load numbers
nproc # how many cores? (load is meaningless without this)
cat /proc/loadavg # the kernel's raw load figures (what uptime reads)✅ Expected result — click to reveal
23:21:55 up 49 min, 0 users, load average: 0.07, 0.02, 0.00
2
0.07 0.02 0.00 1/82 2245What to read out of it (your numbers reflect your machine's activity):
- load average: 0.07, 0.02, 0.00 on a 2-core machine (nproc = 2) — essentially idle: 0.07 processes wanting to run against 2 cores available, ~3.5% busy. The three numbers falling (0.07 → 0.02 → 0.00) means even that flicker of activity is subsiding.
- The load-÷-cores math is the whole point: to know if load average: 3.5 is fine, you need nproc. On this 2-core box, 3.5 would mean overloaded (queuing); on a 4-core box it'd be busy-but-fine. Never report a load number without the core count.
- /proc/loadavg adds two fields uptime hides: 1/82 = 1 process currently runnable out of 82 total, and 2245 = the PID of the most recently created process. The raw source (Module 2's /proc) behind the friendly uptime.
A2. top — the live triage screen
🧠 Module 8 introduced top as a process viewer; here it's the incident triage screen — the "server is slow, what first?" answer. Its header answers the four triage questions in order: load (A1, first line), the task census (running/sleeping/zombie — Module 8), the %Cpu(s) line (where the CPU's time goes), and the memory lines. Then the process table, sortable by CPU (P) or memory (M), names the culprit. The single most diagnostic field is the %Cpu(s) breakdown, because it tells you what kind of slow you have — and each kind sends you somewhere different.
The %Cpu(s) letters, and what each high value means:
| Field | Means | If it's high… |
|---|---|---|
| us | User code — your programs computing | An app is CPU-bound; find it in the process table |
| sy | System — kernel work (syscalls, Module 1) | Heavy syscall load; often lots of I/O or context switching |
| wa | I/O wait — CPU idle, waiting on disk | Storage is the bottleneck, not the CPU — go to Part C / Module 12 |
| id | Idle — doing nothing | The CPU is not the problem; look elsewhere (memory, I/O, network, locks) |
| st | Steal — a VM's CPU taken by the hypervisor | A noisy neighbour on shared cloud hardware; not your fault, but real |
top is the triage nurse who, in ten seconds, reads the vital signs and routes you: chest pain here, broken bone there. The %Cpu(s) line is those vitals — it doesn't treat anything, but it sorts the problem into the right category (CPU-bound? I/O-bound? idle-but-slow?), which is 90% of solving it fast.
Where the analogy stops working. A triage nurse sees the patient in front of them. top shows only this instant, refreshed and discarded — the spike that caused last night's page is simply not there this morning (which is why history-keeping monitoring exists, A-part orange callout). And top can mislead: a process can be the victim rather than the cause (starved by something else), so the nurse's first read is a hypothesis to confirm, not a verdict.
🧪 Exercise A2.1 — Triage in ten seconds (read-only, safe anywhere)
top -b -n1 | head -12 # -b -n1: one non-interactive snapshot (scriptable; the header is the point)
# Interactively: run `top`, watch one refresh, press M (sort by memory), P (by CPU), q to quit✅ Expected result — click to reveal
top - 23:21:56 up 49 min, 0 users, load average: 0.07, 0.02, 0.00
Tasks: 62 total, 2 running, 60 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.5 us, 0.0 sy, 0.0 ni, 95.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 8023.6 total, 6907.4 free, 697.3 used, 705.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 7326.3 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
467 root 20 0 1809260 43356 27168 S 10.0 0.5 0:10.49 environme+What to read out of it (a healthy idle machine here; a sick one tells a louder story):
- Read the header top-to-bottom as a checklist: load (idle), 0 zombie (Module 8 — good), %Cpu 95.5 id (CPU almost entirely idle — so slowness, if any, is not CPU), memory (7.3G available — plenty, Module 12's available column). Four vital signs, ten seconds, and you've already ruled out CPU and memory.
- On a real incident this header is where you'd see the tell: %Cpu 0.0 id, 90 wa (disk-bound → Part C), or %Cpu 95 us with one process at 400% (CPU-bound app), or memory available near zero with swap churning (memory pressure → OOM territory, Part D). The header categorizes before you touch the process table.
- The process table, sorted, names names — %CPU and %MEM columns, the RES (real memory) figure that matters more than VIRT (Module 12/OS track). M and P re-sort live for "who's eating memory?" vs "who's eating CPU?"
You never SSH to 500 boxes to run uptime. Load and CPU become time-series: an agent like Prometheus node_exporter scrapes each host every ~15s, and load ÷ cores and %Cpu land in a dashboard for the whole fleet. The skill inverts — the question is no longer "read this one number" but "which of my 500 hosts is the outlier?", which you answer from a heatmap, not a terminal. You alert on sustained load ÷ cores (say, >1.5 for 5 minutes), never on a single spike, because at 500 hosts something is always momentarily busy. top doesn't scale — it's the tool you reach for after the dashboard has already told you which host to look at.
Part A — Interview questions
🎯 "A server is slow. Which command do you run first and why?" — asked verbatim in GeeksforGeeks' Linux Interview Questions (70+ questions)
top (or htop) — because one screen triages what kind of slow it is before you commit to any theory: the %Cpu(s) line (user-bound? kernel? iowait? idle?), load-vs-cores (overloaded or not), memory/swap pressure, and the ranked process table naming the culprit. Each reading routes the investigation differently — high wa → storage, high us + one hot process → that app, available memory near zero + swap churn → memory. uptime+nproc for the load-in-context one-liner is the even-faster first glance.
The details that separate candidates: naming the decision tree the header drives, not just the tool; load ÷ cores (a load number is meaningless without nproc); iowait as "CPU fine, disk is the bottleneck"; and the honest caveat that top shows now, so you confirm when it started against historical monitoring. (This is Module 8's question, answered here with the full performance vocabulary.)
🎯 "How do you check system uptime and load average?" — asked verbatim in GeeksforGeeks' Linux Interview Questions (70+ questions)
uptime — one line: time since boot, logged-in users, and the 1/5/15-minute load averages; w adds who's doing what; /proc/loadavg is the raw source. Load average is the average count of processes running-or-waiting-to-run (including uninterruptible I/O wait on Linux) over those three windows; the three timescales reveal trend (rising vs falling).
The details that separate candidates: the load-÷-cores interpretation (nproc is mandatory context — the number is unitless without it); reading the three numbers as a trend not three separate readings; and knowing Linux load includes D-state I/O wait (so high load can mean disk-stuck, not CPU-busy — the subtlety that trips people who learned load on other Unixes where it's CPU-only).
Part B — Logs: the story of what happened
B1. /var/log and the journal
🧠 Where top shows now, logs show what happened — the history that answers "when did it start?" and "what was the first error?" Two systems coexist on modern Ubuntu: /var/log/* — plain text files (Module 3's tools work directly), the classics being syslog/messages (general system), auth.log (logins and sudo — Module 4/15), kern.log (kernel), and per-service files (/var/log/nginx/, /var/log/postgresql/); and the systemd journal — structured, queried with journalctl (Module 11). The rule of thumb: journalctl -u service for anything systemd manages, plain-text /var/log files for services and subsystems that write their own. Both are where an incident's story lives; top tells you the machine is sick, logs tell you why and since when.
top is the bridge's live instrument panel — speed, heading, engine temp right now. The logs are the ship's logbook — the timestamped record of everything that happened: when the engine warning first lit, what the previous watch noted, the sequence that led to the current state. You diagnose a problem by reading the logbook backwards from now to where it began.
Where the analogy stops working. A ship keeps one logbook. A Linux machine scatters its history across the journal and a dozen text files in /var/log, each service writing where it likes — so "read the logbook" means knowing which log a given problem writes to. And logs, unlike a bound logbook, rotate and expire (B3) — the entry you need may have aged out. Knowing which log holds what, and how far back it goes, is half the skill.
🧪 Exercise B1.1 — Survey the logs (read-only; some files need sudo)
ls /var/log # the landscape of log files
sudo tail -20 /var/log/syslog # recent general system messages (or /var/log/messages on RHEL)
journalctl -p err -b | tail -20 # this boot's ERROR-priority journal entries, all services✅ Expected result — click to reveal
alternatives.log apt auth.log bootstrap.log dpkg.log journal kern.log
syslog ...then recent syslog lines (timestamp, host, process, message), then this-boot error entries from the journal.
What to read out of it:
- /var/log is a landscape — auth.log (who logged in, sudo use — the security log, Module 15), kern.log (kernel/hardware), syslog (the catch-all), dpkg.log/apt (package history, Module 9), and a journal/ directory (the binary systemd journal). Knowing which file answers which question is the navigation skill.
- journalctl -p err -b is a fantastic incident opener: every error-priority message from every service since boot, in one query — the fastest way to see "what's complaining?" across the whole machine (Module 11's priority filtering, aimed at triage).
- On a real incident you'd pair a symptom with its log: web 500s → /var/log/nginx/error.log; a service won't start → journalctl -u thatservice; failed logins → auth.log; hardware/disk errors → kern.log/dmesg (B2).
You don't grep /var/log on each box. Logs ship off the host: journald or rsyslog forwards to a central store (Loki, Elasticsearch/ELK, or a cloud logging service), and you query all 500 hosts at once with a label selector like {job="nginx"} |= "500". Local /var/log becomes the fallback you read only when shipping itself breaks. That changes the failure mode: a log-pipeline outage blinds you across the whole fleet simultaneously, so the logging path itself needs monitoring — and you keep enough local retention (via logrotate, B3) to reconstruct events after the pipeline recovers.
B2. dmesg and the kernel's own log
🧠 The kernel keeps its own ring buffer of messages — hardware events, driver messages, and crucially the OOM killer's obituaries (Part D) and disk I/O errors. dmesg reads it (dmesg -T for human timestamps; journalctl -k is the journal's view of the same). This is where the lowest-level truth lives: when a disk is failing, when the kernel killed a process for memory, when a network card reset — things no application log would show because they happen below the applications. When a machine misbehaves in ways that don't show up in service logs, dmesg is the next place to look.
Application logs are what the tenants report ("the lift is slow"). dmesg is the building's own structural monitoring — foundation stress, electrical faults, the fire-suppression system triggering. It records events at the level of the building itself, which no tenant would ever see or think to report, and which often explain the tenant complaints.
Where the analogy stops working. Structural sensors log permanently. dmesg's ring buffer is fixed-size and overwrites oldest-first — a flood of messages can push the one you need out of the buffer before you look (which is why persistent kernel logging via the journal, journalctl -k, matters). And its messages are terse and low-level — reading "Out of memory: Killed process 4823 (mysqld)" or a disk's I/O error, dev sda requires knowing the vocabulary, which is exactly what makes it a senior-level tool.
🧪 Exercise B2.1 — Read the kernel's log (read-only; may need sudo)
sudo dmesg -T | tail -15 # recent kernel messages with human timestamps
sudo dmesg -T | grep -iE 'error|fail|oom|killed' | tail # the words that matter in an incident
journalctl -k -b | tail -10 # the journal's view of this boot's kernel log✅ Expected result — click to reveal
[Wed Sep 2 22:33:01 2026] EXT4-fs (vda): mounted filesystem ... r/w without journal.
...(a healthy boot shows filesystem mounts and hardware init; an incident shows the tell)
What to read out of it:
- A healthy dmesg is mostly boot-time hardware and filesystem init — the machine describing itself coming up. Boring is good here.
- The grep -iE 'error|fail|oom|killed' filter is the incident version: it surfaces the lines that matter — a disk I/O error (failing storage, Module 12), an Out of memory: Killed process (the OOM killer, Part D), a link is down (network hardware, Module 13). One dmesg | grep can explain a mystery that no application log touches.
- journalctl -k is the same kernel log, but persisted by the journal (survives the ring buffer overwriting), so on a machine with persistent journaling you can read kernel messages from previous boots (journalctl -k -b -1) — essential when a machine crashed and rebooted and you need to know why it went down.
B3. Log rotation — why logs don't fill the disk
🧠 Logs grow forever unless something stops them — and Module 12 taught what an unbounded /var/log does to a disk. logrotate is that something: a scheduled job (Module 10/11) that periodically renames the active log (syslog → syslog.1), compresses old ones (syslog.2.gz), deletes ones past a retention limit, and signals the service to start writing fresh. Its config lives in /etc/logrotate.conf and /etc/logrotate.d/* (one file per service). This is why a healthy server's logs don't grow without bound — and when rotation breaks (a misconfigured rule, a service that won't reopen its file), that's exactly the "disk filled up with logs" incident from Module 12. The journal self-manages similarly (size caps, journalctl --vacuum-size).
logrotate is the records-management policy: current files stay on the desk (the active log), last month's go to the filing cabinet compressed (rotated + gzipped), anything older than the retention period is shredded (deleted). Without the policy, paper buries the office; with it, storage stays bounded and you keep exactly as much history as you decided to.
Where the analogy stops working. An office worker notices the policy failing — paper piling up is visible. A broken logrotate rule fails silently: the active log just keeps growing, invisibly, until df hits 100% one night (Module 12's incident). And rotation has a subtle failure mode the paper analogy lacks — if the service keeps writing to the old file handle after rotation renamed it (Module 3's deleted-but-open file!), the "rotated" log keeps growing under a new name that du can't easily find. logrotate's copytruncate and signal mechanisms exist precisely to prevent that, tying this straight back to Module 12's lsof | grep deleted.
🧪 Exercise B3.1 — Inspect rotation config and rotated logs (read-only, safe anywhere)
ls /etc/logrotate.d/ # per-service rotation rules
cat /etc/logrotate.d/rsyslog 2>/dev/null | head -20 # how the system logs are rotated
ls -lh /var/log/ | grep -E '\.(1|gz)$' | head # rotated/compressed old logs, if any✅ Expected result — click to reveal
alternatives apt dpkg rsyslog ...then a rotation rule like:
/var/log/syslog
/var/log/mail.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/cron.log
{
rotate 4
weekly
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}What to read out of it:
- Each file in /etc/logrotate.d/ is one service's policy. The rule decodes plainly: weekly (rotate once a week), rotate 4 (keep 4 old copies — ~a month), compress+delaycompress (gzip old copies, but leave the most recent one uncompressed so it stays greppable), notifempty (skip if empty), sharedscripts (run the postrotate script once for the whole group, not once per file). One stanza covers several files at once (syslog, kern.log, auth.log, …), so they all self-limit to ~a month of compressed history — bounded, by design.
- The postrotate … endscript runs after rotation — here signalling rsyslog to reopen its log file, which is the mechanism that prevents the deleted-but-open-file growth trap (B2's analogy, Module 12's incident). When rotation "works but the disk still fills," a missing or broken postrotate signal is a prime suspect.
- Seeing syslog.1, syslog.2.gz etc. in /var/log is rotation working normally. Seeing a single enormous syslog with no rotated copies is rotation failing — and a Module 12 disk-full incident waiting to happen.
Part B — Interview questions
🎯 "How do you view system and application logs in Linux?" — asked verbatim in GeeksforGeeks' Linux Interview Questions (70+ questions)
Two systems: /var/log/* plain-text files — syslog/messages (general), auth.log (logins/sudo), kern.log (kernel), and per-app files (/var/log/nginx/) — read with tail/less/grep (Modules 3, 6); and the systemd journal via journalctl — journalctl -u service for a service, -f to follow, --since for a window, -p err by priority, -k for kernel (Module 11). dmesg reads the kernel ring buffer directly. Match the symptom to the log: service crash → journalctl -u, web 500 → the app's error log, failed login → auth.log, hardware/OOM → dmesg/kern.log.
The details that separate candidates: knowing which log answers which question (not just "logs are in /var/log"); journal vs text-file coexistence and when each applies; journalctl -p err -b as a whole-machine triage query; and that logs rotate (so history is bounded, and the entry you need may have aged out — check rotated .gz files or central logging).
🎯 Corpus note — log rotation and dmesg
The banks ask where logs are and how to view them but rarely test rotation or dmesg/kernel logs directly — yet both are senior-level incident staples. Rotation is the answer to "why don't logs fill the disk, and what breaks when they do?" (ties to Module 12's disk-full and deleted-open-file incidents); dmesg is where OOM kills and disk failures announce themselves when application logs are silent. Be ready to explain a logrotate rule from memory (rotate N, daily/weekly, compress, postrotate-to-reopen) and to reach for dmesg | grep -iE 'oom|error|killed' when a machine misbehaves below the application layer. That's where real troubleshooting lives.
Part C — Targeted performance tools
C1. vmstat — the whole system on one line
🧠 top is a screen; vmstat is a stream — one line per interval summarizing the whole system's activity, so you watch trends unfold. vmstat 1 prints every second: process queue (r runnable, b blocked), memory (free, cache), swap (si/so — swap in/out), I/O (bi/bo — blocks in/out), and CPU (us/sy/id/wa — the same breakdown as top). The columns that earn their keep in an incident: r (if consistently > cores, CPU is the bottleneck — it's the load average's live source), si/so (any sustained nonzero = active swapping, memory pressure — Part D), and wa (high = disk-bound). One vmstat 1 running in a terminal during an incident shows you the system breathing.
top's snapshot is taking a pulse once. vmstat 1 is the continuous EKG trace — the pattern over time that reveals what a single reading can't: is the swap activity a one-off blip or a sustained rhythm? Is wa spiking periodically (a cron job hammering disk every minute) or constant? The trace shows the shape of the problem.
Where the analogy stops working. An EKG measures one patient. vmstat aggregates the whole machine into one line — which is its power (system-wide at a glance) and its limit (it can't tell you which process is causing the swap or the iowait; for that you go back to top/pidstat/iotop). vmstat says "the system is swapping hard"; it takes another tool to say "because mysqld is the memory hog." The EKG shows the arrhythmia; finding the cause needs more.
🧪 Exercise C1.1 — Watch the system breathe (read-only, safe anywhere)
vmstat 1 3 # 3 one-second samples of the whole system
# Watch the r, si/so, and wa columns especially✅ Expected result — click to reveal
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
r b swpd free buff cache si so bi bo in cs us sy id wa st gu
0 0 0 7066300 119780 608208 0 0 181 13 306 1 2 0 98 0 0 0
0 0 0 7068572 119780 608208 0 0 0 0 277 396 2 0 99 0 0 0
0 0 0 7068572 119780 608208 0 0 0 0 248 344 1 0 99 0 0 0(a healthy idle machine; every column near zero except idle CPU near 100)
What to read out of it:
- First line is a since-boot average — ignore it; read from the second line on (a classic vmstat gotcha). The real-time samples are line 2 onward.
- r 0 (nothing queued for CPU), si 0 so 0 (no swapping — healthy memory), wa 0 (no disk wait), id 98-99 (idle). Four columns, and you've confirmed the machine is healthy on CPU, memory, and disk simultaneously. On a sick machine, one of these columns is loud: r 8 on a 2-core box (CPU-bound), or so steadily climbing (swapping — memory crisis), or wa 60 (disk-bound).
- vmstat 1 left running in a corner terminal during any incident is a pro habit — it's the continuous vital-signs monitor that catches intermittent problems a one-shot top misses.
C2. iostat and iotop — the disk dimension
🧠 When top/vmstat show high wa (iowait) — CPU idle but the system slow because it's waiting on disk — you need the disk-specific view. iostat -x 1 (from the sysstat package) shows per-device I/O: %util (how busy each disk is — near 100% means the disk is maxed out), r/s/w/s (reads/writes per second), and r_await/w_await (average read/write wait time in ms — rising await means the disk is struggling to keep up). Modern sysstat splits the old single await column into separate read (r_await) and write (w_await) latencies. iotop (like top, but for disk I/O per process) names which process is hammering the disk. This is the toolset for "the disk is the bottleneck" — the diagnosis that wa in top points you toward but can't complete.
If the CPU is the workers and memory is the workbench, the disk is the loading dock — and iostat is the dock's throughput monitor: how many trucks per hour (r/s/w/s), how long each waits to be unloaded (await), and whether the dock is at capacity (%util near 100%). When work backs up not because workers are idle but because the dock can't move goods fast enough, this is the instrument that shows it.
Where the analogy stops working. A loading dock's capacity is fixed and visible. Disk %util at 100% doesn't always mean "the disk is the problem" — on modern SSDs and cloud storage, %util can read 100% while the device has plenty of headroom (it's a single-queue metric that misleads on parallel devices). So %util is a strong hint, not proof, and await (actual latency) is often the more honest signal. The instrument reads high; interpreting whether that's the bottleneck takes the judgment the analogy's simple gauge lacks.
🧪 Exercise C2.1 — Look at disk I/O (read-only; needs the sysstat package)
sudo apt install -y sysstat >/dev/null 2>&1 # provides iostat/pidstat/sar (Module 9)
iostat # a since-boot summary per device
iostat -x 1 2 # extended per-device stats, 2 one-second samples✅ Expected result — click to reveal
avg-cpu: %user %nice %system %iowait %steal %idle
1.75 0.00 0.49 0.16 0.00 97.59
Device tps kB_read/s kB_wrtn/s ...
vda 7.46 134.16 13.96 ...then extended (-x) per-device rows with %util, r_await/w_await, etc.
What to read out of it:
- The avg-cpu line repeats top's CPU breakdown; the Device rows are the point — per-disk activity. On a healthy idle box these are near zero; on a disk-bound incident, the busy device shows high %util and rising r_await/w_await, and that device is your bottleneck.
- iostat -x is the detailed view: r_await/w_await (read/write I/O latency; older sysstat showed one combined await) climbing into tens or hundreds of ms is the honest "this disk is struggling" signal — more reliable than %util on modern storage (the analogy's caveat). Pair it with iotop (sudo iotop) to name the process doing the I/O.
- This is Module 12's storage world seen through the performance lens: Module 12 asked "is the disk full?"; Part C asks "is the disk fast enough?" — different question, different tools, same devices.
C3. pidstat and per-process attribution
🧠 The system-wide tools (vmstat, iostat) tell you what kind of problem; pidstat (also from sysstat) attributes it to processes over time — pidstat 1 streams per-process CPU each second, pidstat -d 1 per-process disk I/O, pidstat -r 1 per-process memory. Where top snapshots, pidstat streams per process, so you catch the process that spikes intermittently — the one top misses between refreshes. It closes the loop: top/vmstat say "CPU-bound" or "disk-bound," pidstat says "this PID is the cause," and Module 8's tools (kill, renice) act on it.
🧪 Exercise C3.1 — Attribute activity to a process (read-only; needs sysstat)
pidstat 1 2 # per-process CPU, 2 one-second samples
# In another terminal you could run a busy loop to see it appear:
# timeout 5 bash -c 'while :; do :; done' &✅ Expected result — click to reveal
Linux 6.x (myhost) 09/02/26 _x86_64_ (2 CPU)
23:24:54 UID PID %usr %system %guest %wait %CPU CPU Command
23:24:54 0 476 4.00 0.00 0.00 0.00 4.00 1 claude(only processes with activity in the interval appear)
What to read out of it:
- Each line is one process's CPU use in that one-second window — so a process that spikes for a second every minute appears in that second, where a top refresh might miss it entirely. That's pidstat's edge: continuous per-process attribution.
- %usr vs %system per process mirrors top's us/sy but attributed — a process burning %system is making heavy syscalls (lots of I/O or context switching), a different problem from one burning %usr (pure computation).
- The full attribution chain: top/vmstat categorize ("it's CPU"), pidstat names the process, ps//proc (Module 8) detail it, kill/renice (Module 8) act. Each module's tools slot into one investigation.
Part C — Interview questions
🎯 "How do you troubleshoot high CPU usage?" — asked verbatim in GeeksforGeeks' Linux Interview Questions (70+ questions)
Confirm it's actually CPU: top's %Cpu(s) line — high us/sy (not wa, which would be disk) and load above the core count. Identify the culprit: top sorted by CPU (P), or pidstat 1 to catch intermittent spikes per process. Understand it: is the process legitimately busy (scale it, optimize it) or stuck/looping (a bug)? Then act (Module 8): renice to deprioritize a batch job that's starving latency-sensitive work, or kill/restart a runaway — after capturing evidence. Distinguish us (app code) from sy (kernel/syscalls, often I/O) from st (steal — hypervisor took your CPU, a noisy-neighbour problem you can't fix in-guest).
The details that separate candidates: checking wa first to rule out "it's actually disk, not CPU"; pidstat for intermittent spikes top misses; us vs sy vs st interpretation; and the mature move — capture evidence (a top/pidstat snapshot, /proc/PID) before killing, because a killed process can't be diagnosed, and understanding why beats reflexively killing.
🎯 Corpus note — vmstat, iostat, and the layered tools
The banks name top constantly but rarely vmstat/iostat/pidstat — yet these are exactly what separates a surface answer ("run top") from a real one in a performance interview. The high-value framing: top for the first triage, vmstat 1 for the whole-system trend over time, iostat -x/iotop when wa points at disk, pidstat for per-process attribution of intermittent spikes. Being able to say "top showed high iowait, so I ran iostat -x and saw vda at 100% util with 200ms await, then iotop named the backup job" — a tool chain following the evidence — is what a senior performance answer sounds like.
Part D — Memory pressure, the OOM killer, and the method
D1. Memory pressure and the OOM killer
🧠 Module 12 taught free and the available-vs-free distinction; here's what happens when available genuinely runs out. As memory fills, the kernel first reclaims cache and pushes idle pages to swap (Module 12) — the system slows (swapping is slow). If memory is still exhausted with nowhere left to go, the kernel invokes the OOM killer (Out Of Memory killer): it picks a process — scored by memory use and an adjustable priority (oom_score) — and kills it (SIGKILL, Module 8) to save the machine from freezing entirely. This is a last-resort survival mechanism, and its victim isn't always the true cause — it kills the biggest process, which might be your database rather than the small leaking script that ate the memory. The OOM killer's decisions are logged in dmesg/journalctl -k — Out of memory: Killed process 4823 (mysqld) — which is the explanation for "why did my service just vanish?" when no one killed it.
When a lifeboat is sinking from overload, someone makes the brutal call to put a passenger over the side to save everyone else — the OOM killer's job. It's not malice; it's the alternative to everyone drowning (the whole machine freezing). And it throws out the heaviest passenger to lighten the load fastest — which is why it often sacrifices your biggest, most important service rather than the small stowaway who caused the overload.
Where the analogy stops working. A lifeboat's crew sees who's overloading it. The OOM killer acts on a crude score (size + priority), not on who's at fault — it can't tell a legitimately-large database from a runaway leak, so its choice frequently punishes the innocent. And the "heaviest passenger" heuristic means the fix is never "stop the OOM killer" — it's provide enough memory or find the leak so the boat isn't overloaded in the first place. The killer is a symptom-manager of last resort; the disease is upstream.
🧪 Exercise D1.1 — Check memory pressure and hunt for OOM events (read-only; dmesg may need sudo)
free -h # available memory (the column that matters, Module 12)
cat /proc/meminfo | grep -E 'MemAvailable|SwapTotal|Committed_AS' # the raw pressure indicators
sudo dmesg -T | grep -i 'out of memory\|oom\|killed process' # any OOM kills? (usually none — good)✅ Expected result — click to reveal
total used free shared buff/cache available
Mem: 7.8Gi 695Mi 6.7Gi 12Mi 711Mi 7.3Gi
Swap: 0B 0B 0B
MemAvailable: 7503856 kB
SwapTotal: 0 kB
Committed_AS: 339816 kBthen (on a healthy machine) no output from the OOM grep — silence is good here.
What to read out of it:
- available 7.3Gi of 7.8Gi — abundant headroom, no pressure (Module 12: read available, not free). Committed_AS (339MB) is how much memory programs have asked for — far below total, so no overcommit stress. These are the pressure gauges; all green here.
- The OOM grep returning nothing is the healthy result — no process has been memory-killed this boot. On a machine that's been struggling, this grep is where you'd find Out of memory: Killed process NNNN (name) with a timestamp — the smoking gun for a vanished or restart-looping service.
- Swap: 0B here (no swap configured) means a memory spike goes straight to the OOM killer with no cushion — common on containers and some cloud images, and worth knowing: no swap = OOM kills happen sooner and more abruptly.
One OOM kill is a footnote. The same OOM signature across 50 hosts inside ten minutes is an incident — a bad deploy or a memory leak rolling through the fleet. You don't find that by reading dmesg per host; you alert on the node_vmstat_oom_kill counter from node_exporter and correlate the spike against deploy timestamps. The swap-less container case above gets worse at scale: with no cushion, a fleet-wide memory regression turns into a wave of simultaneous kills and restart storms, and the OOM counter is often your first and only early signal before the service-level alerts fire.
D2. The troubleshooting method — USE and the incident ladder
🧠 Tools are nothing without a method — a repeatable order so you don't flail. A battle-tested one is USE: for each resource (CPU, memory, disk, network), check Utilization (how busy?), Saturation (is work queuing/waiting?), and Errors (any failures?). Combined with this track's layered tools, it becomes a concrete ladder for "the server is slow":
Diagram source
flowchart TD
A["Server is slow"] --> B["top / uptime:<br>load vs cores, %Cpu breakdown"]
B --> C{"What's the bottleneck?"}
C -->|"high us/sy CPU"| D["pidstat / top -P:<br>which process? (Part C, M8)"]
C -->|"high wa iowait"| E["iostat -x / iotop:<br>which disk/process? (Part C, M12)"]
C -->|"low available mem<br>+ swap churn"| F["free / dmesg oom:<br>leak? OOM kills? (D1)"]
C -->|"CPU idle, still slow"| G["ss / curl / dig:<br>network? DNS? (M13)"]
C -->|"nothing obvious"| H["logs: journalctl -u,<br>/var/log, dmesg (Part B)"]
D --> I["Confirm, capture evidence,<br>then act (M8/M11)"]
E --> I
F --> I
G --> I
H --> IThe discipline that matters as much as the ladder: observe before you act (a killed process can't be diagnosed — capture top/pidstat/logs first), change one thing at a time (so you know what fixed it), and form a hypothesis from evidence, not a guess (the %Cpu line tells you CPU vs disk vs idle — follow it, don't assume).
A good mechanic with a "car won't start" doesn't just poke at the part they have a hunch about — they run the checklist: fuel? spark? compression? air? Each check rules out a whole system, converging on the cause fast. The USE method is that checklist for a slow server: it methodically rules out CPU, memory, disk, network, so you converge instead of guessing.
Where the analogy stops working. A car is one closed system the mechanic can see entirely. A slow production service spans many machines — load balancers, app servers, databases, caches, disks, networks — and the cause hides in the interactions (the app waiting on the DB waiting on the disk). The checklist has to be run across systems, following the wait from one to the next, which is why distributed-systems troubleshooting is genuinely harder than fixing a car and why "observe, follow the evidence, don't assume" is the whole game.
🧪 Exercise D2.1 — Run the ladder on a healthy machine (read-only, safe anywhere — building the muscle memory)
# Walk the USE method top-to-bottom, narrating what each rules out:
uptime; nproc # 1. load vs cores — CPU saturated?
top -b -n1 | head -5 # 2. %Cpu breakdown — us/sy/wa/id? and memory line
vmstat 1 3 | tail -3 # 3. trend: r queue, si/so swap, wa iowait
df -h | grep -vE 'tmpfs|udev' # 4. any filesystem full? (Module 12)
sudo dmesg -T | grep -iE 'error|oom|killed' | tail # 5. any kernel-level errors?✅ Expected result — click to reveal
A sequence of healthy readings: low load, high id, no swap, no full filesystems, no kernel errors.
What to read out of it:
- Each command rules out a resource: uptime/top (CPU fine), vmstat (no queue, no swap, no iowait), df (disk not full), dmesg (no errors). On a healthy box you run the whole ladder and everything's green — which is exactly the point of practicing it now: you learn the normal readings, so the abnormal one jumps out during a real incident.
- On a sick machine, one of these steps is loud, and it tells you where to go deeper: full disk → Module 12's du drill; high iowait → Part C's iostat; OOM in dmesg → D1; CPU-bound → Part C's pidstat. The ladder routes you; the deep-dive tools solve it.
- Running this ladder from memory, narrating what each rules out, is the answer to "a production server is slow, how do you troubleshoot it?" — the interview question this whole module builds toward.
Part D — Interview questions
🎯 "A production Linux server is slow. How would you troubleshoot it?" — asked verbatim in GeeksforGeeks' Linux Interview Questions (70+ questions)
Run a method, not a guess. Triage with top/uptime: load vs cores, and the %Cpu(s) line to categorize — user-bound, kernel, iowait, or idle-but-slow. Follow the category: high us → pidstat/top for the process (Part C, Module 8); high wa → iostat -x/iotop for the disk (Part C, Module 12); low available memory + swap churn → free/dmesg for leaks/OOM (Part D); CPU idle but slow → network (ss/curl/dig, Module 13) or downstream waits; nothing obvious → logs (journalctl, /var/log, dmesg, Part B). Discipline: observe before acting (capture evidence — a killed process can't be diagnosed), change one thing at a time, and follow the evidence off the symptomatic component when it points elsewhere.
The details that separate candidates: leading with the method (USE — utilization/saturation/errors per resource) and a triage-then-drill-down ladder rather than a random command list; the "symptom ≠ cause" insight (slow web can be a failing disk three layers down); capturing evidence before killing; and confirming when it started against historical monitoring, since top only shows now. This one question is a proxy for whether you've actually operated production — the structured, evidence-driven answer is unmistakable.
🎯 "What is the OOM Killer?" / "How do you troubleshoot high memory usage?" — both asked verbatim in GeeksforGeeks' Linux Interview Questions (70+ questions)
The OOM killer is the kernel's last-resort survival mechanism: when memory is exhausted with nothing left to reclaim or swap, it kills a process (scored by size and oom_score) to save the machine from freezing — logged in dmesg/journalctl -k as Out of memory: Killed process …. Troubleshooting high memory: free -h (read available, not free — Module 12), top/pidstat -r sorted by memory for the consumer, distinguish a genuine need (scale up) from a leak (memory growing without bound — restart mitigates, fixing the code cures), and check dmesg for OOM kills that explain vanished/restarting services.
The details that separate candidates: the OOM killer picks the biggest process, often the victim not the culprit — so you investigate what filled memory, not just what died; a service that "restarts itself" is frequently OOM-killed-then-auto-restarted (systemd Restart=, Module 11) — check dmesg | grep oom; available vs free; and that no-swap machines (containers) OOM sooner and harder. Knowing the killer targets by size and logs to the kernel buffer is the tell of someone who's debugged a real OOM incident.
Part E — Toolkit
E1. Production practice — symptoms and fixes
| Symptom | What is really happening | What to run | The fix / next step |
|---|---|---|---|
| "Load is 8, is that bad?!" | Load is meaningless without the core count | uptime • nproc — compute load ÷ cores | Only alarming if load ≫ cores; below cores is fine |
| Slow, but CPU shows nearly idle (%id high) | CPU isn't the bottleneck — look at iowait, memory, network, or downstream waits | top %Cpu line (check wa); vmstat 1 | High wa → disk (iostat); else memory/network/logs |
| Slow with high %wa (iowait) | Disk is the bottleneck, CPU waiting on it | iostat -x 1 (%util, r_await/w_await); iotop | Find the I/O-heavy process; check for failing disk (dmesg) |
| A service "restarts itself" every few hours | Getting OOM-killed, then auto-restarted by systemd | sudo dmesg | grep -i oom; free -h trend | Find the memory leak / add RAM; the OOM victim ≠ the culprit |
| Disk filled overnight with logs | Log rotation broke, or a service holds the rotated (deleted) file open | ls /etc/logrotate.d/; lsof | grep deleted (M12) | Fix the rotation rule / postrotate signal; restart the holding service |
| The metric you need is from before you looked | top/vmstat show only now; the spike is gone | Historical monitoring; sar (sysstat history); rotated logs | This is why fleets run monitoring — top is for the live moment only |
| Staring at the broken component, finding nothing | Symptom ≠ cause — the wait comes from elsewhere | Run the USE ladder across each resource/machine | Follow the evidence off the symptomatic box (the slow web → slow DB → bad disk chain) |
E2. Capstone — four tickets
🎫 Ticket 1 — "vm-app-4 load average is 30, everything's crawling — fix it"
Ticket text: "Monitoring alerts load average 30 on vm-app-4. The app is barely responding. Diagnose and stabilize."
Worked answer — resist the panic; run the ladder. Context first: uptime + nproc — is 30 on a 32-core box (busy but coping) or a 4-core box (7× overloaded)? Say the load-÷-cores number in the reply. Categorize: top %Cpu(s) — is it us (app compute), sy (kernel), or wa (iowait — the disk, not CPU)? A load of 30 with high wa and low CPU means the "load" is processes stuck waiting on disk (Linux load includes D-state), not CPU overload — a completely different fix. Attribute: if CPU-bound, pidstat 1 / top -P names the process(es) — one runaway, or many legitimate requests (a traffic spike → scale out)? if disk-bound, iostat -x + iotop. Stabilize (Module 8/11): renice a batch job starving the app, kill a confirmed runaway (after capturing evidence), or restart a wedged service — the least-invasive action that relieves the pressure. Then root-cause, don't just clear it. The reply names the number-in-context, the category, the culprit, and the action — a method, not a guess.
🎫 Ticket 2 — "Our API service keeps disappearing and coming back — no one is restarting it"
Ticket text: "api-svc on vm-api-2 goes down and comes back up every 2–3 hours. systemctl status shows it running now. Nobody's touching it. What's happening?"
Worked answer — "restarts itself with no one touching it" is the OOM-kill signature (D1). Confirm: sudo dmesg -T | grep -i 'oom\|killed process' or journalctl -k — expect Out of memory: Killed process NNNN (api-svc) at each disappearance timestamp, matching the 2–3 hour cadence. The mechanism: the service leaks (or legitimately grows) memory → fills the box → OOM killer kills it (it's the biggest process) → systemd's Restart=on-failure (Module 11) brings it back → memory refills → repeat. Two-part fix: immediate — confirm whether it's a leak (memory climbing steadily between restarts — pidstat -r or graph RES over time) vs genuine need; if a leak, the restart is masking it, so escalate to the app team with the evidence. Mitigate — set a systemd memory limit (MemoryMax=) so it's killed cleanly and predictably rather than triggering the OOM killer that might take out a different service, and/or add RAM/swap for headroom. The key teaching point in the reply: the OOM killer's victim (api-svc) may not be the culprit — check what filled memory, and "keeps restarting" almost always means "keeps getting OOM-killed."
🎫 Ticket 3 — "The website is slow but the web servers look completely idle"
Ticket text: "Users report acme-portal is slow. We checked the web servers — top shows them 95% idle, load near zero. But requests take 10+ seconds. Where do we even look?"
Worked answer — this is the "symptom ≠ cause" lesson made flesh (D2). Idle web servers with slow requests means they're waiting on something downstream — the slowness is elsewhere, and the method is to follow the wait. On a web box, the requests are blocked in a downstream call: the database? — check the DB server's own vitals (top, iostat — is it CPU-bound or, classically, disk-bound with high wa?); a slow query or a failing disk under the DB shows up as fast web CPU-idle but slow web responses. Or the network/DNS? — curl -w the exact downstream calls from the web box (Module 13) to time each layer; a slow DNS lookup or a saturated link between tiers adds seconds while every server looks idle. Or a downstream service/cache? — same logic, one hop further. The diagnostic instinct to state: an idle-but-slow component is a pointer, not a dead end — it says "the problem is something I'm waiting on," and you walk the dependency chain (web → DB → disk, or web → API → DNS) until you find the resource that's actually saturated. Wasting the hour staring at the idle web servers is the mistake the method exists to prevent.
🎫 Ticket 4 — "Write us a one-page 'server is slow' runbook the on-call can follow at 3am"
Ticket text: "Half our incidents are 'a server is slow' and everyone debugs it differently. Give us a repeatable runbook a tired on-call engineer can follow step by step."
Worked answer — codify the USE ladder into ordered, copy-pasteable steps. 0. Capture first: note the time, grab a top -b -n1, vmstat 1 5, and dmesg -T | tail before changing anything (a killed process can't be diagnosed). 1. Context: uptime; nproc — load ÷ cores; is it even overloaded? 2. Categorize (top %Cpu): high us/sy → CPU (step 3a); high wa → disk (3b); low available mem + swap → memory (3c); high id but slow → downstream/network (3d). 3a. CPU: pidstat 1 → the process → Module 8 (renice/kill after evidence). 3b. Disk: iostat -x 1 (%util, r_await/w_await) + iotop → the I/O culprit; check dmesg for disk errors. 3c. Memory: free -h (available!), dmesg | grep oom, pidstat -r → leak vs need. 3d. Network/downstream: ss, curl -w the downstream calls, check the next tier's vitals (symptom ≠ cause). 4. Logs throughout: journalctl -u <service> --since, /var/log, dmesg for the story and when it started. 5. Act minimally, one change at a time, verify, document. The framing: a runbook turns a stressful judgment call into a checklist a tired human can execute correctly — the same reason pilots use checklists (Module 10's idempotence analogy), applied to incident response. Everything in it is a command from this track; the value is the order and the discipline.
E3. Documentation reference
| Topic | Authoritative source | Verified link |
|---|---|---|
| Load and uptime | uptime(1) | uptime(1) |
| Live triage | top(1) | top(1) |
| System and disk stats | vmstat(8), iostat(1) | vmstat(8) · iostat(1) |
| Logs and rotation | journalctl(1), logrotate(8) | journalctl(1) · logrotate(8) |
| Memory | free(1) | free(1) |
| The kernel theory beneath this module | Companion track (OS 07–10, 14) | Untitled · Untitled · Untitled |
E4. Self-assessment
Answer out loud, without notes. The section number tells you where to re-read.
- What is load average, why are there three numbers, and why is it meaningless without nproc? (A1)
- In top's %Cpu(s) line, what do us, sy, wa, id, and st each mean, and where does a high value in each send you? (A2)
- What does a high load average with low CPU usage tell you? (A1, D2)
- Which log answers which question: service crash, web 500, failed login, OOM kill, disk error? (B1–B2)
- What is dmesg, and name two incident-critical things that appear there and nowhere else. (B2)
- Why don't logs fill the disk on a healthy server, and what breaks in the Module 12 disk-full incident? (B3)
- top vs vmstat vs pidstat — what does each add, and how do they chain in an investigation? (A2, C1, C3)
- High iowait — what tool next, and which two fields tell you the disk is struggling? (C2)
- What is the OOM killer, why does it often kill the wrong process, and how does it explain a "self-restarting" service? (D1)
- State the USE method and walk the "server is slow" ladder from triage to action. (D2)
- Why is "observe before you act" a rule, and what does it protect? (D2)
- Explain "symptom ≠ cause" with the slow-web / failing-disk example, and how the method finds the real cause. (D2)
E5. Sources
GeeksforGeeks — Linux Interview Questions (70+) (the richest source for this topic, with the slow-server, uptime/load, OOM, high-CPU, high-memory, and log-viewing questions all quoted above).
Corpus honesty note: troubleshooting and performance are well covered in the GeeksforGeeks bank — "a server is slow, what do you do?", OOM killer, uptime/load, high CPU, high memory, viewing logs are all dedicated, recent questions. What no bank captures is the layered tooling (vmstat/iostat/pidstat as a chain) and the method (USE, symptom≠cause, observe-before-act) — which is exactly what separates a real senior answer from "run top." This module is the capstone of the track precisely because "here's a broken server" is the interview, and the answer is a method built from every earlier module's tools.
All documentation links were fetched and confirmed reachable on 2 September 2026. uptime, top, vmstat, iostat, pidstat, free, and the log/dmesg outputs were generated on a live Ubuntu 24.04 system (including a real CPU-hog demonstration showing top catching a process at 100%).