💽Module 10 — I/O, the Block Layer & iowait

device: vda
cold read moved 262472 KiB from the device
warm read moved 0 KiB from the device

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 72.65    0.053882          26      2050           read
 27.35    0.020289           9      2048           write
------ ----------- ----------- --------- --------- ----------------
100.00    0.074171          18      4098           total

What to read out of this.

The cold read moved 262472 KiB — the 256 MiB file, plus a little, because readahead always overshoots the end slightly. The warm read moved 0 KiB. Same command, same file, same amount of data delivered to cat, and the second time the block layer was not involved at all.

That is the single most important fact about measuring I/O: application activity and device activity are different quantities, and the page cache is what separates them. A service can be doing gigabytes of reads per second with a completely idle disk, and a service can be doing almost no reads while hammering the disk.

Now the syscall counts. cat made 2050 reads and 2048 writes to move 256 MiB. That is 128 KiB per call — the buffer size GNU coreutils picks for this. Divide the work differently and the numbers change enormously: a program reading a byte at a time would make 268 million calls for the same data, at roughly a microsecond of kernel-crossing overhead each, and would look like a CPU problem rather than an I/O one.

Sectors are always 512 bytes in /proc/diskstats, regardless of the device's real block size, which is why the script divides by two to get kilobytes. Getting this wrong by a factor of eight is a common error when people write their own collectors.

If strace is not installed, sudo apt-get install -y strace. If it refuses to attach, check kernel.yama.ptrace_scope — the same restriction you met in Module 01.

On a typical cloud VM with a virtio disk:

NAME   ROTA TYPE  SIZE MOUNTPOINTS
vda       0 disk   40G
└─vda1    0 part   40G /

device: vda
rotational             0
nr_requests            256
read_ahead_kb          128
max_sectors_kb         512
logical_block_size     512
physical_block_size    512
nomerges               0
scheduler              none [mq-deadline] kyber bfq

1

Device   r/s   rkB/s  rrqm/s  %rrqm r_await rareq-sz   w/s  wkB/s  wrqm/s  %wrqm w_await wareq-sz  aqu-sz  %util
vda     0.00    0.00    0.00   0.00    0.00     0.00 41.00 892.00   68.00  62.39    1.12    21.76    0.05   3.20

What to read out of this.

ROTA 0 says the device is not rotational, and rotational 0 in sysfs agrees. That is what the kernel uses to decide whether to bother with seek-aware behaviour.

scheduler none [mq-deadline] kyber bfq — the brackets mark the active one, and the rest are available. mq-deadline here, and the line above tells you why: this virtio device exposes exactly one hardware queue, which is the condition for getting a scheduler at all. An NVMe device on the same machine would show [none].

nr_requests 256 is how many requests the block layer will queue per hardware queue before making the submitter wait. Raise it and you get more merging and more throughput at the cost of longer worst-case latency; lower it for a latency-sensitive workload. It is one of the few genuinely useful storage knobs and almost nobody touches it.

read_ahead_kb 128 means a sequential read triggers the kernel to fetch 128 KiB ahead speculatively. This is why sequential reads are so much faster than random ones even on an SSD, and why a database doing random reads sometimes wants it lowered — the readahead is wasted work.

Now the merge columns. wrqm/s 68.00 against w/s 41.00: more requests were merged away than were finally issued. %wrqm 62.39 says 62% of the writes that arrived got absorbed into another request. The average issued write, wareq-sz, ended up at 21.76 KiB — far larger than the 4 KiB pieces the filesystem produced. That is the block layer earning its place.

If %wrqm is near zero on a workload you expect to be sequential, either the writes really are scattered, or nomerges has been set to 1 or 2 — which some tuning guides recommend for NVMe and which is worth checking before blaming the application.

none [mq-deadline] kyber bfq

vda          none [mq-deadline] kyber bfq
loop0        [none] mq-deadline kyber bfq

tee: /sys/block/vda/queue/scheduler: Invalid argument
noop
exit status: 1

was: mq-deadline
async_depth  fifo_batch  front_merges  prio_aging_expire  read_expire  write_expire  writes_starved

[none] mq-deadline kyber bfq
(no iosched dir - none has nothing to tune)

none [mq-deadline] kyber bfq

What to read out of this.

The deliberate failure first. echo noop returns Invalid argument, because noop has not existed since Linux 5.0. Note what tee does: it prints the error and echoes noop to stdout, so a script that only checks the output would think it worked. The exit status is what tells you, and this is exactly how stale tuning scripts keep "working" for years while changing nothing.

The per-device loop is worth running on any real server. loop0 shows [none] while vda shows [mq-deadline], on the same machine and the same kernel. That is not a misconfiguration: the loop driver asks for none explicitly by setting a flag the block layer honours, BLK_MQ_F_NO_SCHED_BY_DEFAULT. It still lists the other schedulers and will accept them. So the default rule has three parts, not two: a driver can opt out entirely, otherwise one hardware queue gets mq-deadline and several get none.

Switching to none worked immediately, with no remount and no restart. Scheduler changes are live and take effect on the next request. That also means anything with root can change it at any time, including tuning packages and vendor agents, which is why it is worth asserting in configuration management.

Watch the iosched/ directory come and go. While mq-deadline is active it exists, listing read_expire, write_expire, fifo_batch, writes_starved and, on 6.x kernels, async_depth and prio_aging_expire. Those are the deadlines the scheduler enforces — read_expire defaults to 500 ms and write_expire to 5000 ms, which encodes the assumption that a process waiting on a read is blocked and a process that wrote is not. Switch to none and the directory disappears entirely, because there is nothing left to tune.

If your machine shows [none] by default, the device exposes several hardware queues — NVMe is the usual case — or its driver opted out. Both are correct, not a misconfiguration.

The call crosses into the kernel and reaches the VFS, which resolves the file descriptor to a filesystem and an inode. The data is copied into the page cache and those pages are marked dirty, and at that point write() returns — the program is finished and nothing has touched a device. Later, either background writeback or an explicit fsync hands the dirty pages to the filesystem, which maps file offsets to device blocks and may also journal metadata. Those become bios, the block layer merges adjacent ones into requests and queues them, the scheduler decides the order, and the driver hands them to the device.

The details that separate candidates: leading with the fact that most I/O calls involve no I/O at all — a cached read never reaches the block layer and a buffered write returns immediately. That single point explains why application-observed latency and device-observed utilisation routinely disagree, and it is the reason "the app says storage is slow, the storage team says the volume is idle" is such a common standoff with both sides correct. It is also worth naming where you would measure each layer, because measuring the wrong one is what makes these investigations drag.

none, which is already the default. NVMe exposes many hardware queues — often one per CPU — so the device is served in parallel and there is very little to gain from ordering requests. The scheduler's work becomes pure overhead, and on a fast device that overhead is a measurable share of the request's total latency. The block layer still merges adjacent requests under none, so you keep the benefit that actually matters.

mq-deadline is the default for single-queue devices — SATA and SAS — where sequencing still helps and where a deadline is needed so writes are not starved by a stream of reads.

The details that separate candidates: knowing the default is chosen by hardware queue count, not by rotational-versus-flash, which is why a SATA SSD gets a scheduler and an NVMe SSD does not. And knowing that cfq and noop were removed in Linux 5.0, so any runbook still setting them has been failing silently — the write to /sys returns Invalid argument and nothing changes. If asked when you would deviate: bfq when one workload must not monopolise a shared disk, and kyber when tail latency matters more than throughput.

The block layer combines bios that address adjacent blocks into a single larger request before handing it to the device. Sixteen 4 KiB writes to consecutive blocks become one 64 KiB write, and the device does a fraction of the work for the same data.

You see it in iostat -x as rrqm/s and wrqm/s — requests merged per second — and as %rrqm/%wrqm, the percentage absorbed. rareq-sz and wareq-sz show the average size of what actually reached the device, which on a sequential workload is far larger than what the filesystem produced.

The details that separate candidates: using the merge percentage as a diagnostic for the workload rather than the device. A workload you believe is sequential but that shows near-zero %wrqm is not sequential in practice — or nomerges has been set to 1 or 2, which some NVMe tuning guides recommend and which is worth checking before rewriting the application. It is also the cleanest way to explain why appending to one file is so much cheaper than scattering the same bytes across a thousand.

--- phase 1: reading, nothing else running ---
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 0  0      0 3512044  41216 293104    0    0   412    88  201  388  2  1 96  1  0  0
 0  1      0 3204112  41216 412088    0    0 118784     0 2104 3881  1  4 49 46  0  0
 0  1      0 3085200  41216 531000    0    0 118912     0 2211 4012  1  5 48 46  0  0
 0  1      0 2965400  41216 650800    0    0 119800     0 2188 3944  1  4 49 46  0  0
some avg10=44.18 avg60=15.02 avg300=4.11 total=88421037

--- phase 2: same read, with the CPUs busy ---
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 0  0      0 3510044  41216 294104    0    0   448    92  208  402  3  1 95  1  0  0
 3  1      0 3198844  41216 413992    0    0 117760     0 2088 3902 96  4  0  0  0  0
 4  1      0 3081084  41216 531752    0    0 117760     0 2194 4088 95  5  0  0  0  0
 3  1      0 2962300  41216 650536    0    0 118784     0 2176 3988 96  4  0  0  0  0
some avg10=43.87 avg60=21.44 avg300=7.02 total=94118204

Read the two blocks side by side. This is the whole lesson.

wa was 46 in phase 1 and 0 in phase 2. If iowait measured storage, you would conclude the disk got infinitely faster the moment you started some busy loops.

Now look at bi — kilobytes read in per second, and note that it matches the growth of the cache column exactly, which is a useful sanity check when you are reading someone else's paste. It is 118784, 118912, 119800 in phase 1 and 117760, 117760, 118784 in phase 2. Effectively identical. The disk is doing exactly the same work at exactly the same rate in both phases.

(The first data row in each block is vmstat's since-boot average, not a live sample — that is why its numbers look nothing like the three below it. Everyone misreads this line once.)

Look at the b column, processes blocked in uninterruptible sleep: 1 in both phases. One process is waiting on the disk, throughout, in both. That column told the truth when wa did not.

And /proc/pressure/io: avg10=44.18 in phase 1, avg10=43.87 in phase 2. Essentially unchanged, because PSI measures time actually lost to I/O rather than CPU ticks that happened to be idle. That is the metric to build an alert on.

What changed in phase 2 is only where the idle ticks went: id 49 and wa 46 became us 96. The CPUs stopped being idle, so there were no idle ticks left to classify as iowait.

The production translation. A busy application server with genuinely saturated storage often shows 0% iowait, because its CPUs are never idle. Anyone using iowait as their storage alert will never see it. Conversely, a quiet host doing a nightly backup shows 40% iowait and generates a page for a machine that is perfectly healthy.

Trimmed to the columns that matter, on a cloud VM with a network-backed SSD:

(trimmed to the columns that matter - the real output has 23)

--- idle ---
Device   r/s  rkB/s  rareq-sz  r_await   w/s   wkB/s  w_await  aqu-sz  %util
vda     0.00   0.00      0.00     0.00  1.00   12.00     0.80    0.00   0.20

--- sequential read ---
vda   231.00 236544.00   1024.00     4.31  0.00    0.00     0.00    1.00  99.60

--- random small reads ---
vda  2104.00   8416.00      4.00     0.46  0.00    0.00     0.00    0.97  98.80

--- synchronous writes ---
vda     0.00      0.00      0.00     0.00 604.00  2416.00     1.58    0.95  99.20

What to read out of this.

Start with %util, because it is the column everyone looks at. It reads 99.6, 98.8, 99.2 across all three loads. By that number, the device is equally and maximally busy in every case. Now look at what it was actually doing:

• Sequential: 231 requests/s moving 236 MB/s, average request 1024 KiB.
• Random small: 2104 requests/s moving 8 MB/s, average request 4 KiB.
• Sync writes: 604 requests/s moving 2.4 MB/s.

Same %util, and the throughput differs by a factor of nearly a hundred. %util said nothing useful in any of the three cases. All it reported is that a request was outstanding essentially all the time, which one dd is enough to achieve.

Now aqu-sz, which is the honest companion: it is about 1.0 in all three, because each load is a single stream with one request outstanding. One request outstanding on average — exactly one dd with no parallelism. A device with %util 99 and aqu-sz 1 is not saturated; it has one customer who never leaves. Saturation looks like aqu-sz climbing well above the device's parallelism while await rises with it.

Then r_await. Sequential reads averaged 4.31 ms and random reads 0.46 ms — the sequential requests are slower per request because each one is 256 times larger. This is why comparing await between workloads without looking at rareq-sz is meaningless, and why a single "average disk latency" panel misleads.

The sync writes are the interesting case. Only 2.4 MB/s, on a device that just did 236 MB/s. oflag=dsync forces each 4 KiB write to be durable before the next begins, so there is no batching, no merging and no page cache — every write pays the full round trip. That is the number a database's commit path lives on, and it is nowhere near the throughput figure on the datasheet.

If your %util shows well under 100 with high throughput, you have a device with real parallelism and a workload using it — which is the one case where %util is telling you something. It still cannot tell you how much headroom is left.

--- after pass 1 ---
rchar:                 536918124
syscr:                      4183
read_bytes:            536870912
--- after pass 2 ---
rchar:                1073789036
syscr:                      8279
read_bytes:            536870912
--- after pass 3 ---
rchar:                1610659948
syscr:                     12375
read_bytes:            536870912

Total DISK READ:       172.44 M/s | Total DISK WRITE:         0.00 B/s
Current DISK READ:     172.44 M/s | Current DISK WRITE:       0.00 B/s
    TID  PRIO  USER   DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
   8412  be/4  root   172.44 M/s    0.00 B/s ?unavailable?  cat /tmp/attrib

cgroup2fs
252:0 rbytes=1610612736 wbytes=48234496 rios=39204 wios=8812 dbytes=0 dios=0
some avg10=38.44 avg60=12.10 avg300=3.02 total=118204773
full avg10=31.02 avg60=9.88 avg300=2.44 total=98412037

What to read out of this.

Follow the two counters. After pass 1 they are almost identical: 536918124 requested, 536870912 read from the device. The cache had just been dropped, so every byte came off the disk.

After pass 3 rchar has reached 1610659948 — the full three passes over the 512 MiB file — while read_bytes has not moved since pass 1. The file was read three times and the device delivered it once. Passes two and three were served entirely from the page cache.

That gap is the value of the cache, measured per process, and you can compute it from one file with no tooling at all: 1 − read_bytes/rchar is this process's cache hit ratio by volume. Here it is 67%, and it climbs towards 100% the more times the data is reused.

syscr rising by about 4096 per pass for 512 MiB works out at 128 KiB per call — the buffer GNU cat uses. Divide the same bytes into 4 KiB calls and each pass would take 131,072 syscalls instead, with the kernel-crossing cost from Module 01 paid on every one.

One subtlety worth keeping: these counters belong to the shell, not to cat. A parent inherits the I/O counters of each child it reaps, which is why the totals accumulate cleanly here — and why, on a real system, a supervisor process can appear to be doing I/O that was actually done by short-lived children it has already collected.

Then iotop -b -o -n 2: -o shows only processes actually doing I/O, -b is batch mode so it works through a pipe with no terminal. It attributes 172 MB/s to one cat. Useful — but note two things. It reports threads (TID), so a multi-threaded database appears as many rows and you have to sum them yourself. And the SWAPIN and IO columns read ?unavailable?: those need per-task delay accounting, which has been off by default since Linux 5.14. sudo sysctl -w kernel.task_delayacct=1 turns it on and the columns start working; DISK READ and DISK WRITE work either way.

Finally the cgroup view. io.stat is keyed by major:minor — 8:0 is the whole device — and gives bytes and IOPS in both directions. On a container host, reading the same file inside each container's cgroup directory tells you which workload is generating the load, which no per-device metric can do. (How a container gets its own cgroup, and how to map one back to a pod name, is Module 12.)

And io.pressure gives the honest answer to "is this hurting": full avg10=31.02 means that for 31% of the last ten seconds, every runnable task in this cgroup was stalled on I/O.

If /proc/$READER/io is empty or permission-denied, you are reading a subshell rather than the cat; the fallback in the script finds the child. Reading another user's io file needs root.

I would start by not using iowait for it. iowait is a kind of idle — a CPU is charged an iowait tick only when it has nothing to run and something on it is blocked on I/O — so it falls to zero the moment the machine has other work, and it rises on a quiet machine doing modest I/O. It has no device dimension and no process dimension.

What I actually run: /proc/pressure/io for how much time was genuinely lost to I/O stalls, iostat -x 1 for per-device r_await/w_await and aqu-sz, and the b column of vmstat 1 for a straight count of tasks blocked in uninterruptible sleep. Then iotop -o or /proc/PID/io to attribute it.

The details that separate candidates: being able to state the failure mode in both directions. A busy application server with saturated storage often shows 0% iowait, because its CPUs are never idle — so an iowait-based alert will never fire on the case you care about. And a host running a nightly backup shows 40% iowait while being perfectly healthy. Quoting the kernel's own documentation helps: /proc documentation says outright that "the iowait is not reliable by reading from /proc/stat".

Not necessarily, and on a modern device almost certainly not. %util is the percentage of time during which at least one request was outstanding. On a spinning disk, which serves one request at a time, that is a real utilisation figure. On an NVMe device with dozens of hardware queues, a single outstanding request out of a possible sixty-four keeps %util at 100% while the device is nearly idle. The iostat man page says so explicitly.

To decide whether it is really saturated I would look at aqu-sz — the average queue depth — together with r_await/w_await. Saturation looks like the queue growing beyond what the device can serve in parallel and latency rising with it. %util 100 with aqu-sz 1 is one process doing steady I/O, nothing more.

The details that separate candidates: giving the concrete demonstration — the same device can show 99% %util while delivering 236 MB/s sequentially, 16 MB/s in 4 KiB random reads, and 3 MB/s of synchronous writes. Seventy times the difference in real work, identical %util. It is also worth mentioning that svctm was removed from iostat in sysstat 12.1.2 and avgqu-sz renamed to aqu-sz, so any runbook still parsing those columns has been broken for years.

iotop -o gives a live view, showing only threads currently doing I/O. For something scriptable and needing no packages, /proc/PID/io has read_bytes and write_bytes, which count bytes that actually reached a block device. On a container host, cgroup v2 io.stat attributes bytes and IOPS per cgroup, which is the only way to name the workload responsible.

The details that separate candidates: the comparison between rchar and read_bytes in the same file. rchar is what the process asked for; read_bytes is what came off the device. A process with a large rchar and near-zero read_bytes is being served entirely by the page cache and is costing the disk nothing — and 1 − read_bytes/rchar is a per-process cache hit ratio you can compute from two lines of a file.

The other detail is a caveat: write_bytes is charged to whoever dirtied the pages, not to the thread that wrote them out, and writeback happens later. So a process can exit before its writes reach the device, and per-process write graphs will not line up with the device graph. That is why iotop often attributes bulk write throughput to kernel threads.

On a cloud VM with a network-attached SSD rated around 3,000 IOPS:

parallel=1   r/s=1204.00  r_await=0.79   aqu-sz=0.95   %util=96.40
parallel=2   r/s=2288.00  r_await=0.85   aqu-sz=1.94   %util=99.20
parallel=4   r/s=2941.00  r_await=1.34   aqu-sz=3.94   %util=99.60
parallel=8   r/s=3012.00  r_await=2.62   aqu-sz=7.89   %util=99.80
parallel=16  r/s=3018.00  r_await=5.24   aqu-sz=15.82  %util=99.80

What to read out of this, column by column.

r/s climbs from 1204 to 2941 as you add readers, and then stops: 3012 at eight readers, 3018 at sixteen. That is the device's service rate, and no amount of extra parallelism moves it. You have found the knee.

r_await tells the story of what that costs. It barely moves from one reader to two — 0.79 ms to 0.85 ms — because the device had spare capacity. From four to sixteen readers it goes 1.34 → 2.62 → 5.24 ms, roughly doubling each time you double the load, while throughput does not improve at all. Every millisecond added after parallel=4 bought nothing.

aqu-sz is the cleanest signal in the table: 0.95, 1.94, 3.94, 7.89, 15.82. It tracks the number of outstanding requests almost exactly, and it started rising immediately — long before latency became noticeable. This is the metric that gives you warning.

And %util is 96, 99, 99, 99, 99 throughout. It was effectively pinned from the very first run, when the device had two and a half times more capacity available. It never once indicated how much headroom was left.

The general rule this demonstrates: r_await × r/s ≈ aqu-sz. At parallel=8, 2.62 ms × 3012/s = 7.9, which is the observed queue depth. When you see queue depth rising while throughput is flat, latency is being manufactured by queueing and adding load will only make it worse.

If your numbers keep climbing past 16 with no latency penalty, you have a device with more parallelism than this one — raise the parallelism until r/s plateaus. The shape of the curve will be the same.

(bash's `time` also prints `user` and `sys` lines; trimmed here)

--- buffered: no durability guarantee at all ---
real	0m0.161s
--- one fsync at the end ---
real	0m1.284s
--- O_DIRECT: bypass the page cache ---
real	0m1.902s
--- O_DSYNC, 1 MiB writes: durable per write ---
real	0m2.106s
--- O_DSYNC, 4 KiB writes: the commit path of a database ---
real	0m6.618s

kernel.io_uring_disabled = 0
io_uring compiled in

What to read out of this.

Buffered wrote 256 MiB in 0.161 s — about 1.6 GB/s, which is the speed of memory, not of storage. Every number below it is closer to the truth.

One fsync at the end took 1.284 s, so the honest sequential write throughput is around 200 MB/s. Eight times slower than the buffered figure, and this is the number to quote if anyone asks what the disk does.

O_DIRECT at 1.902 s is slower than buffered-plus-fsync, which surprises people. Bypassing the cache removes the kernel's ability to batch and merge on your behalf, so unless the application does its own batching — as a database does — you lose more than you gain. O_DIRECT is not a performance flag; it is a control flag.

O_DSYNC with 1 MiB writes: 2.106 s. Each of the 256 writes waits for durability, so you pay 256 round trips instead of one — but each round trip carries a megabyte, so the overhead is diluted.

And then the last line, where it is not diluted at all. O_DSYNC with 4 KiB writes: 4,000 writes totalling only 16 MiB, in 6.6 seconds — about 2.4 MB/s, or 604 writes per second, on a device that wrote at 1.6 GB/s a moment ago. That is a factor of roughly 660, and nothing about the hardware changed. Every write paid a full round trip of about 1.6 ms and none of them could be merged. Note that this figure matches the synchronous-write row in Exercise B2.1 exactly, which is the point: it is a property of the device's per-request latency, not of the tool.

This is the single most important I/O number for anyone running a database. Commit latency lives here, not on the throughput figure in the datasheet, and it is why "the volume is rated for 500 MB/s" tells you nothing about how many transactions per second you will get.

kernel.io_uring_disabled = 0 means unrestricted, which is the Debian and Ubuntu default. On RHEL 9 you would see 2 — disabled for every process. If the sysctl does not exist at all, the kernel predates 6.6 and io_uring is simply on if compiled in.

/dev/vda1 ext4   rw,relatime,errors=remount-ro
relatime

atime=2026-08-21 11:04:13.229481170 +0000
atime=2026-08-21 11:04:13.229481170 +0000

2000 small files wrote 8692 KiB to the device
  (the file data itself is 2000 x 2 bytes = 4 KiB)
one 4 KiB file wrote 32 KiB to the device

no discard mount option
● fstrim.timer - Discard unused filesystem blocks once a week
     Loaded: loaded (/usr/lib/systemd/system/fstrim.timer; enabled)
     Active: active (waiting) since Mon 2026-08-17 09:12:44 UTC

/dev/vda1        40G   22G   16G  59% /
held by deleted-but-open files: 1842.0 MiB

What to read out of this.

The two atime values are identical, so the second read did not write anything. That is relatime working — but note carefully which read it applied to. A freshly created file has atime exactly equal to mtime, and relatime updates when the stored atime is not newer than mtime, so the first read did generate a metadata write. Only once atime had moved ahead did the second read become free.

This catches people writing benchmarks: touch a file, read it once, and you have measured the update, not the skip. Under the old strictatime behaviour every read would have generated a metadata write and, eventually, a journal entry.

Now the striking number. 2000 files containing four kilobytes of actual data cost about 8700 KiB of device writes — more than two thousand times the payload. Part of that is unavoidable: ext4 allocates a whole 4 KiB block per file, so 2000 files occupy 8000 KiB no matter how few bytes they hold. The rest is metadata — an inode, a directory entry, block-bitmap and inode-bitmap updates per file — and every one of those changes goes through the journal. Compare the single 4 KiB file: 32 KiB, almost all of it metadata, but paid once.

This is why "extract this tarball" or "clone this repository" hammers a disk far harder than the byte count suggests, why container image pulls are so I/O-expensive, and why a build system that creates and deletes millions of small files will saturate storage that copes easily with a database doing the same total volume.

fstrim.timer is active (waiting) and there is no discard mount option. That is the modern arrangement and it is correct: discard is batched weekly rather than issued on every delete.

Finally the df/lsof pair. The filesystem is 59% full, and 1842 MiB is held by files that have been deleted but are still open. Nothing on disk corresponds to that space — du will never find it — and it will be released only when the holding process closes the descriptor or exits. On a host where a log rotation deleted files that the service still has open, this is the entire explanation for "the disk filled up and du says there is nothing there".

If your small-file number is much lower, the filesystem batched the journal writes across the loop, which sync placement affects. The ratio to the payload will still be enormous.

Almost certainly the arrival rate crossed the device's service rate. Queueing latency does not rise gently as you approach capacity — it rises without bound. A device comfortably serving 4,000 IOPS at 1 ms can be at tens of milliseconds at 4,500. Nothing broke and the device did not slow down; the queue in front of it grew.

To confirm it I would look at aqu-sz in iostat -x alongside r/s/w/s: if throughput has plateaued while queue depth and await climb together, that is the signature. On a cloud volume I would also check whether the plateau is a provider IOPS quota rather than a physical limit — the provider's throttling metrics will show it when the guest's will not.

The details that separate candidates: explaining why the change looks like a cliff. Average latency is flat right up until the knee, so a latency-based capacity model gives no warning at all; queue depth starts rising much earlier and is the metric to alert on. The corollary is worth stating too: past the knee, adding application workers makes it worse, because more concurrency adds queueing and cannot add service rate.

fsync() flushes everything written to that file so far to stable storage; you call it when you choose, so the cost is amortised across many writes. O_DSYNC makes every individual write durable before it returns — the same guarantee, paid per write, which is enormously more expensive for small writes. O_DIRECT is not a durability mechanism at all: it bypasses the page cache so reads and writes go straight between the application's buffer and the device, and the data can still be sitting in the device's volatile cache when the call returns.

The details that separate candidates: three things. O_DIRECT still needs fsync — conflating them is a real data-loss bug, not a theoretical one. O_DIRECT is usually slower, not faster, unless the application does its own batching, because it gives up the kernel's merging and readahead; it is a control flag, not a performance flag. And the magnitude: on the same device, buffered writes can run at 1.6 GB/s while 4 KiB O_DSYNC writes manage under 1 MB/s — a factor of two thousand, with no hardware change. That number is where a database's commit rate actually lives.

Because the cost is metadata, not data. Each file needs an inode, a directory entry, and updates to the inode and block bitmaps, and on a journalling filesystem every one of those metadata changes is written to the journal first. Two thousand two-byte files can generate several megabytes of device writes for four kilobytes of payload — a ratio in the thousands — while the same bytes in one file cost a few tens of kilobytes.

The details that separate candidates: generalising it to the workloads where it bites — container image pulls, git clone, CI workspaces, node_modules — and noting that these hosts must be sized on write IOPS, not throughput, because a byte-based capacity model predicts none of it. It is also worth naming the ext4 default, data=ordered: metadata is journalled and data is not, but data is forced out before the metadata pointing at it, which is what stops a crash leaving a file that references someone else's old blocks.

--- 1. pressure ---
some avg10=22.41 avg60=6.02 avg300=1.44 total=42118204
full avg10=18.90 avg60=5.11 avg300=1.20 total=38204118

--- 2. device ---
Device   r/s  rkB/s  r_await   w/s   wkB/s  wrqm/s  %wrqm  w_await  wareq-sz  aqu-sz  %util
vda     0.00   0.00     0.00 604.00 2416.00    0.00   0.00     1.58      4.00    0.95  98.40

--- 3. queue depth vs throughput ---
w/s=604.00  w_await=1.58  wareq-sz=4.00  aqu-sz=0.95  %util=98.40

--- 4. attribution ---
Total DISK READ: 0.00 B/s | Total DISK WRITE: 2.36 M/s
    TID  PRIO  USER   DISK READ  DISK WRITE  COMMAND
   9214  be/4  root    0.00 B/s    2.36 M/s  dd if=/dev/zero of=/tmp/w3 ...

wchar:                 32768
write_bytes:           32768
syscw:                     8

Walk it through in order, and notice how each answer narrows the next.

1. Pressure is real. some avg10=22.41 and full avg10=18.90 — for nearly a fifth of the last ten seconds, every runnable task was stalled on I/O. This is a genuine storage problem, not a misread dashboard. Had this been near zero, everything below would have been a waste of time.

2. The device is writing, not reading, at 2.4 MB/s — a trivial amount. And yet %util is 98.4. If you had started here you would have concluded the disk was maxed out. It is not; it is merely never empty.

3. The decisive line. aqu-sz=0.95 — the queue is essentially one request deep. So this is not queueing. wareq-sz=4.00 says every request is 4 KiB, and wrqm/s=0.00 says not one write was merged, which for 400 writes to only 8 files should be impossible unless something is forcing each one out on its own. That is the O_DSYNC signature, and it identifies the cause before you have looked at a single process.

At 604 writes/s and 1.58 ms each, the device is doing exactly what it is asked and doing it promptly. The workload is the problem, not the storage.

4. Attribution confirms it. One dd at a time, and the per-process counters show wchar and write_bytes identical at 32768 — every byte the process wrote went straight to the device, with nothing absorbed by the page cache. That equality is itself diagnostic: on a normal buffered writer, write_bytes lags wchar badly.

syscw: 8 for 32 KiB is 4 KiB per call, matching wareq-sz exactly. The chain from application call size to device request size is unbroken, which is precisely what synchronous, unmergeable I/O looks like.

The conclusion you would put in the ticket: the volume is healthy — sub-2 ms latency at queue depth 1 — and the application is issuing small synchronous writes that cannot be batched. The fix is in the application's write pattern or its durability settings, not in a bigger disk.

/proc/pressure/io first, because it answers whether anything is actually being held up and it takes no packages and no time. If pressure is near zero, storage is not the problem and I stop. Then iostat -x 1 for the device view: r_await/w_await for latency, aqu-sz for queue depth, rareq-sz/wareq-sz for request size. Those three together separate queueing from a slow device from a bad workload. Only then do I attribute it — iotop -o, /proc/PID/io, or per-cgroup io.stat on a container host.

The details that separate candidates: saying which metrics you deliberately do not start with, and why. iowait moves with CPU load rather than with storage health, and %util is pinned near 100 by a single outstanding request on any device with parallelism. Both are on the default dashboard and both will mislead you. The other detail is capturing before mitigating: a restart clears the symptom and destroys the only evidence that could distinguish a workload change from a device problem.

That the device — or the path to it — is genuinely slow, rather than overloaded. If aqu-sz is around 1 and await is in the tens of milliseconds, nothing is waiting behind anything else; each individual request is simply taking a long time. Adding capacity or reducing concurrency will not help, because there is no queue to shorten.

Next I would check request size. Very small requests with high per-request latency and zero merging points at synchronous, unmergeable writes — an application calling fsync per record, or opening with O_DSYNC. If the requests are large and still slow, I look below the guest: a degraded array, a network-attached volume, or a provider throttle.

The details that separate candidates: knowing that on cloud storage the limit is usually a quota rather than physics, and that quotas have tells — a plateau at a round number like 3,000 IOPS, or twenty minutes of good performance followed by a permanent halving as a burst balance runs out. That second pattern is invisible from inside the guest and looks exactly like a device that spontaneously got slower.

What you tell them: %util does not mean what the alert assumes. It is the percentage of time during which at least one request was outstanding — nothing about capacity. On an NVMe device with dozens of hardware queues, one request in flight out of a possible sixty-four pins it at 100% while the device is nearly idle. The iostat man page states this outright: for devices serving requests in parallel, the number "does not reflect their performance limits".

The evidence that ends the discussion, and it takes one minute:

iostat -x 1 3 nvme0n1

If %util is 100 while aqu-sz is around 1 and await is well under a millisecond, the device has one steady customer and enormous headroom. Show that next to the alert and the case is closed.

What to replace it with. Three series and no %util: aqu-sz (rises early and smoothly — this is the alerting metric), r_await/w_await (what the application actually experiences), and /proc/pressure/io (some avg60, how much work is genuinely being held up). Page on pressure, ticket on sustained queue growth, and graph latency.

One caveat worth stating. %util is not always useless — on a genuinely serial device, a single spinning disk, it really is utilisation. If the fleet is mixed, the alert is right on some hosts and wrong on others, which is worse than being uniformly wrong and is probably why it survived this long.

No, and the numbers already say so.

aqu-sz at 1.0 means there is no queue: requests are not waiting behind other requests. %util at 97% with a queue depth of one means the device is never idle but never backed up — one customer who never leaves. And 4 MB/s against a 500 MB/s rating means throughput is not remotely the constraint. Buying a volume with more throughput buys more of a resource that is 99% unused.

Run this to confirm what the workload looks like:

iostat -x 1 5            # look at wareq-sz and wrqm/s
cat /proc/pressure/io
grep -E '^(wchar|write_bytes|syscw):' /proc/<postgres-pid>/io

What you will find. wareq-sz around 4 KiB — 4 MB/s divided by roughly 1,000 writes per second — and wrqm/s at or near zero. Nothing is being merged, because each commit is forced to storage on its own. This is the write-ahead log, and PostgreSQL is doing exactly what it promised: every transaction is durable before it acknowledges.

So what does limit it? Per-write latency. At w_await of around 1 ms, a strictly serialised commit path caps out near 1,000 transactions per second, which is where they are. The lever is latency, not bandwidth — a volume type with lower latency, or storage with a battery-backed write cache.

The cheaper fix to raise first. PostgreSQL's commit_delay and commit_siblings batch concurrent commits into one flush, so several transactions share a single round trip. That is a configuration change that can multiply throughput without touching the hardware, and it costs a few milliseconds of added commit latency. synchronous_commit = off would go further and trades durability for it — a decision for the data owner, not for you.

The numbers describe the queueing knee, precisely. Throughput is flat at 125 MB/s while aqu-sz has reached 22 and r_await has gone from 2 ms to 45 ms. Requests are not slower to serve; they are waiting behind twenty-one others. Latency has risen twentyfold for zero extra work done.

The plateau at 125 MB/s is the number to stare at. That is suspiciously close to a round 1 Gbit/s, and round numbers in storage throughput are almost always a quota rather than physics.

Run, in order:

iostat -x 1 10                    # confirm the plateau is stable
cat /proc/pressure/io             # how much is being lost
# then the provider's own metrics: volume throughput limit,
# burst balance, and any throttling counter

Two candidate causes, and they are distinguishable.

If the volume's size or type imposes a 125 MB/s ceiling, the plateau will be flat and permanent, and the provider's throttling metric will be non-zero. The fix is a larger or different volume — on most providers throughput scales with provisioned size.

If it is a burst balance, the job will have been fast for the first stretch and then halved. Look at the job's own timing: 40 minutes becoming 3 hours is roughly a fourfold slowdown, which fits a burst credit running out partway. That is invisible from inside the guest.

What has changed after a year of working. Almost certainly the data grew past the point where the job's read volume exceeds the burst allowance, or past what the cache could absorb. Nothing was deployed because nothing needed to be — the workload grew into the limit. Check the input data size trend before anything else; it is usually the whole answer.

The 2 GB figure is the payload, not the work. CI builds create and delete enormous numbers of small files — checkouts, dependency trees, object files, container layers — and each file costs an inode, a directory entry, bitmap updates, and journal writes for every one of those. Two thousand tiny files can generate megabytes of device writes for kilobytes of payload; a full node_modules tree or a container image pull does this on an industrial scale.

Confirm it by comparing bytes to IOPS:

iostat -x 1 10                         # look at w/s and wareq-sz
# device writes over the build, from /proc/diskstats field 10 (sectors)
# compared with the build's own byte count

What you will find. w/s in the thousands with wareq-sz around 4–8 KiB. The fleet is IOPS-bound, not throughput-bound, and every capacity model built on gigabytes-per-build predicted none of it.

Fixes, cheapest first.

Size the volumes on write IOPS, since on most cloud providers IOPS and throughput are provisioned separately and the default assumes large sequential I/O.

Put the build workspace on a tmpfs or a local NVMe scratch disk where the machine has one — build artefacts are disposable by definition, so durability buys nothing.

Reduce the file churn itself: a shared dependency cache instead of a fresh install per build, and fewer, larger layers in container images.

One thing not to do: mount -o discard. On a filesystem with this much deletion it adds a discard to every unlink and makes the latency worse. The weekly fstrim.timer does the same job in one batch, and it is already enabled on modern distributions.