🗄️Module 09 — Page Cache, Swap & the OOM Killer

MemFree:         2884416 kB
Buffers:            8104 kB
Cached:           184920 kB

--- cold read ---
real	0m2.914s
user	0m0.004s
sys	0m0.221s

MemFree:         2358140 kB
Buffers:            9216 kB
Cached:           712488 kB

--- warm read ---
real	0m0.118s
user	0m0.008s
sys	0m0.110s

What to read out of this.

The cold read took 2.9 seconds. The warm read took 0.118 seconds — about 25 times faster, running the identical command on the identical file. Nothing was optimised. The data was simply already in RAM.

Now follow the memory. Cached went from 184 MB to 712 MB — an increase of 527 MB, which is the 512 MB file plus a little. MemFree fell by almost exactly the same amount, from 2884 MB to 2358 MB.

If you were watching a free graph you would see 526 MB of memory "disappear" the moment someone read a file. Nothing disappeared. It moved from the free column to the cache column, and it is available again the instant anything asks for it.

Look at the sys times as well. The cold read spent 0.221 s in the kernel and 2.9 s in total — so 2.7 seconds of it was spent waiting, not computing. The warm read spent 0.110 s in the kernel out of 0.118 s total: almost no waiting at all. That waiting is exactly the wa column from Module 07, and it is why a machine with a cold cache looks like it has a slow disk.

If your two times are almost identical, the drop went to a filesystem that was already serving from a lower-level cache — common on some virtualised storage and on machines with a lot of free RAM on the hypervisor. Try a file several times larger than the guest's RAM if you want to force the difference.

               total        used        free      shared  buff/cache   available
Mem:            3844          78        3510          12         276        3766

making a 3766 MiB file

               total        used        free      shared  buff/cache   available
Mem:            3844          74        3700          12          86        3770

--- after reading the file ---
               total        used        free      shared  buff/cache   available
Mem:            3844         138         402          12        3439        3706

MemTotal:        3936256 kB
MemFree:          411648 kB
MemAvailable:    3794944 kB
Buffers:           43712 kB
Cached:          3379968 kB
Shmem:             12928 kB
SReclaimable:      98304 kB

--- allocating 800 MB ---
               total        used        free      shared  buff/cache   available
Mem:            3844         940         150          12        2880        2904

What to read out of this.

After reading the file, free collapsed from 3510 MB to 402 MB. A monitoring dashboard would call that an 89% memory alert. Look at the same row's last column: available went from 3766 to 3706 — a drop of 60 MB, which is nothing. The memory changed category, not owner.

buff/cache absorbed it: 276 MB to 3439 MB.

Now the proof, in the last block. A program asked for 800 MB on a machine showing 402 MB free. It got it — no error, no ENOMEM. buff/cache dropped from 3439 MB to 2880 MB, so the kernel threw away 559 MB of cached pages and took the rest from what was free. Nobody was consulted and nothing was logged.

available fell from 3706 to 2904, a drop of 802 MB for an 800 MB allocation. That is the only column that tracked reality, and it tracked it almost exactly.

One detail worth noticing for later: Cached: 3379968 kB includes Shmem: 12928 kB. Shared memory and tmpfs files appear inside Cached but cannot be dropped, because there is no disk copy to reload them from. It is a rounding error on this machine; Section D2 shows one where it is 871 MB, and it is one of the reasons MemAvailable exists rather than leaving you to add the columns yourself.

Note on used. On Ubuntu 24.04 and other procps-ng 4 systems, used is total − available — which is why it barely moves in this transcript even as free collapses. On an older procps it would be total − free − buff/cache and would look quite different for the same machine.

Dirty:               132 kB
Writeback:             0 kB

vm.dirty_ratio = 20
vm.dirty_background_ratio = 10

--- no fsync ---
real	0m0.612s

Dirty:            412160 kB
Writeback:         42112 kB

--- the bill arrives ---
real	0m3.884s
Dirty:                 8 kB
Writeback:             0 kB

--- with fsync ---
real	0m4.301s

What to read out of this.

dd reported writing 1 GiB in 0.612 seconds — about 1.7 GB/s. That is not the disk's speed. It is the speed of copying into RAM. Immediately afterwards, Dirty reads 402 MB: about 40% of what was "written" had not touched the disk at all, and Writeback: 42112 kB shows another 42 MB in flight.

Then sync took 3.884 seconds doing nothing but waiting for the disk — flushing what was still dirty plus what was still queued behind it. Add the two together and you get about 4.5 seconds, which matches the honest measurement at the bottom: conv=fsync took 4.301 seconds for the same gigabyte.

The write was never fast. The cost was deferred, and dd took credit for work it had not done. This is why benchmark numbers without fsync are meaningless, and why "our storage does 1.7 GB/s" claims should always be met with "measured how?".

Notice Dirty was nowhere near the full 1 GiB. Background writeback had already started well before the write finished, which is dirty_background_ratio doing its job. Push the file size up and you can watch dd itself get blocked partway through on dirty_ratio — the write stalls mid-command, for seconds, with nothing in any log to explain it.

A subtle trap in the numbers. Both ratios are percentages of dirtyable memory — free pages plus reclaimable file pages — not of total RAM, which is why you cannot work out the threshold in your head from MemTotal. On a machine whose available memory is shrinking, the dirty thresholds shrink with it, so a write pattern that ran fine yesterday can start stalling today without anyone changing a setting.

Six columns. total is usable installed RAM. free is memory holding nothing at all. buff/cache is the page cache plus filesystem metadata buffers — file data the kernel is keeping in RAM because throwing it away would be pointless. shared is mostly tmpfs. used is a derived leftover: total minus free minus buff/cache. And available is the kernel's own estimate of what a new program could obtain without swapping.

The one to read is available. free being near zero is the normal, healthy state of a Linux machine that has been up for a while.

The details that separate candidates: saying that available is not free + buff/cache. It comes from MemAvailable, added in Linux 3.14, and it deliberately excludes cache that cannot actually be reclaimed — dirty pages awaiting writeback, tmpfs and shared memory that has no disk copy to reload from, and a working reserve the kernel keeps so it does not grind. Anyone still adding the columns by hand is producing a worse number than the kernel already publishes.

Not at all yet, and I would not act on that number. I would run free -m and read the available column, or grep MemAvailable /proc/meminfo. If the memory is in buff/cache, it is the page cache doing its job and it will be handed back the moment anything needs it. If available is also near zero, then there is something to investigate.

The real signal is not a level at all — it is /proc/pressure/memory, which reports how much time processes actually lost waiting for memory. A machine at 98% with zero memory pressure is fine. A machine at 70% with rising pressure is in trouble.

The details that separate candidates: naming this as the classic bad alert and explaining what to replace it with. "Memory usage above 90%" fires constantly on healthy hosts, so teams either raise the threshold until it is useless or learn to ignore it — and then miss the host that is genuinely failing. Alerting on MemFree is alerting on how much RAM you are wasting. The professional answer is MemAvailable as a ratio, plus PSI memory pressure as the leading indicator.

No. It is in the page cache, marked dirty, and the kernel will write it out at some point over the following seconds. If the machine loses power before then the data is gone, and the file can be left partially updated. Durability requires fsync() on the file descriptor — or fdatasync(), or opening with O_SYNC, or sync for the whole system.

The details that separate candidates: two follow-ons. First, fsync on the file is not always enough: creating or renaming a file also needs the directory fsynced, which is why the safe write pattern is write-to-temp, fsync the temp file, rename, then fsync the directory. Second, this is exactly why storage benchmarks without fsync are meaningless — dd writing a gigabyte in half a second is measuring the speed of memory, and the disk time arrives later on someone else's stopwatch.

MemTotal:        3936256 kB
MemAvailable:    3181204 kB
Cached:           412508 kB
Active(anon):     284116 kB
Inactive(anon):    41208 kB
Active(file):     102344 kB
Inactive(file):   298760 kB
SwapTotal:             0 kB
SwapFree:              0 kB

--- cache filled ---
MemFree:         1332108 kB
Cached:          1948116 kB
Active(file):     118904 kB
Inactive(file):  1821644 kB

--- 1 GB of anonymous memory requested ---
MemFree:          331884 kB
Cached:          1905228 kB
Inactive(anon):  1304552 kB
Inactive(file):  1782108 kB

pgsteal_kswapd 0
pgsteal_direct 0
pgscan_kswapd 0
pgscan_direct 0

What to read out of this.

Start with the split. After reading the file, Inactive(file) holds 1821 MB — the file was read once, so its pages went straight onto the inactive list. Pages only reach Active(file) if they are used a second time. That is the kernel refusing to let a single large sequential read push everything else out, and it is why cat-ing a huge file does not destroy your database's cache.

Then the anonymous request. Inactive(anon) jumped to 1274 MB — note inactive, not active: since Linux 5.8 a page that has just been allocated and written once sits on the inactive anon list and is only promoted if something touches it again. MemFree fell by roughly the gigabyte that was asked for, and Cached gave up 42 MB.

On this run the kernel did not have to sacrifice much cache at all, because there was still a gigabyte free. That is the normal case, and it is worth seeing before the interesting one. Repeat the exercise asking for 2.5 GB instead of 1 GB and Cached will collapse: the kernel discards page cache without hesitating, without logging anything, and without any delay you would notice.

Now the counters, which are the part most people have never looked at. pgscan_kswapd and pgsteal_kswapd are reclaim done by the background thread; pgscan_direct and pgsteal_direct are reclaim done by the allocating process itself, while it waits. All four are 0 here, which is exactly what a machine with room to spare looks like — nothing had to be reclaimed at all.

pgsteal_direct climbing is one of the best early warnings of memory trouble there is — it means allocation is no longer free, and processes are doing the kernel's housekeeping on their own time.

Finally, note SwapTotal: 0. Every anonymous page on this machine is unreclaimable. The kernel got away with it here because there was cache to give up; Section C2 is what happens when there is not.

On a typical cloud VM you will very often get nothing at all from the first two commands:

SwapTotal:             0 kB
SwapFree:              0 kB
SwapCached:            0 kB

procs -----------memory---------- ---swap-- -----io----
 r  b   swpd   free   buff  cache   si   so    bi    bo
 0  0      0  98432  43712 1948116    0    0     2    11
 0  0      0  98432  43712 1948116    0    0     0     0

vm.swappiness = 60
(kernel 6.8.0-40-generic - swappiness range is 0-200 since 5.8)

That empty result is the important one. Most cloud images ship with no swap configured at all. swapon --show prints nothing, /proc/swaps has only its header, and SwapTotal is 0. Every anonymous page on that machine is permanently unreclaimable, and the kernel's only lever under pressure is throwing away page cache. When that runs out, it kills something. There is no gradual phase.

On a machine that does have swap you will see something like:

NAME      TYPE      SIZE  USED PRIO
/swap.img file        2G  412M   -2

SwapTotal:       2097148 kB
SwapFree:        1675260 kB
SwapCached:        18432 kB

 r  b   swpd   free   buff  cache   si   so    bi    bo
 0  0 421888 112884  38104 905228    0    0    14    62
 0  0 421888 112884  38104 905228    0    0     0     0

  204800 kB  mysqld
   96256 kB  snapd
   41984 kB  containerd

Read the level and the rate separately. swpd 421888 says 412 MB is parked in swap — 20% of the swap file. That is a level, and on its own it means nothing except that some pages have not been touched in a long while. si and so are both 0: nothing is moving. This machine is not swapping. It has swapped, once, some time ago, and everything has been quiet since.

The per-process list tells you who. mysqld has 200 MB parked — almost certainly start-up code and configuration structures it touched once and never again. Forcing that back into RAM would gain nothing and cost 200 MB of cache.

The per-process figures sum to 343 MB while SwapTotal − SwapFree is 412 MB. That 69 MB gap is real and not a rounding error: swapped-out tmpfs and shared-memory pages belong to no single process, so they appear in the totals but in nobody's VmSwap. The per-process list will always undercount, which is worth knowing before you go looking for the missing megabytes.

SwapCached: 18432 kB is a subtlety worth knowing: those pages are in both places, RAM and swap. They were faulted back in but the swap copy was kept, so if they need evicting again the kernel can drop them for free. It is not double-counted waste; it is a saved write.

The alerting rule that falls out of this: never alert on swap used. Alert on si/so sustained above zero, or better, on memory pressure from Section B3.

some avg10=0.00 avg60=0.00 avg300=0.00 total=1841203
full avg10=0.00 avg60=0.00 avg300=0.00 total=712044
some avg10=0.12 avg60=0.31 avg300=0.44 total=98412037
full avg10=0.00 avg60=0.02 avg300=0.04 total=2104118

total=1841203
total=712044

available: 3021 MiB - making a 6042 MiB file
--- reading it three times ---
10:22:14 mem some avg10=0.00 avg60=0.00 avg300=0.00 total=1841203
         mem full avg10=0.00 avg60=0.00 avg300=0.00 total=712044
         io  some avg10=1.84 avg60=0.62 avg300=0.19 total=98449237
10:22:16 mem some avg10=4.21 avg60=1.02 avg300=0.28 total=1925403
         mem full avg10=0.31 avg60=0.08 avg300=0.02 total=718244
         io  some avg10=38.72 avg60=12.44 avg300=3.81 total=99223637
10:22:18 mem some avg10=7.88 avg60=2.14 avg300=0.61 total=2083003
         mem full avg10=0.44 avg60=0.14 avg300=0.04 total=727044
         io  some avg10=44.10 avg60=16.92 avg300=5.22 total=100105637
10:22:20 mem some avg10=9.02 avg60=3.31 avg300=0.98 total=2263403
         mem full avg10=0.51 avg60=0.19 avg300=0.06 total=737244
         io  some avg10=46.33 avg60=20.18 avg300=6.90 total=101032237
10:22:22 mem some avg10=8.41 avg60=4.02 avg300=1.34 total=2431603
         mem full avg10=0.48 avg60=0.24 avg300=0.08 total=746844
         io  some avg10=41.27 avg60=22.44 avg300=8.11 total=101857637

What to read out of this.

First, the two pressure files are different things and the exercise prints both. io pressure went to 46%; memory pressure only reached 9%. That is the correct diagnosis of what we actually built: reading a file too large to cache is an I/O problem, not a memory shortage. Nothing was short of memory — the kernel simply had to keep fetching from disk.

This is the distinction that makes PSI worth learning. A level-based metric would have shown "memory 100% used" for both a healthy cache and a genuine shortage. Pressure tells them apart, and it tells you which resource is hurting.

Second, watch the three averages move at different speeds. At 10:22:16, avg10=38.72 while avg300=3.81. The ten-second figure has already responded; the five-minute figure has barely noticed. For alerting use avg60 — avg10 is noisy enough to fire on a single backup job, and avg300 arrives after the incident is over.

Third, full for memory stayed near zero throughout. That is the reassuring signal: at no point was every task stalled. A full avg60 climbing past 10 is the one that means the machine is thrashing rather than merely working hard.

Finally, total= is a monotonically increasing count of microseconds lost. Check the arithmetic against the averages and it holds together: the io some total rises by about 800,000 µs per two-second step, and 800000 ÷ 2000000 is 40% — which is what avg10 says. The delta can never exceed the elapsed wall-clock time, so if you ever compute a figure above 100% you have made an arithmetic error, not found a catastrophe. Two readings a minute apart, subtracted and divided by 60 million, give you the fraction of that minute that was wasted — the most honest single memory metric a fleet can collect.

If your memory pressure stayed at exactly 0.00 throughout, the machine had enough RAM that the file still fitted, or the storage was fast enough that nothing stalled long enough to count. Try a file four times MemAvailable.

Swap is space on storage where the kernel can put anonymous pages — heap, stack, anything with no file behind it — so it can reuse the physical frames. File-backed pages never need it, because they can be dropped and re-read from the file.

Its purpose is not emergency memory. It is a release valve that lets the kernel move cold anonymous pages out of the way so RAM holds things that are actually being used. A machine with plenty of free memory can swap, correctly, as ordinary housekeeping.

The details that separate candidates: stating plainly that without swap, anonymous memory is completely unreclaimable, so the kernel's only lever is evicting page cache — and when that runs out it goes straight to killing a process, with no gradual phase. Turning swap off does not make a machine safer; it makes its failure mode abrupt. Also worth adding: vm.swappiness is not an eagerness dial, it is the kernel's assumed relative I/O cost of swapping versus filesystem paging, on a 0–200 scale with default 60, and values above 100 became valid in Linux 5.8 for cases where swap really is cheaper than re-reading files — zram or NVMe.

First I would establish whether "high swap usage" is a level or a rate, because only one of them is a problem. swapon --show and SwapTotal/SwapFree give the level: how much is parked. vmstat 1 gives the rate: the si and so columns, pages moving in and out per second. A large amount parked with si/so at zero is not a problem at all — those are pages nothing has wanted for hours, and forcing them back would cost cache for no benefit.

If the rate is genuinely non-zero and sustained, I would check /proc/pressure/memory to see how much time is actually being lost, then find which processes hold swap by reading VmSwap from each /proc/PID/status, and look at whether the working set has grown or the available memory has shrunk.

The details that separate candidates: refusing to treat swap-used as an alertable metric, and naming the replacement — si/so sustained above zero, or PSI memory pressure. It is also worth mentioning SwapCached, which is memory present in both RAM and swap: those pages were faulted back in but the swap copy was kept, so evicting them again is free. People see it and assume double-counting or a leak.

Thrashing is when the active working set no longer fits in RAM, so the kernel evicts a page that is immediately needed again, faults it back in, and evicts another that is also immediately needed. Nearly all CPU time goes into reclaim and major faults, and almost no useful work gets done.

The trap is that no level-based metric shows it. Memory utilisation sits at 100% — exactly as it does on a perfectly healthy cache-filled machine. Swap used may not move at all. What moves is time lost: /proc/pressure/memory, especially the full line, plus pgsteal_direct in /proc/vmstat and the major-fault rate.

The details that separate candidates: saying that thrashing is a stable state, not a crash. The machine is not out of memory and will not necessarily be OOM-killed; it can sit at a third of its normal throughput for hours while every dashboard reports full utilisation and no errors. Nothing escalates on your behalf, which is precisely why it needs an alert built on memory full avg60 rather than on any utilisation figure.

vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.overcommit_kbytes = 0

MemTotal:        3936256 kB
CommitLimit:     1968128 kB
Committed_AS:    2841204 kB
SwapTotal:             0 kB

--- mode 0: asking for 2 GB, then 100 GB ---
2 GB: SUCCEEDED - address space reserved, no RAM used yet
100 GB: REFUSED: [Errno 12] Cannot allocate memory

--- switching to strict overcommit (mode 2) for 20 seconds ---
vm.overcommit_memory = 2
CommitLimit:     1968128 kB
Committed_AS:    2841204 kB
2 GB allocation REFUSED: [Errno 12] Cannot allocate memory
vm.overcommit_memory = 0

What to read out of this.

Look at the first block before anything else. Committed_AS is 2841 MB and CommitLimit is 1968 MB. The machine has already promised 44% more than its own strict ceiling would allow — and it is running perfectly. That is overcommit working, and it is the normal state of every Linux machine you have ever used.

CommitLimit here is 1968 MB because it is SwapTotal (0) + 50% of MemTotal. With no swap and the default ratio, strict mode would cap you at half your RAM.

Now the two requests in mode 0. 2 GB succeeded and 100 GB was refused, and the boundary between them is not fuzzy at all: the mode-0 heuristic refuses any single allocation larger than total RAM plus swap, and nothing else. On this machine that is 3.8 GB, so 2 GB sails through and 100 GB is rejected on the spot. Check dmesg after a refusal and you will find the kernel saying so in as many words: __vm_enough_memory: pid: 7647, comm: python3, bytes: 107374182400 not enough memory for the allocation.

That is worth holding next to Module 08. Address space is cheap, but it is not free of all accounting — even the permissive default draws a line, and it draws it at one allocation bigger than the machine.

Then mode 2, and the same 2 GB request that succeeded a moment ago now fails immediately with ENOMEM, on a machine with over 3 GB genuinely available. Nothing was short. The accounting simply said no.

Notice the self-healing timer in the script, and take it seriously. Because Committed_AS already exceeds CommitLimit on this machine, mode 2 makes nearly every subsequent allocation fail — including the sudo sysctl that would set it back. Switching a busy machine to mode 2 without an escape hatch is a good way to need a console.

This is the trade in one screen. Mode 0 says yes and occasionally has to kill something later. Mode 2 says no early and predictably, and wastes capacity to do it. Neither is free, and the right choice depends entirely on whether an unexpected SIGKILL or a failed allocation is worse for your workload.

If your 2 GB allocation succeeded in mode 2, your machine has swap or more RAM than the example — check CommitLimit against Committed_AS and scale the request up until it fails.

736  adj=0  rss=412960kB  mysqld (1198)
686  adj=0  rss=118204kB  python3 (2041)
686  adj=0  rss=116888kB  python3 (2042)
686  adj=0  rss=115932kB  python3 (2043)
672  adj=0  rss=32872kB   dockerd (901)
669  adj=0  rss=14904kB   sshd (688)
667  adj=0  rss=5504kB    bash (4881)
  0  adj=-1000  rss=11208kB  systemd-udevd (412)

--- a small sleeping process ---
666
0
--- after adj=500 ---
1000
--- after adj=-1000 ---
0
--- PID 1 ---
-1000

What to read out of this.

The ranking follows RSS — but look at how little the numbers move. mysqld at 403 MB scores 736; bash at 5 MB scores 667. A process eighty times larger scores 10% higher.

That is because /proc/PID/oom_score is not the raw badness. It is the badness rescaled into a 0–2000 range as roughly (1000 + rss‰) × 2/3, where rss‰ is the process's share of total RAM in thousandths. Every process starts from a floor of 666, and its size only nudges it upward: 403 MB of a 3.8 GB machine is 105 thousandths, and (1000 + 105) × 2/3 = 736. Do not read oom_score as a percentage or a probability — read the ordering, and get the magnitude from RSS.

Now the bias arithmetic, which is where oom_score_adj earns its place. The sleeping process scored 666 — the floor, because it is too small to register. Setting oom_score_adj=500 took it to 1000: the adjustment adds 500 thousandths of total RAM to the badness, which is about 1.9 GB of apparent size, and (1000 + 500) × 2/3 = 1000. That one line made an 800 kB process look bigger than the database. Setting −1000 short-circuits the whole calculation to 0: it is now exempt, and the kernel will kill anything else first, including processes a thousand times its size.

systemd-udevd shows adj=-1000 without anyone touching it, and so does PID 1. systemd protects itself and a handful of critical services by default, because a machine that OOM-kills its own init has no way back.

The practical lesson. oom_score_adj is the only real control you have over who dies — and as the arithmetic above shows, it dominates the size term completely once you use values in the hundreds. It is a single number written to a file. In a systemd unit it is OOMScoreAdjust=. Set it positive on batch jobs and caches you can afford to lose, and slightly negative — not −1000 — on the one service the host exists to run. Reserve −1000 for things that would make the machine unrecoverable, because a fully exempt process that leaks will take the whole machine down with it.

On a healthy machine you get nothing, which is the correct and common answer. On a machine that has had an incident:

[Thu Aug 14 03:14:22 2026] python3 invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
[Thu Aug 14 03:14:22 2026] Tasks state (memory values in pages):
[Thu Aug 14 03:14:22 2026] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[Thu Aug 14 03:14:22 2026] [   1198]   114  1198   478112   103240    1130496        0             0 mysqld
[Thu Aug 14 03:14:22 2026] [   2041]  1000  2041   612884   488204    4325376        0             0 python3
[Thu Aug 14 03:14:22 2026] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice,task=python3,pid=2041,uid=1000
[Thu Aug 14 03:14:22 2026] Out of memory: Killed process 2041 (python3) total-vm:2451536kB, anon-rss:1952816kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:4224kB oom_score_adj:0

What to read out of this, line by line.

python3 invoked oom-killer — this is the process that was asking for memory when the machine ran dry. It is not necessarily the one that gets killed, and confusing the two is the single most common misreading of this log. Here they happen to be the same process, but often they are not.

Tasks state — a full table of every process at the moment of the kill, with rss in pages, not kilobytes. Multiply by 4096. python3 at 488204 pages is about 1.9 GB; mysqld at 103240 pages is about 400 MB. This table is a free memory snapshot from the exact instant of the incident, which is usually the only one you get.

constraint=CONSTRAINT_NONE and global_oom — the whole machine ran out, not a cgroup. If you see CONSTRAINT_MEMCG instead, a container hit its own limit and the machine was fine. That one word changes the entire investigation, and Section C3 is about it.

task_memcg=/user.slice/user-1000.slice — which cgroup the victim was in. On a container host this names the pod.

The final line is the verdict, and it splits RSS usefully: anon-rss:1952816kB against file-rss:0kB says the victim's memory was entirely anonymous — no file backing, therefore unreclaimable, therefore the kernel had no alternative to killing it. On a machine with swap, swapents in the table above would show how much it had already pushed out.

What to do with it. The task table lets you reconstruct where the memory went without any monitoring at all. Sum the rss column, multiply by 4096, and compare to MemTotal: if the sum is far below total RAM, the memory went somewhere that is not a process — kernel slab, huge pages, or a tmpfs mount — and that changes what you go looking for.

cgroup2fs

low 0
high 0
max 0
oom 0
oom_kill 0
oom_group_kill 0

--- running a 500 MB allocation under a 100 MB limit ---
Killed
exit status: 137

--- the cgroup event counters ---
low 0
high 0
max 2841
oom 1
oom_kill 1
oom_group_kill 0

[Fri Aug 21 09:41:07 2026] Memory cgroup out of memory: Killed process 7412 (bash) total-vm:512884kB, anon-rss:99328kB, file-rss:1284kB, shmem-rss:0kB, UID:0 pgtables:296kB oom_score_adj:0
[Fri Aug 21 09:41:07 2026] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/oomtest,task_memcg=/oomtest,task=bash,pid=7412,uid=0

What to read out of this.

Exit status 137. 137 − 128 = 9, and signal 9 is SIGKILL. This is exactly the number Kubernetes reports as OOMKilled, and now you know precisely where it comes from. It is not a Kubernetes code and it is not invented by the container runtime: it is the same 128 + N convention you met in Module 02, reported here by the shell and reported by the container runtime in the Kubernetes case.

Now memory.events, which is the file to reach for during a container incident and is almost unknown:

• max 2841 — the limit was hit 2,841 times. Every one of those is a moment where the kernel had to force reclaim to keep the cgroup under its cap. A high max count with oom_kill 0 is a container that is being throttled hard but surviving — slow, and nothing else will tell you.
• oom 1 — the cgroup entered an OOM condition once.
• oom_kill 1 — one process was killed.

The kernel log line is the other half. constraint=CONSTRAINT_MEMCG and oom_memcg=/oomtest.slice say plainly: this was a cgroup limit, not the machine. Compare that with the global_oom line in Exercise C2.2. Same log format, entirely different incident, and one word apart.

Look at anon-rss:99328kB — just under the 100 MB cap, not exactly at it. That is worth understanding: memory.max bounds the whole cgroup, not one process. The head and tr in the pipeline, plus kernel memory charged to the cgroup, take up the remainder, so the victim's own resident size is always a little below the limit. A number that landed exactly on the cap would be a sign the output was made up.

And file-rss:1284kB is tiny: this victim had almost no reclaimable memory, so the kernel had nothing to take back before killing it.

If systemd-run is unavailable, do it the manual way: write the PID into /sys/fs/cgroup/oomtest/cgroup.procs before allocating. The counters and the log line are identical.

If nothing was killed and the command printed survived, the limit did not apply — check that memory.max really reads 104857600 and that the subtree controller was enabled.

It is the kernel's last resort when an allocation cannot be satisfied and reclaim has already failed. Because Linux overcommits — it hands out more memory than it has, betting most of it will never be touched — there is no way to refuse the allocation at that point, so it picks a process and sends SIGKILL.

The choice comes from oom_badness, which scores each process on resident pages plus swap pages plus page tables, then applies oom_score_adj as a bias worth one thousandth of total RAM per unit, in the range −1000 to +1000. In practice this means the biggest process is killed, not the one responsible.

The details that separate candidates: saying explicitly that the victim is chosen by size, not by blame — a small leaking script can get the database killed. Then: oom_score_adj at exactly −1000 makes a process fully exempt, systemd already sets that for PID 1, and OOMScoreAdjust= is the unit-file way to set it. And the deprecated oom_adj file was removed in Linux 3.7, so any runbook still writing to it has been doing nothing for a decade.

Two different OOM events exist and this was the cgroup one. resources.limits.memory becomes memory.max on the container's cgroup; when usage cannot be reclaimed below that, the kernel runs the OOM killer scoped to that cgroup, so the victim comes from inside the container and the node is untouched. Exit 137 is 128 + 9 — killed by SIGKILL.

To confirm it I would look at the host's kernel log for the OOM line and check constraint=: CONSTRAINT_MEMCG means the container's own limit, global_oom means the node genuinely ran out. Then memory.events for that cgroup, where oom_kill counts kills and max counts how often the limit was hit at all.

The details that separate candidates: three. The OOM record is written by the host kernel, so it never appears in container logs — which is why the pod seems to restart with no explanation. Page cache counts towards memory.max, so a burst of file I/O can push a container over a limit its own working set fits comfortably inside. And memory.high would have throttled instead of killing, but Kubernetes does not expose it, so the only lever most teams have is sizing the hard limit with headroom.

Overcommit is the kernel granting more address space than it could back with RAM plus swap, on the reasonable bet that most of it will never be touched. vm.overcommit_memory controls it: mode 0 is a permissive heuristic and the default, mode 1 never refuses, mode 2 is strict accounting against CommitLimit — swap plus vm.overcommit_ratio percent of RAM, default 50%.

I would not turn it off casually. On a typical machine Committed_AS already exceeds what strict mode would allow, by a wide margin, while the machine runs perfectly. Mode 2 with the default ratio caps a swapless host at half its RAM, and every runtime that reserves large address space up front — Go, the JVM — fails at start-up.

The details that separate candidates: connecting overcommit to the OOM killer as cause and effect: the OOM killer exists because of overcommit. Mode 2 replaces an unpredictable SIGKILL later with an honest ENOMEM at allocation time, which is the right trade only where a failed allocation is genuinely better than a kill — some embedded and financial systems. And if you do set mode 2, you must set overcommit_ratio or vm.overcommit_kbytes deliberately, because the default is far too conservative to be used as-is.

MemTotal:        3936256 kB
MemFree:          198432 kB
MemAvailable:    1204188 kB
Buffers:           81920 kB
Cached:          1841204 kB
AnonPages:       1284116 kB
Shmem:            892416 kB
Slab:             412884 kB
SReclaimable:     298104 kB
SUnreclaim:       114780 kB
KernelStack:       12208 kB
PageTables:        28416 kB

sum of PSS   :   1318.4 MiB
MemTotal     :   3844.0 MiB

 Active / Total Objects (% used)    : 558104 / 560824 (99.5%)
 Active / Total Slabs (% used)      : 30511 / 30511 (100.0%)
 Active / Total Caches (% used)     : 118 / 208 (56.7%)
 Active / Total Size (% used)       : 402884.20K / 412884.00K (97.6%)
 Minimum / Average / Maximum Object : 0.01K / 0.74K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
421204 418992  99%    0.19K  20057       21     80228K dentry
 98412  97108  98%    0.58K   7570       13     57080K inode_cache
 41208  41208 100%    1.06K   2884       14     43712K ext4_inode_cache

TARGET           SIZE  USED USE%
/dev/shm         1.9G  872M  45%
/run             384M   12M   3%
/run/lock          5M     0   0%

-rw-r--r-- 1 postgres postgres 872M Aug 21 04:02 /dev/shm/PostgreSQL.184920114

What to read out of this.

Do the arithmetic first. Processes map 1318 MiB of a 3844 MiB machine. Where is the other 2526 MiB?

Cached: 1841204 kB is 1798 MiB, so most of it is page cache — normal and mostly reclaimable. But look at the next line: Shmem: 892416 kB, 871 MiB, and Shmem is a subset of Cached. So of that 1798 MiB of "cache", 871 MiB cannot be dropped at all. It has no file on disk to reload from.

The findmnt output finds it immediately: /dev/shm is 45% full with an 872 MB PostgreSQL shared-memory segment. That is not a leak and not a problem — it is a PostgreSQL dynamic shared-memory segment, the kind parallel query allocates — but it is 871 MiB of RAM that every "how much cache can I reclaim" estimate must exclude. MemAvailable at 1204 MiB already excludes it, which is one more reason to use that number rather than adding columns yourself.

Slab: 412884 kB is 403 MiB of kernel data structures, and slabtop says most of it is dentry and inode_cache — the kernel remembering filenames and file metadata. On a machine that walks large directory trees (a backup job, a CI runner, a container host pulling images) this can reach several gigabytes. SReclaimable: 298104 of it can be given back; SUnreclaim: 114780 cannot.

PageTables: 28416 kB is the VmPTE cost from Module 08, summed across every process. Small here, but on a host running many large processes it reaches hundreds of megabytes and appears in no per-process metric.

Do the arithmetic yourself, because it is the point of the section. AnonPages 1254 MiB, plus Cached 1798 MiB, plus Buffers 80 MiB, plus Slab 403 MiB, plus PageTables and KernelStack 40 MiB, comes to 3575 MiB. Add the 194 MiB free and you have 3769 of 3844 — within about 75 MiB, which is kernel text, per-CPU areas and vmalloc that /proc/meminfo does not itemise. That accounts for the machine.

Note that Shmem is not added separately: it is already inside Cached, and it is excluded from AnonPages, so it is counted exactly once. Adding it again is the most common way to get this sum wrong.

If your own arithmetic leaves a large unexplained gap, look at huge pages next — HugePages_Total is reserved memory that appears in none of these lines and is excluded from MemAvailable too.

I start by checking whether it is real. free -m and read available, not free — most "high memory" reports are the page cache doing its job. Then /proc/pressure/memory, because a high level with no pressure is not an incident.

If it is real, I find where the memory went. Sum Pss from smaps_rollup across processes; if that accounts for most of MemTotal, it is a process and I look at its growth rate over minutes rather than its level. If it does not, the memory is in the kernel or in tmpfs, and I check Shmem, Slab/slabtop, and findmnt -t tmpfs. Finally I check the kernel log for OOM kills, and read constraint= to see whether it was a container limit or the node.

The details that separate candidates: separating level from rate — a process steady at 4 GB is doing its job; one at 800 MB climbing 50 MB an hour is the problem, and it is the smaller number. And knowing the three places memory hides outside the process list, because "no process is large but the machine is full" is the case that defeats most people. pgsteal_direct from /proc/vmstat is the early warning I would also graph.

Four candidates, in the order I would check them. Page cache — Cached, expected and mostly harmless. tmpfs and shared memory — Shmem, which is counted inside Cached but cannot be reclaimed because there is no disk copy; findmnt -t tmpfs and ls -lh /dev/shm find it. Slab — kernel structures, mostly dentry and inode_cache on machines doing heavy filesystem work; slabtop -s c ranks them, and SUnreclaim is the part you will not get back. Kernel overhead — PageTables summed across processes, KernelStack, and reserved huge pages, which appear in none of the usual totals.

The details that separate candidates: knowing that Shmem is a subset of Cached, so a naive "cache is reclaimable" assumption overstates available memory by exactly that amount — and that MemAvailable already accounts for it, which is the practical reason to trust it over your own arithmetic. Naming dentry and inode_cache growth as the classic cause on container hosts and CI runners shows you have actually met this one.

Run, in order: free -m during the window → grep MemAvailable /proc/meminfo → cat /proc/pressure/memory → check what runs at 02:00.

What you will find. Almost certainly the nightly backup. Reading every database file pulls the whole dataset through the page cache, so buff/cache fills and MemFree collapses. MemAvailable will barely move, and /proc/pressure/memory will show some near zero — nothing is being harmed.

What you tell them: the threshold is not the problem, the metric is. Raising 96% to 98% buys a few weeks and then the same page returns, and in the meantime it makes the alert less able to catch a real shortage. The fix is to alert on MemAvailable / MemTotal instead, which will sit comfortably high through the backup, and add memory some avg60 from PSI as the leading indicator.

The evidence that settles the argument is that the two numbers disagree: at the moment free says 96% used, available says perhaps 75% free. Show that side by side once and the conversation is over.

One thing worth checking before you close it. If MemAvailable does drop hard during the window, the backup is competing with the database for cache and the database's morning performance will suffer from a cold cache. That is a real problem with a real fix — posix_fadvise(POSIX_FADV_DONTNEED) in the backup tool, or nocache-style wrappers — and it is worth ten minutes to rule in or out.

The clue is the gap. Heap is 1.4 GB, the limit is 3 GB, and it is still being killed. So the memory is not in the heap — and a JVM has a great deal of memory that is not heap: metaspace, thread stacks, code cache, direct byte buffers, and any native libraries.

Run, in order, on the node:

sudo dmesg -T | grep -A3 'Killed process'          # constraint=? which task?
cat /sys/fs/cgroup/<pod path>/memory.events        # oom_kill, and max
cat /sys/fs/cgroup/<pod path>/memory.stat          # anon vs file vs kernel
cat /sys/fs/cgroup/<pod path>/memory.current

What confirms what. constraint=CONSTRAINT_MEMCG confirms it is the pod's own limit rather than the node. Then memory.stat splits the usage: a large file figure means the container's page cache is being charged to it — very common in services that write logs or read data files — and that memory was reclaimable, so the kill was avoidable. A large anon figure beyond the heap points at JVM off-heap memory, and the next step is -XX:NativeMemoryTracking.

Is raising the limit to 6 GB right? Only as a stopgap, and only after you know which of those two it is. If it is page cache being charged to the cgroup, doubling the limit wastes 3 GB per replica across the fleet to work around an accounting artefact. If it is genuine off-heap growth, doubling the limit doubles the time between restarts and fixes nothing.

The detail worth raising with the team: memory.events's max counter. If it is large while oom_kill is small, the container spends much of its life being aggressively reclaimed right at the limit — slow, and invisible in every dashboard they currently have.

Read the symptom shape first. Fast for ten minutes after a restart, then degrading, with nothing saturated — that is a working set growing past what RAM can hold. The restart resets it, which is why the restart appears to help and why it keeps coming back.

free -m showing 60 MB free and 3.2 GB cache is not the evidence either way; that is normal. The evidence is pressure.

Run, in order:

cat /proc/pressure/memory                    # some and full, avg60
grep -E 'pgsteal_direct|pgscan_direct' /proc/vmstat   # twice, subtract
grep MemAvailable /proc/meminfo
ps -eo pid,rss,comm --sort=-rss | head -5    # and again 5 minutes later
vmstat 1 5                                   # si/so if swap exists

What confirms it. memory some avg60 well above zero, full climbing, and pgsteal_direct increasing between two readings — meaning processes are being made to do reclaim themselves rather than kswapd keeping up. That combination is thrashing, and no level-based metric would have shown it: utilisation reads 100% both when the machine is healthy and when it is in this state.

Two readings of the process list five minutes apart separate the two possible causes. If one process's RSS is climbing steadily, it is a leak or an unbounded cache in the application and the fix is there. If every process is stable and the machine is simply too small for the load it now carries, the fix is capacity.

What to tell them about the restarts. Each restart destroys the evidence and buys ten minutes. Ask them to capture /proc/meminfo, /proc/pressure/memory and the sorted process list before the next restart — one second of work, and it is the difference between diagnosing this today and having the same conversation next week.

Processes account for 1.1 GB of 16 GB, so 15 GB is somewhere else. Four places, and the commands take a minute:

grep -E '^(MemTotal|MemAvailable|Cached|Shmem|Slab|SReclaimable|SUnreclaim|PageTables|KernelStack|HugePages_Total|Hugepagesize):' /proc/meminfo
findmnt -t tmpfs -o TARGET,SIZE,USED,USE%
sudo slabtop -o -s c | head -10
ls -lh /dev/shm/

Read them in this order.

If Cached is large and Shmem is small, it is ordinary page cache and there is no problem — check MemAvailable, which will be high, and the alert is once again measuring the wrong thing.

If Shmem is large, it is tmpfs: files living in RAM with no disk copy, so none of it is reclaimable even though it is counted inside Cached. findmnt finds the mount. The usual culprits are /dev/shm on a container host where an image requested a large --shm-size, and /run on a host where something is writing logs or state into it.

If Slab is large, slabtop -s c will show dentry and inode_cache at the top. That is the kernel remembering filenames and metadata, and it grows on hosts that walk large directory trees — container image pulls, CI runners, backup jobs. SReclaimable comes back under pressure; SUnreclaim does not.

If HugePages_Total is non-zero, that memory is reserved and appears in none of the other lines, including MemAvailable. A host configured with 12 GB of huge pages for a database looks exactly like this ticket.

The reason nothing was OOM-killed is the tell that this is not a shortage at all in most versions of this ticket: the kernel was never unable to allocate. Check MemAvailable before doing anything else — if it is healthy, this is an alerting defect, not a memory incident.