🧠Module 08 — Virtual Memory, Paging & Page Faults

PID 4412:
7ffc6d3a1000-7ffc6d3c2000 rw-p 00000000 00:00 0                          [stack]
PID 4413:
7ffe91b4c000-7ffe91b6d000 rw-p 00000000 00:00 0                          [stack]
PID 4412:
560f2a1c1000-560f2a1c3000 r--p 00000000 08:01 1049094                    /usr/bin/sleep
PID 4413:
55d8c9e77000-55d8c9e79000 r--p 00000000 08:01 1049094                    /usr/bin/sleep

What to read out of this.

Both stacks live near the top of a very large number space — around 0x7ffc…, which is about 140 terabytes up. Your machine does not have 140 TB of RAM. These are not real memory addresses. They are positions in an imaginary space that only this process can see.

The two stacks are at nearly but not exactly the same place. That small difference is deliberate randomisation (called ASLR), added so attackers cannot predict where things are. Without it both would sit at an identical address — and that would be perfectly fine, because the two processes cannot see each other's memory anyway.

The last two lines are the interesting ones. Both processes loaded the same file: /usr/bin/sleep, inode 1049094 on device 08:01 — literally the same bytes on disk. Yet each sees it at a different virtual address. The kernel has almost certainly loaded those bytes into RAM once and pointed both processes' translation tables at the same physical copy.

That is the whole idea in one screen: the address a process uses is a label, not a location, and the kernel is free to point two different labels at one real thing.

Virtual promised: 38.4 GiB
Actually resident: 1.3 GiB
Installed RAM:    3.8 GiB

What to read out of this.

The processes on this small machine have collectively asked for 38.4 GiB of address space on a box with 3.8 GiB of RAM — ten times more than exists. Nothing is broken. Nothing is even close to broken.

That is because asking for address space is not the same as using memory. A process can say "reserve me a 1 GiB region" and the kernel simply writes that down. Not one byte of RAM is committed until the process actually touches an address inside that region. Section B2 covers the moment it does.

The resident figure — 1.3 GiB — is the honest one, and even that is an overcount, because shared pages like the C library are counted once per process. Section C2 fixes that too.

The interview point buried here: VSZ is close to meaningless as a measure of memory use. A process with a 20 GiB VSZ may be using 40 MiB. People page a team over VSZ graphs constantly.

4096

5581b4a2c000-5581b4a5a000 r--p 00000000 08:01 1052097    /usr/bin/bash
5581b4a5a000-5581b4b3c000 r-xp 0002e000 08:01 1052097    /usr/bin/bash
5581b4b3c000-5581b4b73000 r--p 00110000 08:01 1052097    /usr/bin/bash
5581b4b74000-5581b4b78000 r--p 00147000 08:01 1052097    /usr/bin/bash
5581b4b78000-5581b4b81000 rw-p 0014b000 08:01 1052097    /usr/bin/bash

--- this shell ---
VmSize:	   10756 kB
VmRSS:	    5504 kB
VmPTE:	      68 kB

--- the biggest process on the machine ---
   1198 mysqld
VmSize:	 1912304 kB
VmRSS:	  412960 kB
VmPTE:	    1204 kB

What to read out of this.

4096 is the page size. Now look at the map addresses: every single one ends in 000. 0x5581b4a2c000 to 0x5581b4a5a000 is exactly 46 pages. Nothing in a memory map is ever unaligned, because translation cannot describe a fraction of a page.

Notice the same file appears five times with different permissions. r--p is the ELF header and the dynamic-linking tables; r-xp is the executable code; the next r--p is .rodata, the constants; the third r--p is the RELRO region, which the loader makes read-only again once it has finished relocating it; and rw-p is the writable variables. The kernel maps each part with the least permission it needs. This is protection at page granularity, and it is why a bug that writes to a code address gets killed instead of rewriting the program.

Now the page-table cost. The shell maps 10.7 MB and its page table costs 68 kB. mysqld maps 1.9 GB and pays 1.2 MB. That is 0.6% for the shell but only 0.06% for mysqld, and the difference is worth understanding: one page-table page describes 2 MiB of address space whether you use all of it or not, so a small, scattered process pays proportionally more. A densely mapped region approaches the floor of 8 bytes per 4 KiB page, which is 0.2%. Either way it is real memory, it is per process, and it is invisible in RSS.

Where this bites in production: a process that maps 100 GB pays around 200 MB just for page tables, and a hundred processes each mapping the same shared region each pay their own copy of the table. This is one of the reasons huge pages exist, which is Part D.

On a physical machine you get real numbers:

Performance counter stats for 'bash -c ...':

    1,842,551,203      dTLB-loads
        2,104,887      dTLB-load-misses          #    0.11% of all dTLB cache accesses

      1.284730151 seconds time elapsed

On a cloud VM you will almost certainly get this instead:

Performance counter stats for 'bash -c ...':

  <not supported>      dTLB-loads
  <not supported>      dTLB-load-misses

      1.402318774 seconds time elapsed

What to read out of the failure — this is the useful half.

<not supported> does not mean perf is broken or that you typed something wrong. It means the hardware performance counters are not exposed to the guest. Those counters live in the physical CPU, and most hypervisors do not virtualise them, because doing so would let one tenant observe the behaviour of another. AWS, GCP and Azure general-purpose instances all behave this way; bare-metal instance types generally do not.

This catches people out badly. A performance investigation that depends on perf stat -e cache-misses or dTLB-load-misses cannot be done on a normal cloud VM at all. Knowing that before you promise someone a hardware-level analysis is worth a lot.

One failure that is not this one. If you get "Access to performance monitoring and observability operations is limited" instead of <not supported>, that is the kernel.perf_event_paranoid sysctl refusing you, not the hypervisor. Ubuntu ships a restrictive value. Check it with sysctl kernel.perf_event_paranoid before concluding anything about the hardware — the two failures look similar and mean completely different things.

What still works in a VM: software events, which the kernel counts itself.

$ perf stat -e task-clock,context-switches,page-faults,cpu-migrations -- sleep 1

              1.42 msec task-clock       #    0.001 CPUs utilized
                 2      context-switches #    1.408 K/sec
               248      page-faults      #  174.648 K/sec
                 1      cpu-migrations   #    0.704 K/sec

page-faults is the one this module is about, and it is counted by the kernel — so it works everywhere, VM or not. Part B is built on it.

total regions: 27
file-backed:   19
anonymous:     8

563a1f2d1000-563a1f302000 rw-p 00000000 00:00 0     [heap]
7f16c1d9a000-7f16c1d9e000 r--p 00000000 00:00 0     [vvar]
7f16c1d9e000-7f16c1da0000 r-xp 00000000 00:00 0     [vdso]
7ffd4c9a1000-7ffd4c9c2000 rw-p 00000000 00:00 0     [stack]

      1540 KiB  r-xp  /usr/lib/x86_64-linux-gnu/libc.so.6
       904 KiB  r-xp  /usr/bin/bash
       320 KiB  r--p  /usr/lib/x86_64-linux-gnu/libc.so.6
       196 KiB  rw-p  [heap]
       132 KiB  rw-p  [stack]

What to read out of this.

Twenty-seven separate regions for a shell that is doing nothing. Nineteen come from files — the shell binary, the C library, the terminal library, the dynamic loader — and eight are anonymous.

The named regions are worth knowing by sight. [heap] is where small allocations grow. [stack] is the call stack, and it is only 132 KiB right now even though the limit is usually 8 MiB, because the stack is grown on demand and this shell has not needed more. [vdso] is the trick from Module 01: a tiny page of kernel code mapped into every process so calls like gettimeofday avoid the syscall boundary entirely. [vvar] is the data it reads.

Now the entry people find surprising: the largest single mapping is not the shell at all. It is 1540 KiB of the C library's executable code, bigger than bash itself — and it is shared, so the machine holds one copy of it no matter how many shells you open.

You will also meet mappings with permissions ---p, meaning no read, no write and no execute. They are not gaps left by the loader; on a modern glibc the library's five regions are laid out contiguously with no hole between them. Where ---p regions do appear — heavily threaded programs, and Go or Java runtimes — they are reservations: address space claimed up front so that later allocations can be contiguous, with no physical memory behind them at all. They cost nothing in RAM and everything in VSZ, and Section C1 takes one apart.

If you run this twice you will see every address change. That is ASLR again.

Virtual memory is a layer of indirection between the addresses a program uses and the real locations in RAM. Every process gets its own private address space; hardware in the CPU translates each address through a per-process page table on every access.

It buys three things at once. Protection — a process cannot even name memory outside its own table, so it cannot reach another process's data. Relocation — programs can be compiled to fixed addresses and still run anywhere, because the addresses are not real. Overcommitment — address space can be handed out without RAM being committed, so the sum of what processes have asked for can far exceed installed memory.

The details that separate candidates: most answers stop at "it lets you use disk as extra RAM", which is swap — one consequence, and not the important one. The stronger framing is that virtual memory is primarily an isolation and abstraction mechanism, and that a machine with swap disabled entirely still depends on it completely. The other detail worth adding is that translation happens in hardware on every access, which is why the design is shaped around a small cache (the TLB) rather than around software lookups.

Memory is divided into fixed-size pages on the virtual side and equally sized frames on the physical side — 4 KiB on x86-64. Each process has a page table, a multi-level tree mapping its pages to frames; only the branches in use exist, so the table stays small.

On every access the MMU splits the address: the high bits select the page, the low 12 bits are an offset that passes through unchanged. Walking a four-level table would cost four extra memory reads per access, so the CPU caches recent translations in the TLB. A hit is effectively free; a miss triggers the walk.

The details that separate candidates: being able to say roughly how much memory the TLB covers — a couple of thousand entries at 4 KiB each is only single-digit megabytes — and therefore why a large, randomly accessed working set thrashes the TLB regardless of how much RAM you have. That naturally leads to huge pages as the fix. It is also worth noting that the page table itself consumes RAM (VmPTE in /proc/PID/status), which most people have never looked at.

A virtual address is what the process sees and uses; it is meaningful only inside that process's address space. A physical address is an actual location in the RAM chips. The MMU converts one to the other using the page table, per process, on every access.

The same virtual address in two processes normally refers to two different physical addresses. The reverse also happens: two different virtual addresses, in the same or different processes, can refer to one physical address — that is shared memory, and it is how a single copy of the C library serves every process on the machine.

The details that separate candidates: stating that the mapping is many-to-one in both directions and not stable over time — the kernel can move a page to a different frame, and a virtual page may have no frame behind it at all until it is touched. Adding that unprivileged processes deliberately cannot see physical addresses (/proc/PID/pagemap has been masked for unprivileged users since 2015, because exposing them enabled Rowhammer-style attacks) shows you have actually gone looking rather than recited a textbook.

    167      0   3244
  73503      0  52212
50000000 / 4096 = 12207
  73527      0   3380

What to read out of this.

Minor faults went from 167 to 73,503 — an increase of about 73,300. The arithmetic says 50 MB is 12,207 pages, so the fault count is roughly six times the number of pages involved.

That gap is the lesson. Demand paging guarantees you at least one fault per page. It does not promise only one. bash builds the string by growing a buffer and copying it into a larger one over and over, so the same 50 MB of data gets written into fresh pages several times over, and each fresh page faults again.

That is demand paging happening in front of you. Nobody handed bash 50 MB. It received tens of thousands of separate small events, each one triggered by touching a page that had no frame behind it yet.

maj_flt stayed at 0 throughout. Not one byte came off a disk — anonymous memory is filled with zeroes the kernel produces on the spot, so there was nothing to read. A major fault means reading from storage, and here there was nothing on storage to read.

Now the last line. unset BIG freed the memory and RSS fell straight back to 3,380 kB, essentially where it started — but the fault counter did not go down, it went slightly up. min_flt is a lifetime total, not a current level. Like %CPU in ps from Module 07, it tells you about the past. To use it you take two readings and subtract.

RSS returned cleanly here because an allocation this large is served by its own dedicated mapping, which is handed straight back to the kernel when it is freed. Smaller allocations usually do not behave this way — the allocator keeps those pages for reuse, so RSS stays high after the program has freed everything. That is the single most common reason a "memory leak" ticket turns out not to be a leak, and it is worth knowing that the behaviour depends on the size of the allocation.

parent is PID 4881
   4881  53612 bash

   4881   4102  53612 bash
   4914   4881  52104 bash

sum of RSS: 103 MiB

What to read out of this.

Both processes report about 52 MB, and the sum says 103 MiB — on a machine that never spent more than 53 MB. Not one page was copied. The child's page table points at the parent's existing frames, and RSS counts a frame in full for every process that maps it.

This is the single most important thing to understand about RSS, and it is why the number cannot be added up. It is not lying: the child really can reach 52 MB of memory. It simply is not 52 MB of additional memory.

Worth trying as a variation: replace ( sleep 60; : ) & with bash -c 'sleep 60' & and run it again. The child's RSS drops to about 1.7 MB and comm reads sleep, not bash — because a single simple command is execed in place, which throws the inherited image away entirely. That is the common case in real systems: almost every fork is followed immediately by exec, so copy-on-write ends up copying essentially nothing.

That double-counting is not a bug in ps. RSS honestly answers "how many frames is this process mapping". It just cannot answer "how much memory would I get back if I killed it", which is the question people are actually asking. Section C2 is the fix.

If your run shows a child RSS of only a megabyte or two, the shell exec-ed the child anyway — check that the subshell really contains two commands. Some shells optimise more aggressively than bash.

-rwxr-xr-x 1 root root 5.9M Mar 18 09:12 /usr/bin/python3

  Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.34
  Maximum resident set size (kbytes): 9216
  Major (requiring I/O) page faults: 148
  Minor (reclaiming a frame) page faults: 1032

  Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.02
  Maximum resident set size (kbytes): 9216
  Major (requiring I/O) page faults: 0
  Minor (reclaiming a frame) page faults: 1179

What to read out of this.

The first run took 0.34 seconds; the second took 0.02 seconds. Seventeen times faster, running exactly the same program with exactly the same arguments on the same machine.

The only difference in the numbers is Major page faults: 148 versus 0. One hundred and forty-eight times, the process asked for a page, the kernel found it had to go to the disk, and the process was put to sleep until it arrived. That accounts for essentially the whole 320 ms.

Look at the minor faults: they went up slightly on the fast run, 1032 to 1179. That is the point. Minor faults are not the expensive thing. You had more of them on the run that was seventeen times faster.

Maximum resident set size is identical at 9216 kB both times, which tells you something else useful: the amount of memory used was never the issue. Same memory, same work, wildly different time — all of it explained by where the memory had to come from.

The production translation. A service that is slow only after a deploy, only after a restart, or only on the first request after being idle, and whose maj_flt is non-zero, is not slow because of CPU or because of a memory shortage. It is waiting for disk to hand it its own code.

Segmentation fault
exit status: 139

[ 8241.559102] bash[5127]: segfault at 7ffc2c1a1ff8 ip 000055f0a2b114c1 sp 00007ffc2c1a2000 error 6 in bash[55f0a2ad2000+d6000]
[ 8241.559118] Code: 48 89 e5 48 83 ec 20 48 89 7d e8 ...

8192

What to read out of this.

Exit status 139. From Module 02: a process killed by signal N reports 128 + N. 139 − 128 = 11, and signal 11 is SIGSEGV. You did not need the message to know what happened.

Now the kernel log line, which is the useful part. segfault at 7ffc2c1a1ff8 is the address the program tried to touch. sp 00007ffc2c1a2000 is where the stack pointer was. The faulting address is 8 bytes below the stack pointer — the function tried to push one more value and went off the bottom of the stack.

ulimit -s says 8192, which is 8 MiB. The stack grows downward on demand: each time it needs another page the access faults, the kernel notices the address is just below the existing stack mapping and quietly extends it. That is a normal minor fault and happens constantly. But the kernel refuses once the mapping would exceed RLIMIT_STACK, and the refused access is delivered as SIGSEGV. The limit is what stops the recursion, not a wall in memory.

Separately, the kernel keeps a guard gap below every stack — 1 MiB by default since Linux 4.12, tunable with the stack_guard_gap= boot parameter. That gap exists so a runaway stack cannot silently grow into an unrelated mapping and start corrupting it. It is a safety net against a different bug, not the mechanism you just triggered.

error 6 is a bitmask: bit 1 means it was a write, bit 2 means it happened in user mode. So: a user-mode write to an address with no mapping. That single number tells you read-versus-write, which is often the first thing you want to know.

Why this matters beyond the demo. Unbounded recursion is one of the two classic causes of a stack overflow; the other is a large array declared as a local variable. Both produce exactly this signature, and both are commonly misreported as "the machine ran out of memory" when the machine had gigabytes free. A segfault is not an out-of-memory condition. It is an access to an address that is not mapped, and the two have completely different fixes.

It is a hardware trap raised by the MMU when a virtual address has no valid page-table entry. Control passes to the kernel, which looks at the address and the process's mappings and decides one of three things: the mapping is valid and the data is already in RAM, so it fills in the entry and returns (minor); the mapping is valid but the data must be read from disk, so it starts the I/O and sleeps the process (major); or the address is not part of any valid mapping, so it delivers SIGSEGV. In the first two cases the kernel then re-runs the faulting instruction and the program never learns anything happened.

The details that separate candidates: leading with "a page fault is not an error" and being able to say that a healthy machine does millions per second. Most candidates describe it as a failure or conflate it with a segfault. The other detail is that the program is not notified and the instruction is retried — that is precisely what makes lazy allocation possible, and it is the reason a process can be handed a 4 GB mapping instantly.

A minor fault is resolved without touching a disk: the page is already in RAM (shared with another process, or already held from an earlier read), or it needs no data at all because it is a fresh zero page. It costs microseconds. A major fault requires disk I/O, so the process blocks for milliseconds — three to four orders of magnitude worse.

Alert on major faults. Total page faults are dominated by minor ones and track normal activity, so a "page faults per second" alert fires on healthy workloads and stays quiet during real problems.

The details that separate candidates: being able to give the ratio — the same program can run seventeen times slower purely because 148 of its faults were major — and knowing where to read the two numbers separately (min_flt and maj_flt in ps, or /usr/bin/time -v). It is also worth saying that major faults are expected immediately after a deploy or a restart, so the alert needs to be on a sustained rate rather than a spike, or it will page you on every rollout.

On fork, instead of duplicating the parent's memory the kernel points the child's page table at the parent's existing frames and marks every writable page read-only in both processes. The first write by either one raises a protection fault; the kernel recognises it as a copy-on-write fault, makes a private copy of that single page for the writer, marks it writable, and restarts the instruction. Only pages actually written are ever duplicated.

It matters because the overwhelmingly common pattern is fork followed immediately by exec, which discards the inherited image entirely. Without COW, every command you type would first copy the whole shell.

The details that separate candidates: three things. First, the parent is affected too — its own writes now fault until the sharing is broken, so forking briefly slows the parent down. Second, fork is cheap in memory but not free in time, because the kernel still walks and marks the page tables and allocates the child's own tables; a database process with tens of gigabytes mapped shows a measurable pause. Third, this is exactly why summing RSS across processes overstates real usage — shared frames are counted once per process.

The program accessed an address that had no valid mapping, or accessed it in a way its mapping forbids — writing to a read-only page, or executing a page that is not executable. The MMU raised a fault, the kernel looked for a mapping covering that address, found none that permitted the access, and delivered SIGSEGV. The default disposition kills the process, and the shell reports exit status 128 + 11 = 139.

To investigate I would read the kernel log: dmesg records the process name, PID, the faulting address, the instruction pointer, the stack pointer and an error bitmask that says read-or-write and user-or-kernel mode. Comparing the faulting address to the stack pointer distinguishes a stack overflow from a wild pointer straight away.

The details that separate candidates: stating plainly that a segfault is not an out-of-memory condition — the two get conflated constantly, and the fixes are unrelated. Also: a stack overflow from deep recursion produces a faulting address immediately below the stack pointer, because the stack grows on demand up to ulimit -s and then meets an unmapped guard page. And knowing that segfaults inside a container appear in the host's kernel log, not the container's, which is why they so often go unseen.

1198     1867.5 MB vsz    403.3 MB rss  ratio   4.6x  mysqld
 842     1122.9 MB vsz     18.7 MB rss  ratio  60.0x  containerd
 901      998.4 MB vsz     32.1 MB rss  ratio  31.1x  dockerd
 455      412.6 MB vsz     11.2 MB rss  ratio  36.8x  snapd
 688       92.1 MB vsz     14.9 MB rss  ratio   6.2x  sshd

1024.0 MB  ---p  
 256.0 MB  rw-p  
  64.0 MB  rw-p  
  16.0 MB  r--p  /usr/lib/mysql/mysqld

What to read out of this.

containerd reports 1.1 GB of VSZ and 18.7 MB of RSS — a ratio of sixty. If you graphed VSZ you would conclude it was the biggest memory consumer on the box. It is using less memory than sshd.

The reason is in the second output. The single largest mapping is 1 GB with permissions ---p — no read, no write, no execute, and no file behind it. That is a reservation: the program asked the kernel to set aside a contiguous gigabyte of address space so it can hand out pieces later without them being fragmented. Not one page of RAM is involved. This is standard behaviour for Go runtimes and for glibc's thread arenas, which is why Go and Java processes are the usual suspects in "this process is using 1 GB" tickets that turn out to be nothing.

Compare mysqld: ratio 4.6, and its large mappings are rw-p — actually usable memory it has reserved and partly touched. A ratio near 1 means a process whose address space is nearly all live; a ratio in the tens means most of it is reservation.

The rule to take away: never report VSZ as memory usage, and be suspicious of any dashboard whose "memory" panel is built from it. Several well-known monitoring agents got this wrong for years.

   1198 412960 mysqld
   2041 118204 python3
   2042 116888 python3
   2043 115932 python3
    901  32872 dockerd

Rss:              412960 kB
Pss:              407742 kB
Private_Clean:      1204 kB
Private_Dirty:    401320 kB
Shared_Clean:      10436 kB
Shared_Dirty:          0 kB

--- whole machine ---
sum of RSS :   1284.6 MiB
sum of PSS :    968.3 MiB
installed  :   3936.0 MiB

What to read out of this.

Start with the three python3 processes at ~117 MB each. Summed, that is 351 MB. They are almost certainly workers forked from one parent, so most of those pages are the same physical frames counted three times. RSS cannot tell you that; the sum at the bottom can.

Sum of RSS is 1284.6 MiB. Sum of PSS is 968.3 MiB. The difference — 316 MiB, about 25% — is memory that RSS counted more than once. On a machine running many copies of the same thing, that gap gets much wider; on a container host it is routinely 40% or more.

Now the mysqld breakdown, which shows the other half of the picture. Rss 412960 and Pss 407742 are nearly equal, so mysqld shares almost nothing — it is not one of many copies of anything. Private_Dirty is 401320 kB, and that is the number that matters: this memory has been written to, so it belongs to mysqld alone, and the kernel cannot simply drop it and read it back from a file later, because there is no file behind it. Reclaiming it would mean writing the contents somewhere else first — which is Module 09.

Compare Shared_Clean: 10436 kB — that is the code of mysqld and its libraries. Clean and file-backed, so the kernel can discard it any time and read it back from disk if needed. It costs almost nothing to hold.

The line to take into an interview: Private_Dirty is the memory that genuinely cannot be recovered without killing the process. Everything else has an escape route.

Finally, sum of PSS at 968 MiB against 3936 MiB installed means the processes on this machine are using about a quarter of the RAM. What the rest is doing is Module 09.

VSZ is the total virtual address space a process has mapped, including reservations that have never been touched, file mappings still entirely on disk, and permission-less gaps left by the dynamic loader. RSS is the number of physical frames the process currently has resident, in kilobytes — real memory, but with every shared frame counted in full for every process that maps it.

So VSZ overstates because most of it may not exist, and RSS overstates because shared pages are counted repeatedly.

The details that separate candidates: giving a concrete case. A Go or containerd process routinely shows 1 GB of VSZ against 19 MB of RSS, because the runtime reserves a large contiguous region up front — a ratio of sixty, and zero of it is memory. The second detail is that RSS is not summable: add it across processes and you can exceed installed RAM, which is why capacity work needs PSS instead.

It depends which question is being asked, and I would pin that down first. /proc/PID/smaps_rollup gives all the answers in one cheap read. Pss is the process's fair share, counting each shared frame divided by the number of processes using it — this is the number that sums correctly across the machine. Private_Dirty is memory that has been written and belongs to this process alone, which is what would actually be freed by killing it. Rss is what ps and top show and is the least useful of the three for anything but tracking one process's growth over time.

The details that separate candidates: knowing that Private_Dirty is usually much smaller than RSS for a worker in a pool, so killing it frees far less than the RSS figure suggests — a mistake teams make repeatedly during memory incidents. Also worth mentioning: read smaps_rollup, not smaps, for routine collection, because parsing full smaps on a process with tens of thousands of mappings is slow enough to disturb the process being measured.

First establish that it really is. ps sorted by RSS finds the candidate, then smaps_rollup tells me whether the memory is private and dirty (genuinely held by this process) or shared and clean (code and libraries, cheap and reclaimable). If it is mostly shared, there is no problem to solve on this process.

If it is private and dirty and growing, I take two readings of smaps_rollup a few minutes apart to get a rate rather than a level, and check /proc/PID/maps to see which regions are growing — an expanding [heap] points at the allocator, a growing count of anonymous mappings points at repeated mmap without munmap, and a growing number of file mappings often means leaked file descriptors from Module 03.

The details that separate candidates: separating growth from level. A process sitting at a steady 4 GB is doing what it was configured to do; a process at 800 MB climbing 50 MB an hour is the actual problem, and it is the smaller number. Also: checking whether the process is inside a cgroup with a memory limit, because the behaviour and the correct fix are entirely different when the limit is per container rather than per machine.

always [madvise] never
always defer defer+madvise [madvise] never

AnonHugePages:     94208 kB
ShmemHugePages:        0 kB
HugePages_Total:       0
Hugepagesize:       2048 kB

HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0

40960 kB  /proc/1198/smaps_rollup
22528 kB  /proc/2041/smaps_rollup

What to read out of this.

always [madvise] never — the brackets mark the active setting. madvise is the safe middle ground: the kernel only uses huge pages for regions where the application explicitly asked for them via madvise(MADV_HUGEPAGE). Programs that have not asked are unaffected, so you get the benefit where it was requested and none of the stalls where it was not. It is the Debian and Ubuntu default — but not the RHEL family's, which ships always.

If you see [always], every large anonymous mapping is a candidate, whether the program wanted it or not. That is the setting associated with the latency stalls in the warning above.

The defrag line is the one people forget to check, and it is the one that causes the stalls. [madvise] here means the kernel will only do synchronous compaction — making a process wait — for regions that asked for huge pages. [always] there means any process can be stopped mid-fault while the kernel shuffles memory. enabled and defrag are separate knobs and both matter.

AnonHugePages: 94208 kB is 92 MB currently backed by 2 MiB pages — that is 46 huge pages doing the work of 23,552 normal ones, and 23,506 TLB entries not being consumed.

HugePages_Total: 0 means no explicit huge pages are reserved. That is normal unless a database has been deliberately configured. Hugepagesize: 2048 kB confirms 2 MiB is the huge page size on this hardware.

If the last command printed nothing, no process on your machine is using THP — likely a small VM with nothing large running. That is a perfectly normal result, not a failure.

On a typical cloud VM you will get the boring answer, and it is worth seeing:

available: 1 nodes (0)
node 0 cpus: 0 1
node 0 size: 3936 MB
node 0 free: 2711 MB
node distances:
node   0
  0:  10

/sys/devices/system/node/node0

                           node0
numa_hit                 8814923
numa_miss                      0
numa_foreign                   0
interleave_hit              5211
local_node               8814923
other_node                     0

One node. NUMA does not apply to this machine, and that is the correct conclusion to draw. Most small cloud instances are presented as a single node, and pinning, interleaving and NUMA tuning are all pointless on them. Half of all NUMA tuning advice on the internet is being applied to machines like this, where it does nothing.

On a two-socket physical server you get the interesting answer instead:

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23
node 0 size: 128809 MB
node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31
node 1 size: 129011 MB
node distances:
node   0   1
  0:  10  21
  1:  21  10

                           node0           node1
numa_hit             48812993204     44120384011
numa_miss              892043118               0
numa_foreign                   0       892043118
local_node           48812993204     44120384011
other_node             892043118               0

What to read out of this.

The distance matrix is the key output. 10 is the reference for local access, so 21 means reaching node 1 from node 0 is declared as roughly twice as expensive. Treat that as a hint, not a measurement: the number is a relative cost the firmware publishes, normalised so local is 10, and the kernel uses it for placement decisions. Actual measured remote latency on a two-socket box is usually 1.5–1.8× even where the firmware says 21.

The CPU lists are worth noticing too. Node 0 owns CPUs 0-7 and 16-23 — the second range is the hyperthread siblings of the first. So "CPU 16" and "CPU 0" are the same physical core. Pinning a two-thread job to CPUs 0 and 16 gives you one core, not two, and this trips people up constantly.

Now numastat, and the two counters are easy to get backwards. numa_hit on a node counts allocations that landed there and meant to. numa_miss on a node counts allocations that landed there although a different node was preferred — that node absorbed someone else's overflow. numa_foreign on a node counts allocations that were meant for it and had to go elsewhere. They are two views of the same event. Here node 1 shows 892 million numa_foreign and node 0 the matching 892 million numa_miss, so the node that ran short is node 1, and node 0 supplied the pages.

A ratio of misses to hits under about 1% is normal noise. This one is 1.8% and rising, which is worth investigating. The usual causes are a single large process not confined to one node, or a badly balanced set of container CPU pinnings.

Huge pages are larger units of translation — 2 MiB or 1 GiB on x86-64 instead of 4 KiB. The benefit is TLB reach: a fixed number of TLB entries covers hundreds of times more memory, so a large working set stops thrashing the TLB, and the page tables describing that memory shrink dramatically as well.

They are worth using for processes with large, densely used memory regions — databases, JVM heaps, in-memory caches, scientific workloads. They are not worth using for ordinary services with small footprints, where the only effect is wasted memory from partly used 2 MiB pages.

The details that separate candidates: quantifying the reach — roughly 1500 TLB entries covers about 6 MB with 4 KiB pages and about 3 GB with 2 MiB pages — and distinguishing transparent huge pages, which the kernel applies opportunistically, from explicit hugetlbfs pages, which are reserved at boot, never swapped and never split. Databases generally want the explicit kind and want the transparent kind turned off, and being able to say why is the whole answer to the next question.

Because of the allocation stall. Handing out a 2 MiB page needs 2 MiB of physically contiguous free memory. On a long-running machine memory is fragmented, so the kernel may have to compact memory to produce it — and with defrag set to always, the process that faulted blocks while that happens. The result is sporadic latency spikes of hundreds of milliseconds with no CPU, disk or network activity to explain them, which is exactly the failure mode a database cannot tolerate.

There is a secondary cost: internal fragmentation. A mapping that needs 40 KB still consumes a full 2 MiB page, so RSS inflates for no benefit.

The details that separate candidates: knowing there are two independent settings — /sys/kernel/mm/transparent_hugepage/enabled and .../defrag — and that the stall comes from defrag, not from enabled. The defaults are not uniform: Debian and Ubuntu ship enabled=madvise, while RHEL, Rocky, Fedora and Amazon Linux ship enabled=always; defrag defaults to madvise almost everywhere. So blanket "disable THP" advice copied from a 2014 blog post is often redundant on an Ubuntu fleet and still entirely relevant on a RHEL one. Reading the two values before changing anything is the professional answer, and it is also the answer that shows you know they differ.

On a multi-socket machine each socket has its own directly attached RAM. Accessing your own socket's memory is fast; reaching the other socket's memory crosses an interconnect and costs roughly twice the latency. Linux's default allocation policy is first touch — a page lands on the node of whichever CPU first wrote to it.

The performance impact comes from the mismatch that policy allows: a thread can allocate memory on node 0, be migrated by the scheduler to node 1, and then run for hours accessing all of its memory remotely. Nothing errors, nothing is logged, and the process is simply slower.

The details that separate candidates: naming the observable — numa_miss and numa_foreign in numastat, with a miss-to-hit ratio above about 1% worth investigating — and knowing the fix is placement, numactl --cpunodebind --membind or the equivalent container topology policy, not more memory. The detail that impresses is checking numactl --hardware first: most cloud VMs present a single node, where every piece of NUMA tuning advice is a no-op, and knowing when not to tune is as valuable as knowing how.

Run, in order: ps -eo pid,vsz,rss,comm | grep <service> → then awk over /proc/PID/maps for large regions → then grep Pss /proc/PID/smaps_rollup.

What you will find. The dashboard is almost certainly plotting VSZ. A Go runtime reserves a large contiguous region of address space at start-up — commonly around 1 GB — with permissions ---p and no file behind it. Not one page of RAM is involved. The maps output will show it as a single enormous permission-less mapping, and RSS will be a small fraction of the reported figure.

Two pieces of evidence that settle it in under a minute. First, the number is identical on every host and never moves. Real memory use varies with load; a reservation made at start-up does not. Second, RSS from ps — or better, Pss from smaps_rollup — will be a small fraction of 1.2 GB, and /proc/PID/maps will show the bulk of that address space as one permission-less ---p region with no file behind it. Two readings, no tooling to install, and the question is answered.

The answer: no, do not raise the limit. Fix the dashboard to report RSS or, better, the container's own cgroup memory accounting. Raising a limit to accommodate a number that was never memory wastes real capacity across the whole fleet.

Run, in order: ps -eo pid,ppid,rss,comm to confirm they share a parent → grep -E 'Rss|Pss|Private_Dirty' /proc/PID/smaps_rollup for one worker → sum Pss across all eight.

What you will find. The workers were forked from one parent, so they share the interpreter, every imported module, and any data structure loaded before the fork — via copy-on-write. RSS counts all of that in full for each of the eight. Pss will be far lower, perhaps 120 MB each, and Private_Dirty lower still, perhaps 70 MB.

So will dropping to 4 workers halve memory? No. It removes four copies of the private portion only. If private is 70 MB, you free about 280 MB, not 1.35 GB — while halving your concurrency. That is a poor trade made on the strength of a number that was never real.

The arithmetic to do first: sum Pss across the eight workers and the parent. That total is the true cost, and it is the only figure worth planning against. If the true total is comfortable, the right answer is to change nothing.

If you do need to reduce it, the lever is the private footprint per worker, not the worker count — load large read-only data structures before forking so they stay shared, and avoid touching them afterwards, since a write breaks the sharing for that page.

The shape of the symptom is the clue. Spikes that hit every endpoint simultaneously, including ones that do no work, mean the process itself was stopped — not that any particular operation was slow. Something froze the whole process for a few hundred milliseconds.

With CPU low, disk idle and network clean, the short list is: garbage collection pauses, a lock held across the whole process, or the kernel blocking a page fault. The last one is invisible in every metric mentioned in the ticket.

Run, in order:

cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag
grep -E 'AnonHugePages' /proc/meminfo
grep -E 'compact_stall|compact_fail' /proc/vmstat

What confirms it. defrag set to always, plus a compact_stall counter in /proc/vmstat that increases while the spikes happen. compact_stall counts exactly this: a process was stopped so the kernel could compact memory to produce a contiguous 2 MiB block. Take two readings a minute apart — a rising count during a spike window is the proof.

The fix: set both enabled and defrag to madvise, and assert those values in configuration management so a vendor tuning package cannot quietly change them back. Expect this on long-uptime hosts specifically — a freshly booted machine has plenty of contiguous memory and shows no stalls at all, which is why the problem "appears" a few weeks after a rebuild.

Hypothesis: NUMA placement, decided by chance at start-up.

Two sockets, first-touch allocation, and no pinning means the outcome depends on which CPU happened to run the setup phase and whether the scheduler later moved the threads. A run that allocates and executes on the same node is fast; a run whose threads migrate spends its life reaching across the interconnect at roughly twice the latency. Nothing errors, and CPU, memory and I/O totals are identical on both runs — which is exactly what the ticket describes.

Run, in order:

numactl --hardware                       # confirm there really are 2 nodes
numastat                                 # numa_miss / numa_foreign per node
numastat -p <pid>                        # per-node memory for the job itself
grep -E 'Cpus_allowed_list' /proc/<pid>/status

What confirms it. numastat -p on a slow run shows the job's memory concentrated on one node while Cpus_allowed_list spans both, or a numa_miss count that climbs during the slow runs and not the fast ones.

The fix: pin both CPU and memory together — numactl --cpunodebind=0 --membind=0 <job> — so allocation and execution cannot end up on different nodes. Pinning CPU alone is not enough, and pinning memory alone is worse than nothing.

Check the premise first. If numactl --hardware reports one node, this hypothesis is dead and the variance is something else — most likely a noisy neighbour showing up as steal time, which Module 07 covers.