✂️Module 7 — Text Surgery: sed and awk

listen_port=8080
db_host=db-03
db_port=5432
# temporary override
db_host_backup=db-02
XXX bbb aaa
XXX bbb XXX

What to read out of it:

• The streamed copy shows db-03; the cat (output omitted above — run it) still says db-01. Stream-editing is inherently safe: you can inspect the would-be result forever before committing anything (A3 commits).
• The aaa pair is the g lesson in miniature — first-per-line versus all. Burn it in now; it costs real incidents later.
• Note db_host_backup=db-02 sailed through unchanged — the regex db-01 matched precisely. When it must be even more precise (db-01 but not db-011), Module 6's word boundaries and anchors apply verbatim.

db_host=db-01
listen_port=8080
db_host=db-01
db_port=5432
listen_port=8080
db_host=db-01
db_port=5432
db_host_backup=db-02

What to read out of it:

• -n plus p is "print nothing except what I ask" — without -n, sed echoes every line and prints the addressed ones again (run it; the doubled line 2 is a rite of passage).
• The comment-strike is the classic "show me the effective config": sed '/^#/d; /^$/d' file (two commands, semicolon-joined, the second deleting blank lines) prints what the machine actually reads. This one-liner earns its keep weekly.

listen_port=9090
db_host=db-01
db_port=5432
# temporary override
db_host_backup=db-02
1c1
< listen_port=8080
---
> listen_port=9090

What to read out of it:

• Three steps, three proofs: the preview showed intent, the -i.bak committed it, the diff certifies that exactly one line changed and which. Paste that diff into the change ticket — this is what auditable config editing looks like without version control.
• Try ls -i /tmp/ritual.conf before and after a second edit: the inode number changes each time. The yellow callout, verified on your own machine in ten seconds.

./app/config/app.conf
./app/logs/app.log
./app/logs/app.log
./app/config/app.conf.bak

What to read out of it:

• Before: config and log both matched. After: only the log still says db-01 — correct! Logs are history; rewriting them would be falsifying records. The -name '*.conf' scope expressed that policy in one test.
• The verification grep is not paranoia; it is the completion criterion. A change is done when the check that motivated it comes back clean. (The --exclude='*.bak' keeps the just-created seatbelts — which of course still contain db-01 — out of the verdict.)
• Undo, if ever needed: mv app/config/app.conf.bak app/config/app.conf. (Restore the playground now for later sections: run exactly that.)

The canonical scenario: a hostname, IP, or copyright line must change across a config tree — e.g. decommissioning db-01: grep -rl 'db-01' /etc/app.d to scope, then find /etc/app.d -name '*.conf' -exec sed -i.bak 's/db-01/db-03/g' {} +, then the same grep to verify empty, keeping .bak files until the change proves out. The g flag matters (multiple occurrences per line), the suffix backup matters (rollback), and the scoping test matters (never sed what you haven't grepped).

The details that separate candidates: the four-step ritual (scope, preview, execute-with-backups, verify) narrated as a procedure; the logs-are-history scoping judgment; and the strategic caveat — at fleet scale this is what configuration management replaces, a sentence that reliably upgrades the interviewer's model of you.

Question banks rarely quiz raw sed syntax; they test it live — "here's a config, delete the comments", "change this port". The muscle memory that covers nearly everything asked: s/old/new/g, alternate delimiters for paths (s|…|…|), -n with p for extraction, /pattern/d for deletion, -i.bak for committing, address+command composition (/^db/s/…/…/). If you can produce those six shapes cold, live sed rounds become dictation.

/api/users
/api/orders
/api/orders
/health
/api/users
/api/login
/health
/api/orders
10.0.0.5 200
10.0.0.7 200
10.0.0.5 500
10.0.0.9 200
10.0.0.5 200
10.0.0.7 401
10.0.0.9 200
10.0.0.5 500

What to read out of it:

• Column three — the paths — extracted in one breath; this is Adaface's verbatim interview question ("print the third word of each line"), answered.
• The comma in print $1, $4 inserts the output separator (a space by default); without the comma the fields concatenate into one word. Small syntax, frequent bug.
• Now break it on purpose: run the first command with double quotes. Whole lines come back — your shell expanded $3 to nothing before awk saw it. Diagnose-by-symptom: awk printing full lines = quoting bug, almost always.

10.0.0.5 POST /api/orders 500 431
10.0.0.5 GET /api/orders 500 502
POST /api/orders
GET /api/orders
10.0.0.5 POST /api/orders 500 431
10.0.0.5 GET /api/orders 500 502

What to read out of it:

• Line-by-line: both 500s hit /api/orders — different methods, same endpoint. The reshaped second view says it loudest: the orders endpoint is the problem. You are now doing analysis, not just extraction.
• The compound condition reads like the sentence it is: status at-least-400 AND this client. grep would need chained pipes and could not compare numbers at all — grep 500 would also match a response-time of 500 ms in the last column. Numeric column comparison is awk's home turf.

listen_port
db_host
db_port
# temporary override
db_host_backup
db-01
5432
db-02

What to read out of it:

• The keys column includes the comment line, whole — no = on it, so the entire line is field one. awk never errors on shape surprises; it adapts, and your program must expect that (a /^#/ {next} guard, or B2's patterns, filters such lines).
• The second command is a config query: all db-related values, one pipeline-ready value per line. Compare the Module 3 way (open in less, read with human eyes) — you have graduated from reading files to interrogating them.

1283
10.0.0.5 4
10.0.0.7 2
10.0.0.9 2

(The tally's three lines may arrive in any order — that is the unspecified-order rule, live.)

What to read out of it:

• 1283 ms summed across the stream in one pass, no temporary files, constant memory. The same command handles eight lines or eight billion.
• The tally says 10.0.0.5 dominates traffic — four of eight requests, including both 500s (cross-reference B2.1). Two one-liners in, you have a suspect. This is what "processing log files with awk" means in the Adaface question — and in the job.

awk '{print $3}' filename — awk splits each line on runs of whitespace into $1…$NF; the action prints field three for every line (no pattern = all lines). Single quotes are mandatory: double-quoted, the shell expands $3 to nothing and awk prints whole lines.

The details that separate candidates: volunteering the quoting failure mode before being asked; knowing the default splitting subtleties (leading whitespace ignored, any run of spaces/tabs is one separator — so columns align even in ragged output); and the reach for -F the moment the delimiter isn't whitespace.

The skeleton is pattern { action }: regex patterns (/error/ {print $1} — timestamps of error lines), field comparisons ($4 == 500), compounds ($4 >= 500 && $3 ~ /api/). Add accumulation for summaries: '{count[$1]++} END {for (k in count) print k, count[k]}' groups requests by client in one pass; '{sum += $5} END {print sum/NR}' averages a column (NR = number of records). One pass, constant memory, no temp files — which is why awk survives on gigabyte logs where spreadsheet thinking dies.

The details that separate candidates: producing the group-and-count idiom from memory — it is the awk interview move; explaining why numeric comparisons beat grep for status codes (grep can't tell a status 500 from a 500 ms latency); and knowing END exists because streams can't be re-read.

10
100
2
9
2
9
10
100

What to read out of it:

• First block: the dictionary logic, character by character — 10 and 100 cluster because they share a prefix; 9 exiles to the bottom. Second block: sanity, via -n.
• (printf with \ns is the compact way to feed test lines into a pipe — a Module 10 regular making an early appearance; the quoting rules you know apply.)

1 10.0.0.5
1 10.0.0.7
1 10.0.0.5
1 10.0.0.9
1 10.0.0.5
1 10.0.0.7
1 10.0.0.9
1 10.0.0.5
4 10.0.0.5
2 10.0.0.7
2 10.0.0.9

What to read out of it:

• (In your terminal the counts arrive right-aligned in a 7-character column — uniq -c pads with leading spaces; the block above shows them flush-left.) The unsorted run "worked" — produced plausible-looking counts — and every one of them is a streak length, not a total. This is the most dangerous kind of wrong: no error, reasonable shape, false content. The sorted run gives the true 4/2/2.
• Recite until reflexive: uniq without sort is a bug unless you specifically want streaks (rare, real: "how many identical lines in a row did the app log?" — that one time, it's the tool).

10.0.0.5 200
10.0.0.7 200
10.0.0.5 500
10.0.0.5 200
10.0.0.7 200
10.0.0.5 500

What to read out of it:

• Identical outputs. -f4,1 silently normalized to file order — cut selects, never arranges. The command lied by omission; now you know its one lie.
• Need status IP order? awk '{print $4, $1}'. Choosing the smallest tool that can actually do the job — and knowing when it can't — is the fluency being tested when interviewers ask "cut or awk?"

HELLO WORLD
too many spaces
id123456

What to read out of it:

• Three transformations no regex required: case-fold, squeeze, strip. In pipelines these are the sandpaper passes before the joinery — tr -s ' ' | cut -d' ' -f2 tames space-padded tables that would defeat cut alone.

4 10.0.0.5
2 10.0.0.9
2 10.0.0.7
2 /api/orders
1 /api/login

(Ties — the two 2-count lines — may order either way; sort -rn compares the numbers and leaves equal keys in whatever order they arrived.)

What to read out of it:

• (As in C2.1, uniq -c's counts arrive right-aligned with leading spaces in your terminal.) Two commands, and the incident story is complete: client .5 dominates traffic, and /api/orders owns the failures. On a real outage this pipeline against the last ten thousand lines is often the first real evidence anyone has.
• Notice the pipeline is B4's awk tally plus sorting — two roads to the same summit. The awk form wins on gigantic inputs (no sort of the raw stream); the pipeline form wins on composability and recall-under-pressure. Fluency in both is the interview flex.

The reading set from Module 3 — cat, less, head, tail — plus this module's processors: grep (select lines), sed (edit streams), awk (fields and computation), sort, uniq, cut, tr, wc. The answer that lands is organized by job, not alphabet: view, select, transform, summarize.

The details that separate candidates: following the list with one composed example (awk '{print $1}' log | sort | uniq -c | sort -rn | head) and narrating it stage by stage — a list proves reading; a pipeline proves practice.

No recent published bank asks these directly; they are tested inside pipeline questions — most commonly some variant of "find the top N X in this log", which is C5's idiom verbatim. Practice until you can type the four-stage pipeline in one breath and answer its two classic follow-ups: why must sort precede uniq (adjacency), and why -rn not -r (dictionary-order 9-beats-80 bug). Those two follow-ups are the interview.

2
2
2

What to read out of it:

• Three tools, same count — proof of the overlap. Now the judgment: grep's version is shortest and clearest → correct choice for this question. The sed version needed an escaped slash (\/) inside its pattern — friction signaling wrong tool. The awk version earns its keep only when the question grows conditions ("...and slower than 100 ms": $5 > 100 — try it; the others simply cannot).
• "Simplest tool that answers the question" is the review standard for pipelines, and saying it out loud in interviews lands well.

mawk 1.3.4 20240123
sed (GNU sed) 4.9

(On some Ubuntu installs the first line reads GNU Awk 5.x instead — meaning gawk is installed and selected; both run everything in this module.)

What to read out of it:

• Ubuntu's default awk is mawk — everything this module taught runs identically, but exotic gawk extensions (in-place editing, some time functions) would not. --version before relying on an extension is the thirty-second insurance policy.
• On a Mac, both commands answer differently (BSD tools often lack --version entirely — itself the tell). The habit generalizes far beyond text tools: identify the dialect before trusting the syntax.

1
{"host": "db-03", "note": "moved to db-02, do not use db-01"}

What to read out of it:

• grep said 1 — it counts lines, and JSON often arrives as one line; the true occurrence count (2) needed grep -o 'db-01' | wc -l. First mismatch.
• sed changed the first occurrence (no g), which happened to be the host — this time. Reorder the keys and the same command corrupts the note while leaving the host wrong. Text tools cannot see JSON's structure, only its characters; that a regex worked once is luck wearing the costume of correctness.
• The honest fix is structure-aware: jq '.host = "db-03"'. Say exactly that in interviews when handed JSON — reaching for jq at the right moment reads as seniority, not weakness.

grep selects: fastest way to find or count matching lines, exit code doubles as a test. sed transforms: substitution and line surgery on streams or files, the tool for "change X to Y everywhere". awk computes: field-aware filtering, reshaping, sums and group-counts, the tool the moment columns or arithmetic appear. They compose in pipelines and overlap at the edges; choose by the question's center of gravity, and default to the simplest that answers it.

The details that separate candidates: one crisp example per tool from your own hands (a grep -c, a sed -i.bak ritual, an awk group-and-count); the numeric-comparison point (only awk can ask $4 >= 500); and volunteering the boundary beyond all three — structured data goes to jq/python, which turns a syntax question into an engineering-judgment answer.

Among all this track's topics, sed and awk have the widest gap between published questions (a handful — Adaface's three are the best of them, quoted in Parts A–B) and live interview weight (enormous: log-analysis rounds, "fix this config" rounds, pipeline whiteboards). Prepare accordingly: the six sed shapes (A-part corpus note), the awk skeleton sentence (pattern selects rows, action shapes them), the group-and-count idiom, and C5's four-stage pipeline. Those four memorized artifacts cover the overwhelming majority of what is actually asked.

Ticket text: "On vm-web-1, /var/log/app/access.log (format: IP METHOD PATH STATUS MS). Management wants: top clients, top failing endpoints, and whether failures are slow. Now."

Worked answer, three pipelines: who — awk '{print $1}' access.log | sort | uniq -c | sort -rn | head (top talkers); what fails — awk '$4 >= 500 {print $3}' access.log | sort | uniq -c | sort -rn | head (failing endpoints, ranked); are failures slow — awk '$4 >= 500 {sum+=$5; n++} END {if (n) print sum/n " ms avg over " n " errors"}' access.log versus the healthy baseline awk '$4 == 200 {sum+=$5; n++} END {print sum/n}'. Three numbers and two rankings, no tools installed, evidence paste-ready. The habit worth naming in the ticket reply: every claim ("orders endpoint, avg 466 ms on errors vs 68 ms healthy" — the playground's own numbers) arrives with the command that produced it, so anyone can re-derive it.

Ticket text: "Every file under /etc/metricsd/conf.d must change prefix statsd.legacy. to metrics.v2. — dots included, hence the fear. Provide the exact procedure."

Worked answer, the A4 sequence with regex care: scope — grep -rl 'statsd\.legacy\.' /etc/metricsd/conf.d | wc -l (expect about 40 — a surprise count is a stop sign); preview one — sed 's/statsd\.legacy\./metrics.v2./g' somefile | diff somefile - (dots escaped in the pattern — unescaped they'd match statsdXlegacyY; replacement side needs no escapes); execute — find /etc/metricsd/conf.d -name '*.conf' -exec sed -i.bak 's/statsd\.legacy\./metrics.v2./g' {} +; verify — the scoping grep, re-run with --exclude='*.bak' (the fresh backups still contain the old prefix, by design), returns nothing, and diff file.bak file on two samples shows only intended lines; rollback plan — for loop restoring .baks (Module 10 gives the loop; until then, the .bak files are the plan). Auditable = every step leaves evidence; reversible = originals exist until decommissioned.

Ticket text: "A vendor script ran sed -i on /etc/app/current.conf last night. The file looks correct in cat, but the running app still uses the old value, and our config framework says current.conf 'is no longer a symlink'. Explain both hauntings."

Worked answer: both are A3's yellow callout in production. Haunting one — the running app: it opened the old file at startup and holds it still; sed -i made a new inode and renamed it over the old path, but open file handles follow inodes, not paths (Module 3), so the app reads the ghost until restarted (Module 11's territory — schedule it). Haunting two — the framework's complaint: current.conf was a symlink into a release directory; GNU sed -i, run on a symlink, writes its new file at the link's own path — replacing the pointer with a regular file and orphaning the target. Fix: restore the link (ln -sfn releases/v42/app.conf /etc/app/current.conf), apply the edit to the target file, and file a bug against the vendor script: it should edit link targets (or use sed's --follow-symlinks). Prevention: config trees that use link-swapping (Module 3's deploy pattern) must never be touched by naive -i tooling.

Ticket text: "Product wants, per PATH, the count of distinct client IPs — not requests. One-liner if possible; explain it line by line for the wiki."

Worked answer: distinct-then-count is a two-stage dedup: awk '{print $3, $1}' access.log | sort -u | awk '{count[$1]++} END {for (p in count) print count[p], p}' | sort -rn. Wiki narration: stage 1 emits endpoint-IP pairs; sort -u keeps each distinct pair once (dedup = distinctness); stage 3 counts surviving pairs per endpoint — which is now "distinct IPs per endpoint" by construction; stage 4 ranks. On the playground data: /api/orders 2, /api/users 1, /health 1, /api/login 1 — check by eye against the eight lines to convince yourself (both /health hits come from the same client, so its distinct count is 1 — distinctness doing its job), then trust the pipeline at scale. The teachable subtlety: "unique" questions always need what is unique made explicit as a key — the pair, not the IP — before any counting starts.