🌐Module 13 — Networking

lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0             UP             192.0.2.2/24

(from ip -brief addr; ip addr shows the same with more detail. Your interface names and IPs differ — cloud VMs use ens5, enp1s0, etc.)

What to read out of it:

• lo with 127.0.0.1/8 — the loopback, always present. eth0 with 192.0.2.2/24 — the real interface and its address (the /24 is the network size, A2). 192.0.2.x is a documentation range; yours will be a real private or public IP.
• UP on eth0 means the link is active. An interface showing DOWN is your first suspect when a machine can't reach the network — before touching routing or DNS, confirm the interface is even up (ip link set eth0 up brings it up, but on a cloud VM that's usually managed for you).
• ip -brief is the operator's quick-look; full ip addr adds the broadcast address, scope, and lease details you need when things are actually broken.

4: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc pfifo_fast state UP ...
    link/ether 02:fc:00:00:00:01 brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.2/24 brd 192.0.2.255 scope global eth0

What to read out of it:

• inet 192.0.2.2/24 — the IPv4 address and its network size. /24 = 256 addresses on this local network (192.0.2.0–.255), of which .2 is this host.
• brd 192.0.2.255 is the broadcast address (the "everyone on this network" address); link/ether 02:fc:... is the MAC address — the interface's hardware-level ID, used for delivery within the local network (IPs are for the journey; MACs for the final local hop).
• mtu 1400 — the maximum packet size on this link. Usually 1500; a smaller MTU (common on cloud/VPN links, as here) that's mismatched somewhere causes a maddening class of bug where small requests work and large ones hang. Worth noting when "it works until the response is big."

ip addr (or ip -brief addr for a summary) shows every interface and its IP addresses; ip link shows the interfaces and their state/MAC. The old ifconfig (from net-tools) does similar but is deprecated in favour of ip (from iproute2). To see just one interface: ip addr show eth0.

The details that separate candidates: reaching for ip over ifconfig and knowing why (net-tools is deprecated, ip is the maintained iproute2 tool); reading the output (the /NN CIDR, UP/DOWN state, loopback vs real interface); and the operational instinct — when a box is "off the network," ip -brief addr and ip link are the first checks (is the interface even up and addressed?) before blaming routing or DNS.

The banks ask how to check addresses but rarely test understanding them — CIDR, private vs public ranges, loopback semantics — yet those are exactly what a real networking interview probes, and what you need to debug ("is this host even on the same network as the one it can't reach?"). Be ready to explain /24 from memory (network bits vs host bits, so same-prefix hosts talk directly), name the private ranges (10/8, 172.16/12, 192.168/16), and state the loopback trap (a service on 127.0.0.1 is unreachable remotely by design). That fluency turns a command-recall answer into a systems answer.

default via 192.0.2.1 dev eth0
192.0.2.0/24 dev eth0 proto kernel scope link src 192.0.2.2

then ip route get shows the chosen route for that specific destination.

What to read out of it:

• Two rules, and every packet matches one. 192.0.2.0/24 dev eth0 — "my local network, deliver directly out eth0." default via 192.0.2.1 — "everything else, hand to the gateway 192.0.2.1." That gateway is the door to the wider world.
• ip route get 8.8.8.8 is the diagnostic gem: it tells you exactly which route and gateway a packet to a given destination would use — invaluable when a machine has multiple interfaces or VPNs and you need to know which way traffic actually goes. No guessing.
• Missing default line = the classic "local works, internet doesn't" state. Present but wrong gateway = traffic leaves but goes nowhere useful. Reading this table is a 5-second check that saves long detours into DNS.

On a machine with normal connectivity:

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=8.42 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=8.11 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=8.30 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 8.11/8.27/8.42/0.13 ms

What to read out of it:

• 0% packet loss and consistent time= values = healthy reachability with ~8ms latency. Rising or erratic times, or partial loss, point at a congested or flaky path.
• Pinging the IP vs the name is a deliberate two-step diagnosis: if ping 8.8.8.8 (no DNS needed) works but ping google.com fails, the network is fine and DNS is broken (Part D) — you've split the problem in half with two commands. This IP-then-name technique is the single most useful networking-debug reflex.
• In a locked-down environment (this authoring sandbox, many corporate networks, some clouds) ICMP is blocked, so you'd see 100% packet loss even though the machine reaches the internet fine via TCP — proof of the counter-intuition. On such a box, test with curl (Part C) instead of ping.

ip route (or ip route show) prints the routing table; the line beginning default via <gateway> is the default route. ip route get <destination> shows which route a packet to a specific destination would actually take — the precise diagnostic. The legacy equivalents are route -n and netstat -rn. A missing default route is the classic "reaches the local network but not the internet" symptom.

The details that separate candidates: ip route get as the pinpoint tool (especially with multiple interfaces/VPNs) rather than just dumping the table; connecting a missing default route to its exact symptom; and knowing the routing table is consulted for every outbound packet — an address gets you on the network, a route decides where each packet goes, and both must be right.

ping barely appears as a dedicated bank question, yet "walk me through debugging a machine that can't reach the internet" is one of the most common live networking interviews — and it's pure Part A+B: check the interface is up and addressed (ip addr), check there's a default route (ip route), ping the gateway, ping an external IP, then ping an external name to isolate DNS. Being able to narrate that ladder — each rung isolating a layer (link → address → route → reachability → DNS) — is exactly what the interviewer wants, and it's far more valuable than any single command. The IP-vs-name ping split (network vs DNS) is the star move.

Netid State  Recv-Q Send-Q Local Address:Port  Peer Address:Port
tcp   LISTEN 0      128          0.0.0.0:22         0.0.0.0:*
tcp   LISTEN 0      4096       127.0.0.1:5432       0.0.0.0:*

with sudo adding a Process column like users:(("sshd",pid=812,fd=3)).

What to read out of it:

• Read the Local Address:Port column first. 0.0.0.0:22 — SSH listening on all interfaces, reachable remotely (as it must be). 127.0.0.1:5432 — PostgreSQL listening on loopback only, reachable only from this machine. If a remote app can't reach that database, this line is the diagnosis — no packet capture needed.
• LISTEN = a service waiting for connections; the sudo version names the owning process (Module 8), so "what's using port 8080?" is one command. Peer Address:* on a listener means "will accept from any peer."
• Recv-Q/Send-Q on established connections show queued data — a persistently full queue can indicate a stuck or overwhelmed service. For now, the headline skill is reading listen-address-and-port fluently.

On a connected machine:

HTTP/2 200
content-type: text/html; charset=UTF-8
...
code=200 time=0.183s

What to read out of it:

• HTTP/2 200 — the service answered with status 200 (OK). The status code is the health signal: 200 good, 3xx redirect, 401/403 auth problem, 404 not found, 5xx the server is broken. curl -I is the fastest "is this HTTP service healthy?" test there is.
• The -w "%{http_code} ... %{time_total}" form prints just the code and timing — designed for scripts and monitoring (Module 10): a health-check script curls the endpoint and checks the code, exiting non-zero on failure. This one-liner is the skeleton of half the HTTP monitors in existence.
• If curl fails, how it fails localizes the problem: Could not resolve host = DNS (Part D); Connection refused = reached the host but nothing's listening on that port (check with ss, C1); Connection timed out = can't reach the host at all (routing/firewall, Part B); a TLS error = certificate problem (your SSL track). Four different failures, four different fixes — curl tells you which.

On a machine with ufw active:

Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
To                         Action      From
--                         ------      ----
22/tcp (OpenSSH)           ALLOW IN    Anywhere
80,443/tcp                 ALLOW IN    Anywhere

On a minimal image or container, ufw is often absent — the fallback message is a valid result.

What to read out of it:

• Default: deny (incoming) is the security posture — nothing gets in except the explicitly-allowed ports. The rules below are the only open doors: SSH and web. Everything else is refused at the door, before it ever reaches a service.
• Read this together with ss (C1): ss shows what's listening, ufw status shows what's reachable through the firewall. A service can be listening on 0.0.0.0 (open to all interfaces) yet blocked by the firewall — the two layers combine, and "the service is up and bound correctly but still unreachable" is often the firewall. Check both.
• The commented enable-sequence is the muscle memory to build: allow-yourself → confirm → enable, never enable-then-hope. On production, firewall changes go through the same care as any change that can cause an outage.

ss -tulpn (or the legacy netstat -tulpn): tcp + udp, listening, process, numeric — lists every listening port and the program that owns it (process needs sudo). For a specific port: sudo ss -tulpn | grep :5432 or sudo ss -tlpn 'sport = :5432'. To find and kill whatever holds a port: sudo ss -tlpn | grep :8080 for the PID, then kill (Module 8), or sudo fuser -k 8080/tcp.

The details that separate candidates: reading the listen address (0.0.0.0 reachable vs 127.0.0.1 local-only — the standout answer to "service is up but I can't connect"); ss over netstat and why (ss is the maintained iproute2 tool, faster on busy systems); and combining it with the firewall view (ufw status) since listening ≠ reachable — the two-layer check that solves the real incident.

curl transfers data to/from a URL — the reflexive tool for testing HTTP services and APIs: curl -I for headers/status, curl -v for the full DNS→connect→TLS→request→response trace, -w '%{http_code}' for scriptable health checks; how it fails (resolve/refused/timeout/TLS) localizes the problem by layer. Firewall: on Ubuntu, ufw over netfilter — sudo ufw default deny incoming, sudo ufw allow 22/tcp (and only the ports you need), sudo ufw enable, sudo ufw status; the principle is default-deny + least exposure, and the discipline is allow-your-own-access before enabling on a remote box.

The details that separate candidates: for curl, using it as a layered diagnostic (the four failure modes → four fixes) rather than just "it fetches URLs," and the difference from ping (service vs host); for firewalls, default-deny as the posture, ufw as the front-end over iptables/nftables, and the lock-yourself-out warning that signals real remote-server experience.

nameserver 8.8.8.8
nameserver 8.8.4.4

then /etc/hosts (localhost lines plus any overrides), then dig:

172.66.147.243

(example.com's IP at the time of writing; yours may differ as sites move)

What to read out of it:

• /etc/resolv.conf lists the DNS servers queried — here Google's 8.8.8.8/8.8.4.4. If this file is empty or wrong, no name resolves while raw IPs still work — which is exactly the "ping 8.8.8.8 works, ping google.com fails" split from B2. DNS broken, network fine.
• dig +short returns just the address DNS gives — the fast "what does DNS say?" check. Full dig example.com (try it) shows the whole answer: the query, the ANSWER SECTION with the record and its TTL (how long it's cached), and which server answered — the detail you need when resolution is misbehaving.
• getent hosts returns what the system actually resolves (checking /etc/hosts first, then DNS) — so if dig and getent disagree, you've found a /etc/hosts override. That two-command comparison is the cleanest way to catch the counter-intuition's trap.

142.250.190.14
;; ANSWER SECTION:
google.com.		term  IN	A	142.250.190.14

(the specific IP and TTL vary; the term above is the TTL in seconds, e.g. 300)

What to read out of it:

• dig @1.1.1.1 queries a specific server directly, sidestepping whatever /etc/resolv.conf points at. If your configured resolver fails but @1.1.1.1 works, the problem is your resolver config, not DNS — you've isolated it to the machine. This "ask a known-good server directly" move is the DNS-debug power tool.
• The TTL (the number before IN A) is how long this answer stays cached — relevant when a DNS record was changed and some machines still see the old IP: they're within the TTL window, caching the pre-change answer. "Why does one server still hit the old IP after we updated DNS?" is almost always TTL/caching.
• getent vs dig one more time: getent is what the system uses (the truth for actual connections), dig is what DNS alone says. When an app "can't find" a host that dig resolves fine, suspect the resolution path (nsswitch, systemd-resolved, an /etc/hosts line) between them — getent shows which answer actually wins.

Isolate the layer with the IP-vs-name split: if the site's raw IP loads (or ping/curl an IP works) but the name doesn't, it's DNS. Then walk it: cat /etc/resolv.conf (are resolvers configured and reachable?), dig name (does DNS answer, with the right IP?), dig @1.1.1.1 name (is it your resolver or DNS globally?), getent hosts name vs dig (an /etc/hosts override?), and check the TTL if a recently-changed record still resolves to an old IP (caching). Each step isolates a cause.

The details that separate candidates: the resolution order (/etc/hosts before DNS — and the override trap it creates); getent (system truth) vs dig (DNS only) as distinct tools; querying a specific resolver with dig @ to split "my config" from "DNS itself"; and TTL/caching as the reason changed records don't take effect everywhere at once. "It's always DNS" is a joke because this ladder so often ends there — showing you can run the ladder is the answer.

netstat displays network connections, routing tables, interface statistics, and listening ports — the classic all-in-one network inspector (netstat -tulpn for listening ports+processes, netstat -rn for routes). It's deprecated in favour of the iproute2 tools: ss replaces netstat for sockets/connections (faster, more capable on busy systems) and ip route replaces netstat -rn for routing. Recognize netstat in older docs and on legacy systems; reach for ss/ip on modern ones.

The details that separate candidates: naming the modern replacements and why (net-tools deprecated, iproute2 maintained — the same ip-over-ifconfig story as Part A); mapping the old flags to new tools (netstat -tulpn → ss -tulpn, netstat -rn → ip route); and not being caught out when a minimal image lacks netstat entirely (it's often not installed now) while ss is always there.

Ticket text: "vm-app-1 gets 'connection refused' connecting to Postgres on vm-db-1:5432. On vm-db-1, systemctl status postgresql shows it active and healthy. What's wrong?"

Worked answer — "connection refused" means the packet reached the host but nothing accepted it on that port (contrast "timed out" = didn't reach the host). Since Postgres is confirmed running, it's almost certainly bound to loopback. On vm-db-1: sudo ss -tulpn | grep 5432 — expect 127.0.0.1:5432, the smoking gun: Postgres is listening on loopback only, so it accepts connections from vm-db-1 itself but refuses vm-app-1 by design. Fix: set listen_addresses = '0.0.0.0' (or the specific internal IP) in postgresql.conf, add the client's network to pg_hba.conf (Postgres's own access control), and sudo systemctl restart postgresql (Module 11). Then verify the firewall isn't the next wall: sudo ufw status should allow 5432 from the app network (and check the cloud security group). Re-test from vm-app-1 with curl -v telnet://vm-db-1:5432 or nc -zv vm-db-1 5432. The one-line diagnosis to lead with: the service was healthy but listening on the wrong address — read the Local Address column, not just "is it up."

Ticket text: "Everyone in the office reports acme-portal.com won't load; it loads fine from people's home connections and phones. Is the site down or is it us?"

Worked answer — "works from some places, not others" points at local config, not the site. Diagnose from an affected office machine with the isolation ladder: network vs DNS — curl -I https://<the-site's-IP> (get the IP from a working machine's dig) versus curl -I https://acme-portal.com; if the IP works but the name doesn't, it's DNS. Whose DNS — dig acme-portal.com (does the office resolver answer, and with what?) vs dig @1.1.1.1 acme-portal.com (a public resolver); if the public one returns the right IP and the office resolver doesn't (or returns a wrong/old one), the office DNS is the culprit. Overrides — getent hosts vs dig, and cat /etc/hosts, in case a deployment pushed a bad hosts entry to office machines. Most likely outcome: the office DNS resolver has a stale or broken record (or the office resolver itself is down), which is why home connections using different DNS are fine. Fix is on the office DNS/network side; the site is up. The framing that reassures leadership: "the site is healthy — the problem is our office's name resolution, isolated to these commands."

Ticket text: "We deployed api-svc on port 8080. curl localhost:8080/health on the box returns 200. But the load balancer marks it unhealthy and no traffic flows. Debug."

Worked answer — curl localhost:8080 succeeding proves the service works and is reachable via loopback — which tells you nothing about whether it's reachable from off the box, exactly where the load balancer sits. Check the bind address: sudo ss -tulpn | grep 8080 — if it shows 127.0.0.1:8080, the service listens on loopback only, so localhost works but the load balancer (a different machine) gets refused. That's the bug in most cases: fix the app's bind config to 0.0.0.0:8080 and restart. If it's already 0.0.0.0, climb the layers: firewall (sudo ufw status — is 8080 allowed from the LB's network?), cloud security group (the off-host firewall the OS can't see), and the health-check path/port the LB is configured to hit (is it checking :8080/health or a different port/path?). Reproduce the LB's exact check from another host: curl -I http://<this-box-IP>:8080/health. The lesson threaded through: localhost working is a loopback test, not a reachability test — the LB and the loopback are different network paths, and the whole bug lives in that gap.

Ticket text: "An engineer ran ufw enable on a remote production box to 'tighten security' and now nobody can SSH in. There's cloud console access. Recover it and write the prevention rule."

Worked answer — the cause is textbook: ufw enable with default-deny and no rule allowing port 22 refused all new SSH connections — the engineer's live session may have limped on (established connections are permitted) but every reconnection, and everyone else, was locked out. Recover via the cloud provider's console/serial access (the out-of-band path that doesn't go through SSH): log in there and either sudo ufw allow OpenSSH (add the missing rule — SSH comes back immediately) or sudo ufw disable (turn the firewall off to restore access, then reconfigure properly). Prevention rule for the runbook, the discipline this module drills: always allow your own access before enabling enforcement — sudo ufw allow OpenSSH, confirm with sudo ufw show added, then sudo ufw enable; and on remote boxes, stage a self-recovering safety net first (a scheduled ufw disable in 10 minutes that you cancel once you've confirmed you're still in — Module 10's at/cron, or a systemd timer). Note this is the same shape as Module 4 (don't chmod away your own access) and Module 12 (don't fstab yourself unbootable): never enable an enforcement mechanism that can sever your own access without a tested escape hatch. The postmortem line: the firewall did exactly what it was told — the process failure was enabling enforcement before securing the operator's own path.