⚙️Module 11 — systemd and Services

On a booted Ubuntu 24.04 VM:

systemd 255 (255.4-1ubuntu8.15)
    PID COMMAND
      1 systemd
running

What to read out of it:

• systemd 255 — the version; Ubuntu 24.04 ships systemd 255. Yours may differ by a point release.
• PID 1 is systemd — Module 8's abstract "PID 1" now has a name and a job. Every service you'll manage is its descendant.
• is-system-running returns one word: running (healthy), degraded (something failed — investigate with systemctl --failed), starting, etc. It's the fastest "is this box OK?" check there is.
• In a non-systemd container you'll instead see ps -p 1 show something else (a container init) and is-system-running fail with Failed to connect to bus — that's the caveat from "Before you start"; switch to a VM for the live parts, or read on for the systemd-analyze exercises that work anywhere.

/usr/lib/systemd/system/basic.target
/usr/lib/systemd/system/graphical.target
/usr/lib/systemd/system/multi-user.target
...

then the target file:

[Unit]
Description=Multi-User System
Documentation=man:systemd.special(7)
Requires=basic.target
Conflicts=rescue.service rescue.target
After=basic.target rescue.service rescue.target
AllowIsolate=yes

then, on a booted machine, a table of .service units with their load/active/sub state and description.

What to read out of it:

• Target files are tiny — a target is pure dependency wiring, no program of its own. Requires=basic.target and After=basic.target say "I need the basic system up first, and I come after it" — the ordering-and-dependency vocabulary that Part C teaches you to write.
• The list-units table is the live view: each service's name, whether it's loaded, active, and its fine-grained sub state (running/exited/dead). This is the fleet operator's at-a-glance status.
• Notice you read a packaged unit from /usr/lib — the read-only shelf. Your own units go in /etc/systemd/system, next section onward.

systemd is the init system and service manager on most modern Linux distributions — PID 1, started by the kernel at boot. It brings the system up (resolving dependencies and starting units in parallel where possible), supervises services (restarting crashed ones, stopping them cleanly), manages logging (the journal), timers, mounts, sockets, and more — everything as units. It replaced SysV init and Upstart, whose sequential shell-script approach couldn't do dependency-aware, parallel, supervised startup.

The details that separate candidates: "PID 1 and service manager" as the two-part definition; naming what it replaced and why (parallelism, dependency ordering, supervision, unified logging — not just "it's newer"); and knowing the everyday interface is systemctl for control and journalctl for logs. The mature nuance: systemd is controversial for its scope ("does too much"), which you can acknowledge while noting it's the de-facto standard you must know.

Firmware (BIOS/UEFI) powers on and runs its POST, then hands off to the bootloader (GRUB on most Linux), which loads the kernel and an initial ramdisk (initramfs) into memory. The kernel initializes hardware, mounts the real root filesystem, and starts PID 1 — systemd. systemd then brings the system to its default target (usually multi-user.target for servers, graphical.target for desktops), starting all the units that target requires — networking, logging, sshd, the application services — in dependency order, ending at a login prompt.

The details that separate candidates: naming the handoffs (firmware→bootloader→kernel→systemd→target) as a chain of increasing abstraction; that systemd reaching the default target is "boot finished"; the modern correction to older answers — there is no /etc/inittab or numeric runlevels on systemd systems, those are SysV concepts mapped onto targets (the runlevel question below); and systemd-analyze blame / critical-chain to debug slow boots, which shows real operational experience.

● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/usr/lib/systemd/system/ssh.service; enabled; preset: enabled)
     Active: active (running) since Wed 2026-09-02 14:02:11 UTC; 3h 40min ago
   Main PID: 812 (sshd)
      Tasks: 1 (limit: 4633)
     Memory: 5.8M
        CPU: 142ms
     CGroup: /system.slice/ssh.service
             └─812 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"

Sep 02 14:02:11 myhost systemd[1]: Starting ssh.service...
Sep 02 14:02:11 myhost systemd[1]: Started ssh.service.

What to read out of it, line by line (this output is a literacy test interviewers use):

• The coloured dot and Active: active (running) — the service is up. active (running) = a process is running; active (exited) = a one-shot that finished successfully; inactive (dead) = stopped; failed = crashed.
• Loaded: … enabled — the second word is the boot setting (B2): enabled means it starts at boot, separate from whether it's running now.
• Main PID: 812 (sshd) — Module 8's process, owned by this service. CGroup: shows systemd tracks all the service's processes in a control group (Module 16's cgroups, previewed) — so it can stop the whole tree cleanly, not just the main PID.
• The bottom lines are the last few journal entries — status tails the logs automatically, which is why status is usually your first debugging command. The restart didn't drop your SSH session because your session is a separate process from the sshd listener being restarted — a genuinely reassuring fact worth understanding before you ever restart ssh on a box you're logged into.

active
enabled
enabled
     Loaded: loaded (/usr/lib/systemd/system/ssh.service; enabled; preset: enabled)

What to read out of it:

• is-active and is-enabled are different questions with different answers — the whole point. Scripts and monitoring use these one-word commands (they set exit codes too, Module 5) rather than parsing status.
• On a real machine, run systemctl is-enabled on a few services and you'll find some active but disabled (started manually, won't survive reboot) — go find out why, because that's exactly the state that causes "it worked until we rebooted" incidents.
• The Loaded: line embeds the enabled state (; enabled;) so status answers both questions at once — but is-active/is-enabled are the scriptable forms.

sudo systemctl start/stop/restart name to control it now; reload to re-read config without downtime (if supported); systemctl status name to inspect state, PID, and recent logs; sudo systemctl enable/disable name to control auto-start at boot; enable --now / disable --now to combine boot-setting and immediate action; systemctl is-active / is-enabled for scriptable one-word answers. Logs via journalctl -u name.

The details that separate candidates: the enable-vs-start distinction stated unprompted (running-now vs starts-at-boot are independent — the classic trap); reload-vs-restart (graceful config re-read vs an outage); and reaching for enable --now as the safe default so "works now" and "works after reboot" can't drift apart. Reciting the five verbs is recall; explaining the two orthogonal axes (now/boot) is experience.

On systemd, "runlevels" are targets. Set the default with sudo systemctl set-default multi-user.target (the server default, ≈ old runlevel 3) or graphical.target (≈ runlevel 5). Check it with systemctl get-default; switch the running system to a target without rebooting with sudo systemctl isolate multi-user.target. The old SysV runlevels map: 0→poweroff.target, 1→rescue.target, 3→multi-user.target, 5→graphical.target, 6→reboot.target.

The details that separate candidates: knowing there is no /etc/inittab on modern systems and that numeric runlevels are a compatibility mapping onto targets, not the real mechanism; get-default/set-default/isolate as the actual commands; and that a "target" is a more flexible concept than a runlevel (arbitrary named groupings, not a fixed 0–6 ladder) — which reframes an old-Unix question as current knowledge.

systemd-analyze verify prints nothing and exits 0 when the unit is valid (verified on a systemd-255 machine) — silence is success (Module 1, still true). Then on a booted machine:

verify exit: 0
Created symlink /etc/systemd/system/multi-user.target.wants/hello.service → /etc/systemd/system/hello.service.
● hello.service - Hello demo service
     Loaded: loaded (/etc/systemd/system/hello.service; enabled; preset: enabled)
     Active: active (running) since Wed 2026-09-02 18:15:03 UTC; 1s ago
   Main PID: 20194 (hello-daemon.sh)

What to read out of it:

• systemd-analyze verify is your pre-flight check — it catches typos, missing ExecStart, non-executable commands, and bad directives before you try to run the service. Run it on every unit you write; it works even in a container with no booted systemd.
• enable --now printed the symlink it created — literally Module 3's symlink, from multi-user.target.wants/ to your unit. That symlink is what "enabled" means — its presence is why systemd starts your service at boot. You can see the whole mechanism in one line of output.
• Loaded: … /etc/systemd/system/hello.service; enabled — your file, from the admin directory, enabled. Active: active (running), Main PID — it's alive and supervised. Module 8's nohup could never say enabled or restart this on crash; now systemd will.
• Test the supervision: sudo kill <Main PID>, wait 5 seconds (RestartSec), systemctl status hello again — a new PID, because Restart=on-failure brought it back. That auto-restart is the entire reason services beat nohup.

bad.service: Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing.
Unit bad.service has a bad unit file setting.
verify exit: 1

What to read out of it:

• An empty ExecStart= means "no command to run" — systemd refuses the unit outright, and verify caught it with a precise message and a non-zero exit (scriptable — you can lint units in CI, Module 10's discipline applied to infrastructure).
• The lesson is the habit: systemd-analyze verify before daemon-reload, every time. A unit that fails verify would fail to start on a real machine too, but catching it with verify means you find out in one second instead of after a daemon-reload, an enable, and a confusing status reading failed.

multi-user.target
● ├─ssh.service
● ├─cron.service
● ├─rsyslog.service
● ├─systemd-logind.service
● ├─basic.target
● │ ├─...
...

then ssh's unit showing its [Unit] After=network.target auditd.service and [Install] WantedBy=multi-user.target.

What to read out of it:

• list-dependencies draws the tree systemd walks at boot: multi-user.target wants sshd, cron, logging… each of which pulls in its dependencies, down to basic.target and the early-boot units. This is the boot process (Part A's interview question) as a live graph.
• systemctl cat prints a unit's real, merged configuration — the fast way to read what a service actually does, including any admin overrides. Reading After=/Wants= in real units trains the vocabulary you'll write in your own.
• The green dots are live state — this isn't a static file, it's the running dependency graph with each unit's status attached. Boot order and current health, in one view.

Background processes with no controlling terminal that provide services continuously — sshd, cron, databases, web servers (Module 8's TTY-? definition). On modern Linux, daemons are managed as systemd services: rather than the program daemonizing itself (the old double-fork ritual), you write a .service unit with Type=simple and let systemd own the lifecycle — starting, supervising, restarting, and logging it. The unit file is how you make a program a daemon today.

The details that separate candidates: connecting the Module 8 concept (a daemon is a detached background process) to the Module 11 practice (you don't hand-daemonize any more — systemd does it via a unit); Type=simple vs Type=forking (the latter for legacy programs that still daemonize themselves, where systemd needs to know to expect the fork); and that Restart=on-failure gives you supervision the old self-daemonizing model never had.

Write a .service unit in /etc/systemd/system/: [Service] with the absolute-path ExecStart=, Restart=on-failure and RestartSec= for crash supervision, User= for least privilege; [Install] with WantedBy=multi-user.target for boot; [Unit] with After=/Requires= for dependencies. Then systemd-analyze verify it, sudo systemctl daemon-reload, and sudo systemctl enable --now name. Logs go to the journal automatically (journalctl -u name).

The details that separate candidates: the full lifecycle in order (write → verify → daemon-reload → enable --now); why each directive earns its place (Restart for supervision, User for security, After+Requires for dependencies); the daemon-reload-after-edit habit; and framing it against the alternatives — this is strictly better than nohup (Module 8, no supervision) and cron (no long-running supervision), which is the reason services exist.

Sep 02 14:02:11 myhost systemd[1]: Starting ssh.service...
Sep 02 14:02:11 myhost sshd[812]: Server listening on 0.0.0.0 port 22.
Sep 02 14:02:11 myhost sshd[812]: Server listening on :: port 22.
Sep 02 14:02:11 myhost systemd[1]: Started ssh.service.
Sep 02 15:47:03 myhost sshd[1423]: Accepted publickey for zaeem from 10.0.0.5 port 51234 ssh2
...

What to read out of it:

• Every line is tagged: timestamp, hostname, the process (sshd[812] — name and PID), then the message. The journal added that structure automatically — the service just wrote plain text to stderr.
• The Accepted publickey for zaeem from 10.0.0.5 line is a real security-relevant event (a successful SSH login — Module 15) surfaced by a simple -u ssh query. Filtering one service's logs out of the whole system's noise is what makes the journal powerful.
• -p warning filtered by priority — the journal knows each line's severity (the err/warning/info levels), so "show me only problems" is one flag, not a grep guess. This is the structured-logging advantage over plain text files.

Normalized form: *-*-* 02:30:00
    Next elapse: Thu 2026-09-03 02:30:00 UTC
       From now: 3h 41min left
  Original form: Mon..Fri 09:00
Normalized form: Mon..Fri *-*-* 09:00:00
    Next elapse: Thu 2026-09-03 09:00:00 UTC
  Original form: weekly
Normalized form: Mon *-*-* 00:00:00
    Next elapse: Mon 2026-09-07 00:00:00 UTC

then, on a booted machine, list-timers showing each timer, its NEXT run, LAST run, and the unit it activates.

What to read out of it (the Next elapse dates are relative to when you run it):

• systemd-analyze calendar proved what each expression means — Mon..Fri 09:00 normalized to "weekdays at 09:00:00", next firing this Thursday. Compare cron, where you decode 30 2 * * * in your head and hope. Verifying the schedule before deploying it is the whole point — no more "the backup ran at the wrong time" surprises.
• weekly is a named shortcut (systemd has daily, hourly, monthly too) — readable schedules without cron's five-field puzzle.
• list-timers is the fleet-friendly view cron never had: all scheduled work and its next-run time in one place. On a real server it's the first command for "what automated jobs run on this box, and when?"

journalctl queries the systemd journal — the central, structured, indexed log of every service's output plus kernel and boot messages. Key usage: -u name (one service), -f (follow live), --since/--until (time windows), -p err (by priority), -b (this boot). It replaces hunting through scattered /var/log files: the journal tags every entry with its source service, timestamp, and priority, making "errors from the database service in the last hour" a one-line query.

The details that separate candidates: structured/indexed vs plain-text log files (filter by unit and priority, not grep); that status shows only a teaser tail while journalctl -u has the full story; the retention caveat (journals rotate and may be non-persistent — Storage=persistent, and ship to central logging for real retention); and that your own services get journaled for free by writing to stdout/stderr, so you no longer manage per-service log files.

The published banks cover systemd's existence and systemctl basics well (quoted throughout this module) but barely touch writing unit files, timers, or journalctl querying as dedicated questions — yet these are exactly what a hands-on systemd interview probes: "write me a service unit for this script", "how would you schedule a backup on a modern system?", "a service keeps crashing — walk me through debugging it". Prepare the artifacts: a minimal .service from memory (the three sections and the key directives), the write → verify → daemon-reload → enable --now lifecycle, a timer as the cron alternative with its two files, and the status → journalctl -u --since debugging flow. That's the live systemd interview, and the corpus gap makes fluency here a clear differentiator.

Ticket text: "The metrics-collector app runs great — we systemctl start metrics after each deploy — but every maintenance reboot it comes up missing and someone has to start it again. Fix it permanently."

Worked answer: the direct cause is the enable-vs-start distinction — start runs it now but never registers it for boot, so every reboot brings the box up without it. Confirm: systemctl is-enabled metrics → disabled (while is-active → active, the classic split). Fix: sudo systemctl enable metrics (it's already running, so no --now needed — but enable --now is the safe habit that makes the two states impossible to desync). Then verify the fix survives what it's meant to: on a maintenance window, actually reboot and confirm is-active comes back active, because "enabled" is a promise you should test once. The deeper process fix for the ticket: deploys should enable --now, never bare start, so this can't recur — and monitoring should alert on is-enabled disabled for services that must persist.

Ticket text: "Edited /etc/systemd/system/nginx.service to add a memory limit, ran systemctl restart nginx, but systemctl show nginx still reports the old limit. The file is definitely saved. What's wrong?"

Worked answer: systemd is running its in-memory copy of the unit, loaded at last boot or last reload — editing the file on disk doesn't reach the running manager until you tell it to re-read. systemctl cat nginx will show systemd still holds the old content. Fix: sudo systemctl daemon-reload (re-read all unit files), then sudo systemctl restart nginx (apply). The order matters — reload updates systemd's model, restart applies it to the process. systemd even warns about this: after editing a unit it often prints "Warning: The unit file changed on disk. Run 'systemctl daemon-reload'". The habit to install team-wide: edit unit → daemon-reload → restart, always in that order; and prefer systemctl edit nginx (which creates an override and reloads for you) over hand-editing the file, to make the reload impossible to forget.

Ticket text: "Our app.service fails immediately at boot with a database connection error, but systemctl start app by hand a moment after boot works perfectly. The unit has Requires=postgresql.service. Diagnose."

Worked answer: textbook Requires= without After=. Requires=postgresql.service pulls Postgres in but says nothing about ordering, so systemd — parallel by default — launches app and Postgres simultaneously at boot; app wins the race, tries to connect to a database still starting up, and crashes. Starting by hand "later" works only because Postgres is up by then — masking the real bug. Confirm with systemctl cat app (a Requires= with no matching After=) and journalctl -u app -b (connection-refused at the exact boot timestamp). Fix: add After=postgresql.service to [Unit] (keep the Requires= too — you want both: pulled in and ordered). Consider also Restart=on-failure so a transient race self-heals, and if the app needs the network, After=network-online.target with Wants=network-online.target. The one-sentence lesson for the postmortem: in systemd, "depends on" and "starts after" are separate switches, and a real dependency needs both.

Ticket text: "Our backup is a cron line that failed silently for a week last month (Module 10's incident). Leadership wants it on something visible and catch-up-capable. Build it with systemd."

Worked answer: convert to a timer + service pair. The service (backup.service): [Service] Type=oneshot, ExecStart=/usr/local/bin/backup.sh (absolute path), User=backup — a plain run-once unit whose output the journal captures automatically (solving cron's silent-failure trap: journalctl -u backup shows every run). The timer (backup.timer): [Timer] OnCalendar=*-*-* 02:30:00, Persistent=true (runs a missed backup after the machine comes back — the catch-up cron lacked), [Install] WantedBy=timers.target. Then: systemd-analyze calendar '*-*-* 02:30:00' to prove it fires at 02:30, daemon-reload, sudo systemctl enable --now backup.timer (enable the timer, not the service), and systemctl list-timers to confirm its next run is visible. Monitoring hooks into systemctl is-failed backup.service or the journal. The framing to deliver: this gives leadership exactly what they asked — visibility (list-timers, journal), catch-up (Persistent), and the same script — with the silent-failure mode designed out rather than papered over.