🌐Module 08 — Deploying TLS: NGINX, Apache & the Chain-Order Trap

-rw------- 1 zaeem zaeem 1956 Aug 20 15:40 chain.pem
-rw------- 1 zaeem zaeem 3384 Aug 20 15:40 fullchain.pem
-rw------- 1 zaeem zaeem  241 Aug 20 15:40 privkey.pem

=== what is in each? ===
fullchain.pem    2 certificate(s)
chain.pem        1 certificate(s)
-----BEGIN PRIVATE KEY-----

What to read out of this.

• fullchain.pem has 2, chain.pem has 1. fullchain = leaf + intermediates. chain = intermediates only. The names are Let's Encrypt's convention and have become the general standard — and mixing them up is a common source of confusion, because both files are valid PEM and neither tool will complain.
• privkey.pem is 0600 and everything else is too, because of umask 077. In a real deployment the two public files should be 0644 so the web server's worker processes can read them without special privilege — but the key must stay 0600 and owned by root.
• head -1 on the key shows BEGIN PRIVATE KEY — PKCS#8, unencrypted. That is what a web server needs: an encrypted key means the service cannot start unattended (Module 02, A3.2).

💡 The three-second pre-deploy check, which catches most file mistakes before they become incidents:

grep -c 'BEGIN CERTIFICATE' fullchain.pem     # expect 2 or 3, NEVER 1
head -1 privkey.pem                            # expect BEGIN PRIVATE KEY, not ENCRYPTED
diff <(openssl x509 -in fullchain.pem -noout -pubkey) <(openssl pkey -in privkey.pem -pubout)

Two files at minimum, three if you use OCSP stapling:

• fullchain.pem — the leaf certificate followed by the intermediates. Public. It is sent to every client on every full handshake.
• privkey.pem — the private key. Secret. 0600, owned by root, never in version control, never leaves the host it was generated on.
• chain.pem — intermediates only. Public. Needed by nginx's ssl_trusted_certificate for OCSP stapling verification.

The root is never served. A client that trusts it already has it; a client that does not will not trust it because a server offered one.

The detail worth adding: the certificate being public is not a nuance to hedge — it is genuinely published in Certificate Transparency logs and anyone can fetch it from your server. Treating it as sensitive leads to strange workflows. The key is the only irreplaceable, secret thing in the set.

=== what would the server treat as ITS certificate? ===
correct.pem  -> subject=C = MY, O = Zaeem Labs, CN = app.internal.test
reversed.pem -> subject=C = MY, O = Zaeem Labs, CN = Zaeem Labs Issuing CA 1

=== but BOTH still verify as a bag of certificates ===
app.crt: OK
app.crt: OK

What to read out of this — the two blocks say opposite things, and both are true.

• openssl x509 -in reversed.pem -noout -subject prints the CA's name. openssl x509 reads only the first certificate in a file — exactly as nginx does. That one command tells you what your server will actually present.
• Both files verify fine. As a set of certificates, order is irrelevant. This is why "it verifies, so the file must be right" is a false conclusion.
• The reversed deployment fails with hostname mismatch (code 62), not with a chain error — because the chain is fine, and the certificate being served is simply for the wrong name.

🔑 So the pre-deploy check is one line, and it is the right one:

openssl x509 -in fullchain.pem -noout -subject

If that does not print your website's name, the file is in the wrong order. Checking grep -c BEGIN tells you the count; this tells you the order. You want both.

💡 Where reversed files come from in real life: a cat written the wrong way round in a renewal script, a CA portal that offers the download as "certificate bundle" with the root first, or someone assembling the file by hand from a support email. It is almost never a tool's fault.

Yes — but not for the reason people usually give.

For path building, order is irrelevant. The client treats the certificates as a set and searches, so openssl verify accepts any order.

For the server, order is everything. nginx and Apache read the file positionally: the first certificate is the one presented as the server certificate, and everything after is the chain. Reverse the file and nginx serves the intermediate as your certificate — producing a hostname mismatch on every client, because the intermediate's name is not your site.

RFC 8446 also specifies leaf-first on the wire; modern clients tolerate a mis-ordered chain, but tolerating is not the same as correct.

The check that catches it: openssl x509 -in fullchain.pem -noout -subject. That reads only the first certificate — exactly what nginx does. If it does not print your site's name, the order is wrong.

Why this is worth knowing precisely: the resulting error blames the hostname, so people go looking at DNS and SANs. Knowing that a reversed bundle produces a hostname error saves a long detour.

-rw-r--r-- 1 zaeem zaeem 3384 Aug 20 15:40 fullchain.pem
-rw------- 1 zaeem zaeem  241 Aug 20 15:40 privkey.pem

=== find any key on this machine that is too open ===
  ✅ /home/zaeem/tls-lab/m05/app.key                    600
  ✅ /home/zaeem/tls-lab/m05/privkey.pem                600
  ✅ /home/zaeem/tls-lab/m05/int/private/int.key        600
  ✅ /home/zaeem/tls-lab/m05/root/private/root.key      600

What to read out of this.

• -rw-r--r-- for the certificate, -rw------- for the key. The certificate is public; making it 0600 achieves nothing and occasionally breaks tools that read it as a non-root user.
• Everything is 600 because umask 077 has been set at the top of every exercise since Module 01. That habit is doing real work — without it, openssl writes keys with whatever the default umask gives, which on many systems is 0644.
• The permissions on the enclosing directory matter too. A 0600 key inside a 0755 directory is fine; inside a world-writable directory it is not, because someone could replace it. Production key directories are usually 0700 root:root.

⚠️ Two exposures that permissions do not fix, and both are more common than bad permissions:

1. The key in git. Deleting the file does not remove it from history. If a key has ever been committed, it is burned — rotate it (Module 02, A2 interview question).
2. The key in a backup, a container image layer, or a config-management repo. A key baked into a Docker image is in every registry copy of that image, permanently.

Now imagine this at 500 hosts. The structural fix is that keys are generated on the host and never travel (Module 04, D3). ACME clients do this by default. If your process involves a human moving a key file between machines, that is the thing to change — not the permissions.

0600, owned by root, in a directory that is 0700 root:root. The certificate can be 0644 because it is public.

The key does not need to be readable by www-data or nginx. Both nginx and Apache start as root, read the key into memory, then drop privileges for the worker processes. Making the key readable by the worker user means any web application vulnerability that can read files can read your private key — for no benefit.

The exposures permissions do not cover, and which matter more:

• Git history — permanent. A key ever committed must be rotated, not deleted.
• Container image layers — a key baked into an image exists in every copy of that image in every registry.
• Backups and config-management repos — usually less protected than the server itself.

The structural answer: generate the key on the host that will use it and never move it. That is what ACME clients do by default, and it removes the whole class of problem rather than mitigating it.

=== 1. does the file order put the leaf first? ===
subject=C = MY, O = Zaeem Labs, CN = app.internal.test

=== 2. is the chain complete? ===
2

=== 3. does the key match the certificate? ===
MATCH

=== 4. is the key unencrypted? ===
-----BEGIN PRIVATE KEY-----

=== 5. permissions ===
600 privkey.pem
644 fullchain.pem

What to read out of this — these five checks catch nearly every deployment failure.

🔑 Notice that four of the five have completely different symptoms, and two of them do not show up until a restart that may be days later. That is why these belong in a pre-deploy gate rather than in your head. Five commands, no server needed, and they run in under a second.

💡 When you do have nginx, add the two commands it provides:

nginx -t                  # parse and validate the config - ALWAYS before a reload
nginx -T | grep ssl_      # dump the FULLY RESOLVED config, includes and all

nginx -T is the underused one. It shows what nginx actually ended up with after every include, which is how you find the stray ssl_protocols TLSv1; in a file nobody remembered was included.

The two directives that matter are ssl_certificate pointing at fullchain.pem — leaf first, then intermediates — and ssl_certificate_key pointing at the key, which should be 0600 root:root. nginx starts as root, reads the key, and drops privileges, so the workers never need access to it.

Then listen 443 ssl and http2 on (or listen 443 ssl http2 before nginx 1.25.1), and server_name for SNI-based virtual hosting.

For the rest I would take a maintained configuration from TLSRef rather than hand-writing cipher strings, and set explicitly: ssl_session_cache shared:SSL:10m — because the default is none, meaning no resumption at all — and ssl_session_tickets off unless ticket-key rotation is handled, because the default is on with a key that never rotates.

The check I would always run before reloading: nginx -t to validate, and nginx -T to dump the fully resolved config, which is how you find directives inherited from an include you had forgotten about.

And the five file-level checks — leaf first, chain count, key match, key unencrypted, permissions — because four of the five have completely different symptoms and two only appear at the next restart.

HTTP/1.1 301 Moved Permanently
Location: https://example.com/
Content-Length: 0

=== follow it and see where you land ===
https://example.com/  final=200  redirects=1

=== does it preserve the path and the hostname? ===
location: https://example.com/some/path?a=1

What to read out of this.

• 301, not 302. A permanent redirect lets the browser remember it and skip the insecure request next time. 302 means it tries HTTP again on every visit, leaving a plaintext request on the wire every time.
• redirects=1. One hop. A common misconfiguration produces http://x → https://x → https://www.x, which is two hops and two round trips before anything useful happens. %{num_redirects} is the quick way to spot it.
• The path and query string survived. $request_uri includes both. If you see Location: https://example.com/ for a request to /some/path, someone used $uri (path only, no query) or hard-coded a /, and every deep link into the site is being dropped on the floor.

🔑 The three things a good redirect does: status 301, one hop, and the full original path and query preserved. curl -sSL -w '%{url_effective} %{num_redirects}' checks all three in one command.

⚠️ And the thing a redirect cannot fix: the very first request still went out over plain HTTP, revealing the hostname and path to anyone on the path. That is what HSTS solves (Part D1) — it stops the insecure request being made at all after the first visit.

=== fingerprint of the certificate the RUNNING SERVER is serving ===
sha256 Fingerprint=3F:9A:2C:8E:5B:1D:47:F0

=== fingerprint of the certificate now ON DISK ===
sha256 Fingerprint=A1:7C:44:2B:E9:06:D3:58

=== and what the server is STILL serving ===
sha256 Fingerprint=3F:9A:2C:8E:5B:1D:47:F0

What to read out of this.

• The disk changed and the wire did not. Two different fingerprints, same moment in time. The renewal "worked" by every measure a file-based check would use.
• Nothing here is broken. The server is behaving correctly — it read the certificate at startup and has no reason to look again. openssl s_server has no reload; a real server does.
• This is exactly what happens in production, except the gap between the two fingerprints is weeks rather than seconds, and the failure arrives when the old certificate expires.

🔑 The two rules that prevent it entirely:

1. Every renewal must trigger a reload. In certbot that is a --deploy-hook, which runs only when a certificate was actually renewed:

certbot renew --deploy-hook "systemctl reload nginx"

Use --deploy-hook, not --post-hook — the latter runs on every check, reloading nginx twice a day for no reason.

2. Monitor the live endpoint, not the file. This is the same lesson as Module 05 (D3.1), and this exercise is the proof of why it matters. Compare them directly:

disk=$(openssl x509 -in /etc/ssl/shop/fullchain.pem -noout -fingerprint -sha256)
wire=$(openssl s_client -connect shop.example.com:443 -servername shop.example.com </dev/null 2>/dev/null \
       | openssl x509 -noout -fingerprint -sha256)
[ "$disk" = "$wire" ] && echo "in sync" || echo "MISMATCH - the server needs a reload"

That check takes a second and catches the outage weeks before it happens.

Almost certainly the renewal wrote a new certificate to disk and nothing reloaded the service. The running process read its certificate at startup and kept it in memory, so it carried on serving the old one until that expired.

It hides well: the renewal logs success, the file on disk is correct and current, and any monitoring that checks the file reports healthy. The renewal and the outage are separated by weeks, so nobody connects them.

The two fixes, and both are needed:

1. A deploy hook on renewal — certbot renew --deploy-hook "systemctl reload nginx". Specifically --deploy-hook, which runs only when a certificate actually changed, rather than --post-hook, which runs on every check.
2. Monitor the live endpoint, not the file. Comparing the fingerprint on disk with the fingerprint on the wire detects the drift immediately rather than at expiry.

And the detail worth adding: use reload, not restart. nginx's master process re-reads the certificate and starts new workers while old workers finish their in-flight requests, so no connections are dropped. A restart works but has no advantage.

1. SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2 — deprecated protocols enabled, TLS 1.3 missing.

TLS 1.0 and 1.1 were deprecated by RFC 8996 in 2021 and removed from browsers in 2020. There is also no -all, so this is adding to whatever was inherited. Should be SSLProtocol -all +TLSv1.2 +TLSv1.3.

2. SSLCertificateChainFile — obsolete since Apache 2.4.8 (2014).

Not broken, but it dates the config by a decade. Modern form: point SSLCertificateFile at fullchain.pem and delete the chain directive entirely.

3. Header set without always.

The HSTS header will be missing from redirects and error responses. Should be Header always set.

4. max-age=300 — five minutes.

Technically HSTS, practically useless. A browser forgets after five minutes, so almost every visit still starts with an insecure request. Production values are 31536000 (one year) or 63072000 (two). Five minutes is, however, exactly right while you are testing HSTS — see Part D1, where that is the recommended approach.

A fifth, arguable one: SSLHonorCipherOrder on has no effect in TLS 1.3 and is now generally set to off even for 1.2, since modern clients pick sensibly and often know better than the server which cipher performs best on their hardware.

🔑 What this exercise is really teaching: reading a TLS config is a skill in itself, and most of the problems you will find are age rather than error. Configs are copied forward for years, and a directive that was correct in 2016 is often the thing quietly holding a site back in 2026.

It was Apache's directive for supplying intermediate certificates separately from the leaf. It became obsolete in Apache 2.4.8, when SSLCertificateFile was extended to load intermediates from the same file — so the modern form is simply to point SSLCertificateFile at fullchain.pem.

It still works, which is why it survives in so much documentation and so many copied configs.

What it tells you when you see it: the config was written for Apache 2.4.7 or earlier — 2014 or before. That is worth noticing, because whatever else is in that file is likely to be of the same vintage: probably SSLProtocol including TLS 1.0, probably a hand-written cipher string that has not been revisited.

The general point: most problems in a TLS config are age rather than mistakes. Directives get copied forward for a decade, and nobody re-reads them because nothing is failing.

example.com    (none)
github.com     strict-transport-security: max-age=31536000; includeSubdomains; preload
google.com     strict-transport-security: max-age=31536000

What to read out of this.

• GitHub has gone all the way: one year, all subdomains, preloaded. That is a deliberate, audited commitment — every *.github.com must be HTTPS forever, and they cannot quickly change their mind.
• Google has one year but no includeSubDomains on this hostname. With an estate that size, committing every subdomain in one header would be reckless; they manage it per-host instead. A useful signal that includeSubDomains is a scope decision, not a strength decision.
• example.com sends nothing — it is a documentation domain with nothing to protect.
• Header names are case-insensitive and servers vary, which is why the grep -i matters. GitHub sends includeSubdomains with a lowercase d; that is fine, the directive is case-insensitive too.

🔑 Reading someone's HSTS header tells you how confident they are. max-age=300 means they are testing. A year plus includeSubDomains plus preload means they have audited every subdomain and accepted that they cannot easily reverse it. It is one of the few places a header reveals an organisation's operational maturity.

💡 To check whether a domain is actually in the preload list rather than merely requesting it, look it up at hstspreload.org — the preload directive is a request, not a confirmation. Sites frequently send it without ever having submitted.

A response header telling the browser to use HTTPS only for this host for max-age seconds, remembered locally and refreshed on every visit. It closes the gap a redirect leaves open: with only a redirect, every visit begins with a plaintext request that leaks the hostname and path and can be intercepted.

The critical limitation: it only helps after the first visit. The very first request to a host is still made in the clear. Preloading is what closes that.

The two things to be careful about, and both are self-inflicted:

• includeSubDomains silently covers subdomains you have forgotten. An HTTP-only internal tool becomes completely unreachable from any browser that has seen the header — no click-through — and stays that way for max-age even after you remove it.
• preload is close to irreversible. hstspreload.org states removal takes months to reach users via a Chrome update, with no guarantees for other browsers. You are committing every current and future subdomain, for years.

How I would roll it out: max-age=300 first, so mistakes undo themselves in five minutes; then a day; then add includeSubDomains only after auditing every subdomain including internal and staging; then a year; and only then consider preload.

And the nginx detail: add_header ... always, or the header is missing from redirects and error responses — exactly the responses an attacker might induce.

A site that staples:

OCSP response:
======================================
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Cert Status: good
    This Update: Aug 19 12:00:00 2026 GMT
    Next Update: Aug 26 12:00:00 2026 GMT

A site that does not:

OCSP response: no response sent

What to read out of this.

• -status is the flag that asks for a stapled response. Without it you learn nothing about stapling.
• Cert Status: good is the actual answer — the CA saying this certificate is not revoked.
• This Update and Next Update are the dates on the letter. The response is only trusted within that window, which is why the server must keep re-fetching. A stapled response past its Next Update is treated as absent.
• no response sent means stapling is not enabled, or it is enabled and quietly failing — which on nginx usually means a missing resolver.

🔑 Two things stapling buys you, and one it does not.

✅ Speed. The client does not make a separate connection to the CA mid-handshake, which on a slow network is a visible delay.

✅ Privacy. Without stapling, the client tells the CA which site it is visiting, every time. The CA becomes an unintended log of everyone's browsing.

❌ It does not make revocation reliable. A server that simply stops stapling is treated as "no information available" by nearly every client, and the connection proceeds. An attacker with a revoked certificate just does not staple. That is Module 09's subject, and it is the reason the industry's real answer to revocation became short certificate lifetimes rather than better revocation checking.

The server fetches its own OCSP response from the CA periodically and includes it in the TLS handshake, so the client does not have to contact the CA itself.

Two clear benefits: it removes a separate network round trip to the CA from the client's connection setup, and it removes a privacy leak — without stapling, the client tells the CA which site it is visiting on every connection, making the CA an unintended log of everyone's browsing.

The nginx specifics worth knowing: ssl_stapling on, ssl_stapling_verify on and ssl_trusted_certificate pointing at the intermediates — all three, because the last two default to off and absent. And a resolver directive, without which nginx cannot do the DNS lookup to reach the responder and stapling silently does nothing, with no error at all. That last one is the commonest reason someone enables stapling and sees no change.

The honest limitation: stapling improves speed and privacy but does not make revocation reliable. A stapled response is optional from the client's point of view, so an attacker holding a revoked certificate simply does not staple one, and nearly every client proceeds. Verify with openssl s_client -status.

=== TLS 1.2 resumption against a real site ===
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Reused, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Reused, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Reused, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Reused, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Reused, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256

=== does it issue session tickets? ===
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
Session-ID: A81F...

What to read out of this.

• One New then five Reused — resumption is working, exactly as in Module 06 (D3.1). If every line said New, resumption is broken, and behind a load balancer that usually means mismatched ticket keys.
• TLS session ticket lifetime hint: 7200 — the server suggests two hours. That is a hint, not a guarantee; the server can refuse to resume at any point.
• Both a ticket and a Session-ID are present. Servers commonly offer both mechanisms, and the client chooses. Turning tickets off does not turn resumption off, as long as ssl_session_cache is configured.

🔑 The diagnostic for a fleet: run this repeatedly against a load-balanced hostname. If you see New far more often than you expect, the backends are not sharing resumption state — mismatched ticket keys, or ssl_session_cache set per-server with no shared store. It is a pure performance loss and completely invisible unless you look for it.

A session ticket is the session state, encrypted with a ticket key held by the server. Anyone who obtains that key can decrypt every session resumed with it — which undermines the forward secrecy you got from ephemeral key exchange, because the ephemeral keys no longer protect resumed sessions.

The risk is entirely about rotation. nginx enables tickets by default with a key generated at startup and never rotated for the life of the process, so a long-running server can be protecting months of resumed sessions with one static key.

The two workable answers:

• Single server: ssl_session_tickets off and use ssl_session_cache instead. Simple, and resumption still works.
• A fleet: keep tickets, but manage ssl_session_ticket_key explicitly — rotate hourly, keep two or three generations so in-flight tickets still decrypt, and distribute the keys to every backend.

The operational failure that comes with the same setting: behind a load balancer, each server generating its own key means a client resuming against a different backend silently falls back to a full handshake. Resumption looks enabled and mostly does not happen — a pure performance loss that nothing alerts on. openssl s_client -reconnect against the load-balanced name is how you spot it.

#!/usr/bin/env bash
# tlsdeploy-check <certdir> [live-hostname]
#   certdir must contain fullchain.pem and privkey.pem
set -uo pipefail
dir="${1:?usage: tlsdeploy-check <certdir> [hostname]}"; host="${2:-}"
FC="$dir/fullchain.pem"; PK="$dir/privkey.pem"
fail=0
ok()   { printf '  ✅ %-26s %s\n' "$1" "$2"; }
bad()  { printf '  ❌ %-26s %s\n' "$1" "$2"; fail=1; }
warn() { printf '  ⚠️  %-26s %s\n' "$1" "$2"; }

printf '\n  PRE-DEPLOY  (%s)\n  %s\n' "$dir" "$(printf '=%.0s' {1..58})"

[ -r "$FC" ] || { bad "fullchain.pem" "missing or unreadable"; }
[ -r "$PK" ] || { bad "privkey.pem"   "missing or unreadable"; }
[ "$fail" -eq 1 ] && exit 1

# 1. leaf first?
subj=$(openssl x509 -in "$FC" -noout -subject 2>/dev/null | sed 's/^subject=//')
bc=$(openssl x509 -in "$FC" -noout -ext basicConstraints 2>/dev/null | tail -1 | tr -d ' ')
case "$bc" in
  *CA:TRUE*) bad "bundle order" "FIRST cert is a CA - file is REVERSED ($subj)" ;;
  *)         ok  "bundle order" "leaf first: $subj" ;;
esac

# 2. chain complete?
n=$(grep -c 'BEGIN CERTIFICATE' "$FC")
if [ "$n" -lt 2 ]; then
  bad "chain length" "$n certificate(s) - intermediates MISSING"
else
  ok "chain length" "$n certificates"
fi

# 3. key matches cert?
if diff -q <(openssl x509 -in "$FC" -noout -pubkey 2>/dev/null) \
           <(openssl pkey -in "$PK" -pubout 2>/dev/null) >/dev/null 2>&1; then
  ok "key ↔ certificate" "match"
else
  bad "key ↔ certificate" "MISMATCH - the service will refuse to start"
fi

# 4. key usable unattended?
case "$(head -1 "$PK")" in
  *ENCRYPTED*) bad "key encryption" "passphrase-protected - service will hang at boot" ;;
  *)           ok  "key encryption" "unencrypted, starts unattended" ;;
esac

# 5. key permissions
perms=$(stat -c '%a' "$PK" 2>/dev/null || stat -f '%Lp' "$PK")
case "$perms" in
  600|400) ok  "key permissions" "$perms" ;;
  *)       bad "key permissions" "$perms - must be 600 or 400" ;;
esac

# 6. expiry, across the WHOLE bundle
d=0; tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
( cd "$tmp" && csplit -sz -f c- -b '%02d.pem' "$FC" '/BEGIN CERTIFICATE/' '{*}' )
for c in "$tmp"/c-*.pem; do
  end=$(openssl x509 -in "$c" -noout -enddate | cut -d= -f2)
  if   ! openssl x509 -in "$c" -noout -checkend 0       >/dev/null; then bad  "expiry depth=$d" "EXPIRED $end"
  elif ! openssl x509 -in "$c" -noout -checkend 2592000 >/dev/null; then warn "expiry depth=$d" "under 30 days: $end"
  else ok "expiry depth=$d" "$end"; fi
  d=$((d+1))
done

[ -z "$host" ] && { printf '\n  %s\n\n' "$([ $fail -eq 0 ] && echo 'PRE-DEPLOY PASSED' || echo 'PRE-DEPLOY FAILED')"; exit $fail; }

# ---------- post-deploy ----------
printf '\n  POST-DEPLOY  (%s)\n  %s\n' "$host" "$(printf '=%.0s' {1..58})"
S="openssl s_client -connect ${host}:443 -servername ${host}"

wire=$($S </dev/null 2>/dev/null | openssl x509 -noout -fingerprint -sha256 2>/dev/null)
disk=$(openssl x509 -in "$FC" -noout -fingerprint -sha256)
if [ -z "$wire" ]; then
  bad "live connection" "could not retrieve a certificate"
elif [ "$wire" = "$disk" ]; then
  ok "disk ↔ wire" "in sync"
else
  bad "disk ↔ wire" "MISMATCH - the service has NOT been reloaded since renewal"
fi

nsent=$($S -showcerts </dev/null 2>/dev/null | grep -c 'BEGIN CERTIFICATE')
[ "$nsent" -ge 2 ] && ok "chain served" "$nsent certificates" \
                   || bad "chain served" "$nsent - browsers may hide this; curl/Java will fail"

hsts=$(curl -sS -o /dev/null -D - "https://$host/" 2>/dev/null \
       | grep -i '^strict-transport-security' | tr -d '\r')
if [ -z "$hsts" ]; then
  warn "HSTS" "no header"
else
  ma=$(printf '%s' "$hsts" | grep -oiE 'max-age=[0-9]+' | cut -d= -f2)
  [ "${ma:-0}" -ge 31536000 ] && ok "HSTS" "max-age=$ma" || warn "HSTS" "max-age=$ma (under 1 year)"
fi

staple=$($S -status </dev/null 2>/dev/null | grep -m1 'OCSP response')
case "$staple" in
  *"no response sent"*) warn "OCSP stapling" "not stapling" ;;
  "")                   warn "OCSP stapling" "not stapling" ;;
  *)                    ok   "OCSP stapling" "stapled" ;;
esac

printf '\n  %s\n\n' "$([ $fail -eq 0 ] && echo 'ALL CHECKS PASSED' || echo 'CHECKS FAILED')"
exit $fail

Try it:

chmod +x tlsdeploy-check
cd ~/tls-lab/m05
./tlsdeploy-check .                       # pre-deploy only
./tlsdeploy-check . example.com           # both halves

The five design decisions worth understanding.

1. It detects a reversed bundle by reading basicConstraints, not by name matching. If the first certificate says CA:TRUE, the file is reversed — that works for any site without knowing what the hostname should be. Comparing against an expected name would need configuration; this does not.

2. Expiry is checked at every depth, by splitting the bundle. An expired intermediate behind a valid leaf is invisible to any check that only reads the first certificate — and openssl x509 -in fullchain.pem reads exactly that one.

3. The disk-versus-wire comparison is the point of the post-deploy half. It is the only check that catches "renewed but never reloaded", and it catches it weeks before the expiry outage rather than during it.

4. HSTS and stapling are warnings, not failures. Neither is required for a working deployment, and a gate that fails the build for a missing HSTS header will simply be disabled by whoever is trying to ship. Reserve failures for things that are actually broken.

5. It exits non-zero on pre-deploy failure so a pipeline can gate on it. All six pre-deploy checks run on files alone — no server, no network — so this can run in CI before anything is deployed anywhere. Four of the six failures it catches have completely different symptoms, and two of them would not surface until a restart days later.

Where this goes next: add revocation status (Module 09), certificate transparency monitoring (Module 11), and full protocol and cipher auditing (Module 13). Together with certinfo, tlsinfo and tlsdiag from Modules 03, 06 and 07, this is most of a real certificate operations toolkit.

--with-http_ssl_module
--with-http_v2_module
--with-openssl=/build/nginx/openssl-3.0.13

What to read out of this.

• nginx -V is the honest answer to "does this nginx support X". The documentation describes nginx in general; -V describes your build. A directive documented on nginx.org will fail if the module was not compiled in.
• --with-openssl= tells you which OpenSSL nginx is linked against, which is not necessarily the openssl binary on your $PATH. That matters for TLS 1.3, for ssl_early_data, and — as Module 06 (D4) showed — for whether post-quantum key exchange is available at all.
• --with-http_v2_module confirms HTTP/2 support. Without it, http2 on; is an unknown directive and nginx refuses to start.

💡 nginx does not ship its directive reference offline, unlike OpenSSL's man pages. The practical answer is to keep nginx.org/en/docs bookmarked, and to remember that nginx -T tells you what your running configuration resolved to — which is usually the question you actually have.

fullchain.pem (leaf then intermediates — public, 0644) and privkey.pem (secret, 0600 root:root). A separate chain.pem of intermediates only is needed for OCSP stapling.

The root is never served: a client that trusts it has it, and one that does not will not trust it because a server sent it.

Not to the client's path building — it searches, so any order verifies. It matters to the server, which reads the file positionally: nginx treats the first certificate as the leaf. Reverse it and nginx serves the intermediate as your certificate, producing hostname mismatch on every client.

Check with openssl x509 -in fullchain.pem -noout -subject — that reads only the first certificate, exactly as nginx does.

No. nginx and Apache start as root, read the key into memory, then drop privileges. The workers never need it. Making it worker-readable means any file-read vulnerability in the web application exposes the private key, for no benefit.

0600 root:root, in a 0700 root:root directory.

The service was never reloaded. It read its certificate at startup and kept serving the old one from memory until it expired. The renewal logged success, the file on disk was correct, and file-based monitoring saw nothing.

Fix: certbot renew --deploy-hook "systemctl reload nginx", and monitor the live endpoint, comparing the fingerprint on the wire with the one on disk.

Reload. nginx's master re-reads config and certificates, starts new workers, and lets old workers finish in-flight requests before exiting — no dropped connections. Restart achieves the same certificate update while dropping connections, so there is no reason to prefer it.

nginx -s reload or systemctl reload nginx; Apache is apachectl -k graceful.

302 is temporary, so the browser tries HTTP again on every visit — leaving a plaintext request on the wire each time. 301 lets it remember.

$server_name is the first name in the server_name list, so a request for www.example.com gets silently redirected to example.com. $host preserves what the client asked for. And use $request_uri, not $uri, so the query string survives.

/.well-known/acme-challenge/. The CA fetches it over plain HTTP during an HTTP-01 challenge. If HTTPS is broken or the certificate has expired, redirecting there means renewal depends on the very thing that is broken — and you cannot renew your way out of it.

A header telling the browser to use HTTPS only for this host for max-age seconds. It closes the gap a redirect leaves — with only a redirect, every visit starts with a plaintext request.

includeSubDomains covers subdomains you have forgotten, making HTTP-only internal tools unreachable with no click-through, for the full max-age even after you remove the header. preload is close to irreversible — removal takes months to reach users and is not guaranteed across browsers.

Roll out with max-age=300 first so mistakes undo themselves in five minutes.

It buys speed — no separate client connection to the CA — and privacy, because otherwise the client tells the CA which site it is visiting on every connection.

It does not make revocation reliable: a stapled response is optional, so an attacker with a revoked certificate simply does not staple one and clients proceed.

nginx needs ssl_stapling on, ssl_stapling_verify on, ssl_trusted_certificate pointing at the intermediates, and a resolver — without which it silently does nothing.

Tickets are encrypted with a server-held key; whoever obtains it can decrypt every session resumed with it, undermining forward secrecy. nginx's default is tickets on with a key generated at startup and never rotated.

Single server: turn tickets off and use ssl_session_cache. A fleet: manage ssl_session_ticket_key explicitly with hourly rotation and a couple of generations retained — and note that unmanaged keys behind a load balancer break resumption entirely, silently.

ssl_session_cache none — no session-ID resumption at all out of the box, a pure performance loss fixed by one line. And ssl_session_tickets on — resumption is enabled by default, but via the mechanism with the forward-secrecy caveat and an unrotated key.

So the defaults give you the mechanism with the security problem and not the one without it. Set both explicitly.

That the config was written for Apache 2.4.7 or earlier — the directive became obsolete in 2.4.8, released in 2014, when SSLCertificateFile was extended to load intermediates from the same file.

It still works, so it is not urgent. But it dates the whole file, and whatever else is in there is likely the same vintage — probably TLS 1.0 enabled and a hand-written cipher string nobody has revisited.

Question selection was cross-referenced against publicly published 2026 SSL/TLS and web-server interview question sets, then rewritten and deepened:

• InterviewPrep — Top 25 SSL Certificate Interview Questions and Answers
• ClimbTheLadder — 10 SSL Certificate Interview Questions and Answers
• ClimbTheLadder — 15 PKI Interview Questions and Answers
• Devinterview.io — Web Security interview questions for 2026

Every configuration directive, default value and version note in this module was verified against the official documentation — nginx ngx_http_ssl_module and Apache mod_ssl — rather than written from memory. That includes the two surprising nginx defaults (ssl_session_cache none and ssl_session_tickets on), the fact that ssl_prefer_server_ciphers already defaults to off, and Apache's own wording that SSLCertificateChainFile "became obsolete with version 2.4.8". The HSTS preload requirements and the removal warning are quoted from hstspreload.org.

The file-level exercises — chain order, key matching, permissions, and the disk-versus-wire fingerprint comparison — were executed on OpenSSL 3.0.13. nginx itself was not run for this module, so the configuration blocks are documentation-verified rather than execution-verified; the exercises are deliberately written to need only openssl.

Answers were rewritten and deepened rather than reproduced — published versions are usually correct but shallow, and the added operational detail is what actually differentiates a candidate in the room.