🔧Module 13 — Troubleshooting & Operating TLS at Scale

good       subject=CN = good.lab.test notAfter=Oct 26 07:35:58 2026 GMT 
expiring   subject=CN = soon.lab.test notAfter=Aug 31 07:35:58 2026 GMT 
expired    subject=CN = dead.lab.test notAfter=Aug 24 07:35:58 2026 GMT 
wrongname  subject=CN = other.lab.test notAfter=Oct 26 07:35:59 2026 GMT

What to read out of this — your dates will be your own, but the relationships must hold.

• expired has a notAfter in the past. That is the whole trick of this lab: openssl ca -startdate/-enddate lets you issue a certificate that was born and died before today. Module 07 introduced this because openssl x509 -req cannot do it on OpenSSL 3.0.
• expiring is a few days out, which is the state you actually want your monitoring to catch. By the time a certificate is expired it is already an incident.
• wrongname has a SAN that does not include localhost or 127.0.0.1. That is what will make it fail when we ask for a different name in A3.
• Every certificate is signed by the intermediate, not the root. That matters: it is what makes the 9005 trap possible.

🔑 Keeping a deliberately broken lab is a professional habit, not a training exercise. When you are trying to work out whether your monitoring, your alerting or your runbook actually detects something, you need a thing that is genuinely broken to point it at. Teams that have one find gaps on a Wednesday morning. Teams that do not find them during an incident.

9001  HTTP 200
9003  curl: (60) SSL certificate problem: certificate has expired
9004  curl: (60) SSL: no alternative certificate subject name matches target host name 'good.lab.test'
9005  curl: (60) SSL certificate problem: unable to get local issuer certificate

What to read out of this — three different faults, three different sentences, no interpretation needed.

• All three failures are curl: (60). Exit code 60 means peer certificate cannot be authenticated. The exit code tells you it is a certificate problem; the text tells you which one. That split is worth internalising, because scripts see the code and humans see the text.
• certificate has expired — nothing else to investigate. Renew it.
• no alternative certificate subject name matches target host name — the certificate is fine, it is just not for this name. Either the wrong vhost answered, or nobody reissued after a hostname was added.
• unable to get local issuer certificate — the chain is incomplete. Module 08's trap: someone deployed the leaf instead of the full chain.

🔑 This is the single highest-value habit in the module. Before opening a config file, before reading a runbook, before asking anyone anything — run one curl and read the sentence. It is right about which of the five buckets you are in most of the time, and it costs ten seconds.

⚠️ Do not use -k (or --insecure) to "check if the site is up". It disables exactly the verification that is failing, so it always succeeds and tells you nothing. It is the TLS equivalent of turning off the smoke alarm to stop the noise. If you need to see the page anyway, use -k after you have recorded what the error was.

💡 Now imagine this at 500 hosts. The same loop, reading hostnames from a file, is a complete estate check — and Part C builds exactly that. The reason it scales is that the output is one line per host with the fault named in it, which means you can grep it.

9001  good.lab.test   2 certs  Verify return code: 0 (ok)
9002  soon.lab.test   2 certs  Verify return code: 0 (ok)
9003  dead.lab.test   2 certs  Verify return code: 10 (certificate has expired)
9004  good.lab.test   2 certs  Verify return code: 62 (hostname mismatch)
9005  good.lab.test   1 certs  Verify return code: 21 (unable to verify the first certificate)

=== the chain a healthy server sends:
Certificate chain
 0 s:CN = good.lab.test
   i:CN = Estate Issuing CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 26 14:29:20 2026 GMT; NotAfter: Oct 25 14:29:20 2026 GMT
 1 s:CN = Estate Issuing CA
   i:CN = Estate Root CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 26 14:29:20 2026 GMT; NotAfter: Aug 25 14:29:20 2031 GMT
---
=== the chain the broken one sends:
Certificate chain
 0 s:CN = good.lab.test
   i:CN = Estate Issuing CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 26 14:29:20 2026 GMT; NotAfter: Oct 25 14:29:20 2026 GMT
---

What to read out of this — the certificate count column diagnoses 9005 on its own, before you read a single error.

• Four of the five sent 2 certificates. One sent 1. That is the fault, visible as a number. The healthy chain ends with i:CN = Estate Root CA — a certificate whose issuer is not in the list, which is correct: the root is never sent.
• 9002 says 0 (ok) even though it expires in three days. TLS has no concept of "nearly expired". Validation will keep saying ok right up until the second it starts saying 10, which is precisely why expiry needs monitoring rather than checking.
• 9004 is code 62, and note it only appears because of -verify_hostname. Without that flag s_client does not check the name and would report 0 (ok) — a genuinely dangerous default when you are trying to reproduce a browser's behaviour.
• 9005's chain stops at entry 0. The leaf claims i:CN = Estate Issuing CA and no such certificate follows. The server is asking the client to already have something it has no reason to have.

🔑 Read the s: and i: lines as a linked list. Each entry's i: should equal the next entry's s:. Where that stops matching, or stops early, is where the chain is broken — and you can see it without understanding a single thing about cryptography.

⚠️ -verify_hostname is not the default and never has been. Plain openssl s_client -connect host:443 will happily report Verify return code: 0 (ok) for a certificate issued to a completely different name. If you are checking what a browser would do, you must pass it. This has misled a great many people into declaring a certificate healthy shortly before their users could not reach it.

💡 Now imagine this at 500 hosts. The loop is already the right shape — the only change is where the target list comes from. Part C turns it into an estate sweep, and the reason it works is that both signals (certificate count and verify code) are single values you can compare, sort and alert on.

Run one curl against it and read the error text. Almost every TLS failure is one of five things — expired certificate, wrong hostname, incomplete chain, the wrong certificate being served, or it is not a TLS problem at all — and curl's message usually names which one in plain English. That takes ten seconds and rules out most of the space.

If I need more detail I escalate to openssl s_client -connect host:443 -servername host -verify_hostname host, which shows the full chain the server sent and a numbered verify code.

The detail worth adding: mention that you would not reach for -k or --insecure to check whether the site is up, because it disables the very check that is failing and tells you nothing. And say that you record the error text before doing anything else — in an incident the fastest way to lose the diagnosis is to start changing things before anyone has written down what the original symptom was.

Error 20, unable to get local issuer certificate, means the chain reached a certificate whose issuer is not in your trust store — a client-side problem, typically a missing internal root or a container without ca-certificates installed. Error 21, unable to verify the first certificate, means the server sent a leaf without the intermediate that signed it, so the chain cannot be built at all — a server-side problem, fixed by deploying fullchain.pem instead of cert.pem.

You tell them apart by counting the certificates in the s_client chain listing. One certificate from a leaf that claims an intermediate issuer means 21.

The detail worth adding: the practical trap is that curl reports both as "unable to get local issuer certificate", so the distinction is invisible until you use s_client. Getting it backwards sends you off editing trust stores when the real fix is one cat on someone else's server — it is one of the most reliably time-wasting mistakes in TLS operations, and knowing the difference is a strong signal you have actually done this.

Most likely the hostname. s_client does not check the certificate name against the host you connected to unless you pass -verify_hostname (or -verify_hostname via -servername plus the flag). So a certificate for an entirely different name validates as 0 (ok) while every browser rejects it.

The other common causes are things browsers enforce and OpenSSL does not: Certificate Transparency policy (Module 11), a CA the browser distrusts but the OS still trusts, or the browser using its own root store rather than the system one.

The detail worth adding: the general point is that s_client validates a chain, browsers enforce a policy, and the gap between those two is where the confusing incidents live. Saying "I'd re-run with -verify_hostname first, then check SCT count and the issuer against the Chrome Root Store" shows you know the browser is doing strictly more work than OpenSSL is.

Testing SSL server 127.0.0.1 on port 9001 using SNI name good.lab.test

  SSL/TLS Protocols:
SSLv2     disabled
SSLv3     disabled
TLSv1.0   disabled
TLSv1.1   disabled
TLSv1.2   enabled
TLSv1.3   enabled

  Supported Server Cipher(s):
Preferred TLSv1.3  128 bits  TLS_AES_128_GCM_SHA256        Curve 25519 DHE 253
Accepted  TLSv1.3  256 bits  TLS_AES_256_GCM_SHA384        Curve 25519 DHE 253
Accepted  TLSv1.3  256 bits  TLS_CHACHA20_POLY1305_SHA256  Curve 25519 DHE 253
Preferred TLSv1.2  256 bits  ECDHE-RSA-AES256-GCM-SHA384   Curve 25519 DHE 253
Accepted  TLSv1.2  256 bits  ECDHE-RSA-CHACHA20-POLY1305   Curve 25519 DHE 253
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-GCM-SHA256   Curve 25519 DHE 253
Accepted  TLSv1.2  256 bits  ECDHE-RSA-AES256-SHA384       Curve 25519 DHE 253
Accepted  TLSv1.2  256 bits  ECDHE-RSA-CAMELLIA256-SHA384  Curve 25519 DHE 253
Accepted  TLSv1.2  256 bits  ECDHE-RSA-AES256-SHA          Curve 25519 DHE 253
Accepted  TLSv1.2  128 bits  ECDHE-RSA-AES128-SHA          Curve 25519 DHE 253
Accepted  TLSv1.2  256 bits  AES256-GCM-SHA384            
Accepted  TLSv1.2  256 bits  AES256-CCM8                  
Accepted  TLSv1.2  256 bits  AES256-CCM                   
Accepted  TLSv1.2  128 bits  AES128-GCM-SHA256            

What to read out of this — look at the last four lines, and notice what is missing from them.

• ECDHE-RSA-... lines end with Curve 25519 DHE 253. The AES256-GCM-SHA384 lines end with nothing. That blank is the whole finding. Those cipher suites have no ephemeral key exchange, which means no forward secrecy (Module 06): record the traffic today, steal the server key next year, and you can decrypt it retroactively.
• This is a default nginx configuration. Nobody chose to enable those. They are simply what OpenSSL offers when you do not restrict ssl_ciphers, and they are what a scanner finds on an enormous number of production servers.
• s_client would never have shown you this. It negotiated TLS_AES_256_GCM_SHA384 and looked perfect. The weak options only appear when something asks for them specifically.
• Preferred versus Accepted tells you the server's own order. With ssl_prefer_server_ciphers on the client gets the server's first choice; without it, a client can pick any Accepted line — including the ones with no forward secrecy.

🔑 The lesson generalises well beyond ciphers: "what did we negotiate" and "what would we accept" are different questions, and only the second one is a security property. An attacker never picks your preferred option. They pick your weakest permitted one.

💡 The fix, and where to get it. Do not hand-write a cipher list — the syntax is unforgiving and the advice goes stale. Use the TLS configuration generator from Module 08, pick the Intermediate profile unless you have a specific reason, and paste what it gives you. Re-check it once a year, because the recommendations do move.

Because they answer different questions. s_client performs one handshake and reports what the two ends negotiated — which is normally the strongest option both support, so it looks good almost regardless of configuration. sslscan enumerates the full set the server is willing to accept, protocol by protocol and cipher by cipher.

The risk lives in the second set. A server can negotiate TLS 1.3 with a modern client and still accept a cipher suite with no forward secrecy from an older one.

The detail worth adding: give the concrete example, because it is real and it is on a lot of servers. A default nginx build accepts static-RSA suites like AES256-GCM-SHA384 alongside the ECDHE ones. There is no ephemeral key exchange, so traffic recorded today is decryptable if the server key is compromised later. Nobody enabled it deliberately — it is simply the OpenSSL default when ssl_ciphers is not set, and it is invisible to any check that only makes one connection.

Testing server defaults (Server Hello) 
TLS extensions               "server name/#0" "supported_groups/#10" "EC point formats/#11"
                             "encrypt-then-mac/#22" "extended master secret/#23" "session ticket/#35"
                             "supported versions/#43" "key share/#51" "renegotiation info/#65281"
Session Ticket RFC 5077 hint 300 seconds, session tickets keys seems to be rotated < daily
Session Resumption           tickets: yes, ID: no
TLS 1.3 early data support   no early data offered
TLS clock skew               Random values, no fingerprinting possible 
Client Authentication        none
Signature Algorithm          SHA256 with RSA
Server key size              RSA 2048 bits (exponent is 65537)
Server key usage             Digital Signature, Key Encipherment
Server extended key usage    TLS Web Server Authentication
Serial                       1000   NOT ok: length should be >= 64 bits entropy (is: 2 bytes)
Fingerprints                 SHA1 F3FFA748A0CEE80BB222D1894E8BF6F8013C860C
                             SHA256 C57D0358C89CA0F64DA732618E91290CF4181959C8282DE3F8802596E508956A
Common Name (CN)             good.lab.test 
subjectAltName (SAN)         good.lab.test localhost 127.0.0.1 
Trust (hostname)             Ok via SAN and CN (same w/o SNI)
Chain of trust               NOT ok (chain incomplete)
EV cert (experimental)       no 
Certificate Validity (UTC)   expires < 60 days (59) (2026-08-26 14:29 --> 2026-10-25 14:29)
ETS/"eTLS", visibility info  not present

What to read out of this — three findings you did not go looking for, in a run that took about fifteen seconds.

• Chain of trust NOT ok (chain incomplete). It found the 9005 fault and said so in English, without you needing to count certificates. This is the difference between a scanner and a debugger: it knows what to check.
• Serial 1000 NOT ok: length should be >= 64 bits entropy (is: 2 bytes). Nobody asked about serial numbers. This is the CA/Browser Forum requirement from Module 11 (Ballot 164): serials must carry at least 64 bits of entropy, because predictable serials once enabled certificate-forgery attacks on weak hash functions. Our lab CA counts 1000, 1001, 1002, which a public CA is forbidden to do.
• Certificate Validity expires < 60 days (59) — flagged as a warning even though nothing is wrong yet. That is the right behaviour, and the opposite of the 0 (ok) that s_client gave for a certificate three days from death.
• Client Authentication none connects to Module 12 — it tells you whether the endpoint asks for a client certificate, which is a fast way to confirm an mTLS rollout actually reached a host.

🔑 The reason to run this on a schedule rather than during an outage. Everything above is true right now on a server that is working perfectly. Scanners find the problems you do not yet have — and the whole point of an operations practice is to shorten the list of problems you find out about from your users.

💡 --add-ca root.crt is doing real work here. Without it, testssl.sh does not trust your internal CA and reports the chain as untrusted for a completely different reason, drowning the real finding. Any scan of an internal estate needs the internal root passed in, and forgetting it produces a page of alarming red that means nothing.

This is a full run, so it takes a minute or two. The last block is the one you want:

Rating (experimental) 
Rating specs (not complete)  SSL Labs's 'SSL Server Rating Guide' (version 2009r from 2025-05-16)
Specification documentation  https://github.com/ssllabs/research/wiki/SSL-Server-Rating-Guide
Protocol Support (weighted)  100 (30)
Key Exchange     (weighted)  90 (27)
Cipher Strength  (weighted)  90 (36)
Final Score                  93
Overall Grade                A+
Done 2026-08-26 14:31:40 [  78s] -->> 127.0.0.1:9001 (good.lab.test) <<--

What to read out of this — an A+ on a server with no public DNS, no public IP, and a private CA.

• The three category scores are the SSL Labs weights, 30/30/40, applied locally. 100 (30) means full marks on protocol support, contributing its whole 30 points.
• Key Exchange and Cipher Strength scored 90, not 100 — those are the non-forward-secret suites B1 found. The grade is A+ and there is still something to fix, which is a useful reminder that a good grade is a floor, not a finish line.
• (experimental) and (not complete) are honest labels. testssl.sh's implementation of the rating is close but not identical to Qualys's. Treat it as "roughly what SSL Labs would say", which is exactly what you need for an internal service.
• 78 seconds. That is why this belongs in a nightly job or a CI stage, not in an incident.

🔑 This solves a genuinely awkward problem. "Get us to an A on SSL Labs" is a common instruction, and it is normally impossible for internal services — they are not reachable, and you would not want them published if they were. Running the same rating locally gives you the number your stakeholder is asking for, on a service the public grader can never see, and without disclosing a single hostname.

💡 Which tool for which job, in one line each. curl — is it broken, and how? s_client — show me the chain and the verdict. sslscan — what would you accept? testssl.sh — audit everything and give me a grade. SSL Labs — an external, authoritative grade for something already public. They are not competitors; they are different magnifications.

Run the scanner locally. sslscan enumerates the protocols and ciphers the server will accept, and testssl.sh does a full audit — certificate, chain, protocols, vulnerabilities, headers — and will even produce an SSL Labs-style grade using the published rating guide. Pass the internal root with --add-ca or nothing will validate.

SSL Labs itself cannot help, because it is a public service that has to reach the server from the internet.

The detail worth adding: the point that usually lands is that testssl.sh implements the SSL Labs rating guide, so when someone says "we need an A", you can give them that number for a service the public grader could never see. And it avoids the disclosure problem — submitting a hostname to SSL Labs publishes it on a public results board by default, which is the same internal-name leak Certificate Transparency causes.

Supporting TLS 1.0 or 1.1, not supporting forward secrecy, not supporting AEAD cipher suites, or using DH parameters weaker than 2048 bits. Any of those caps the grade at B no matter how well the site scores otherwise. Certificate problems — expired, self-signed, untrusted — are an outright F, and a name mismatch or untrusted certificate produces the special M and T marks instead of a grade.

The score itself is Protocol Support 30%, Key Exchange 30%, Cipher Strength 40%.

The detail worth adding: the useful framing is that the caps matter far more than the score. Teams tune cipher lists chasing points when a single legacy protocol or one non-forward-secret suite is holding the grade down regardless. And be honest about the limit of the exercise: the grade measures configuration, not security. An A+ site with an unmonitored certificate that expires on Saturday is worse off than a B site that renews automatically.

Only scan what you are responsible for. A scan is thousands of probing connections; against infrastructure you do not own it is unauthorised scanning, indistinguishable from an attack in the target's logs, and unlawful in many jurisdictions whatever your intent. That applies to local tools as much as to hosted ones.

Even on your own estate, scans are noisy — they can trip IDS rules and fill logs — so tell whoever runs the monitoring before you start, and prefer a maintenance window for a full run.

The detail worth adding: mention the disclosure angle, since people rarely think of it. Public graders publish results by default, so submitting admin-staging.company.com puts that hostname on an indexed public page forever. testssl.sh --sneaky reduces the traces you leave in a target's logs, and it is worth saying plainly that reducing traces is not the same as having permission — anyone who offers --sneaky as the answer to the authorisation question has answered the wrong one.

EXPIRED dead.lab.test:9003         -3d  Aug 24 07:35:58 2026 GMT
   WARN soon.lab.test:9002          3d  Aug 31 07:35:58 2026 GMT
     ok other.lab.test:9004        59d  Oct 26 07:35:59 2026 GMT
     ok good.lab.test:9001         89d  Nov 25 07:36:18 2026 GMT
     ok good.lab.test:9005         89d  Nov 25 07:36:18 2026 GMT

What to read out of this — and then look very carefully at the last line, because it is a lie.

• Sorting by days remaining is the whole design. A monitoring output nobody can scan is a monitoring output nobody reads. The thing needing attention is always the top line.
• -3d is more useful than "EXPIRED". How long it has been broken tells you whether this is an active incident or a decommissioned host nobody removed from the list.
• 9005 says ok, 89 days — and it is completely unusable. Its chain is broken; no client can connect to it at all. Expiry monitoring is blind to every fault that is not a date.
• 9004 says ok too, and it serves a certificate for the wrong name. Also invisible here.

🔑 This is the most important thing in Part C. Most organisations' entire TLS monitoring is an expiry check, and expiry is only one of the five failure modes from Part A. A broken chain, a wrong name, a weak protocol and a mis-served certificate all report ok forever. That gap is exactly what the capstone in Part E closes.

💡 In production, use something built for it. The shape above is right, but a shell loop has no history, no alert routing and no deduplication. Prometheus's blackbox_exporter exposes probe_ssl_earliest_cert_expiry as a Unix timestamp, and the standard alert is:

probe_ssl_earliest_cert_expiry - time() < 7*24*3600

Set that threshold longer than your renewal cycle plus your worst-case deployment time. Seven days is a common default and it is too short for anything needing a change window — if renewal is monthly and a release takes a week, alert at 21 days, not 7.

=== SNI good.lab.test :
subject=CN = good.lab.test notAfter=Nov 25 07:36:18 2026 GMT 
=== SNI other.lab.test:
subject=CN = other.lab.test notAfter=Oct 26 07:35:59 2026 GMT 
=== no SNI at all     :
subject=CN = good.lab.test notAfter=Nov 25 07:36:18 2026 GMT

What to read out of this — one IP, one port, three questions, two different certificates.

• The certificate you get depends on the name you send. Same socket, different answer. Nothing about the connection changed except one string in the ClientHello.
• The third line is the dangerous one. With no SNI you silently got the default_server certificate. It is valid, it expires in 89 days, and it says nothing whatsoever about other.lab.test — which expires 30 days sooner. A monitor without SNI would report other.lab.test as healthy for a month after it died.
• -noservername is how you reproduce that bug deliberately. Worth keeping: when a monitor disagrees with reality, this is the first thing to test.
• In nginx, default_server decides who answers a nameless request. If you do not declare one, the first matching listen block wins — which means the answer can change when someone adds an unrelated vhost.

🔑 Two rules for monitoring anything behind a load balancer, and they matter more than any tool choice. One — always send SNI. Every check, every time, explicitly. Two — check each node by IP, with the right SNI, not just the DNS name. openssl s_client -connect 10.0.0.7:443 -servername www.example.com is the shape; testssl.sh --ip does the same thing. Checking the name only ever tests wherever DNS happened to send you, which on a bad day is the one node that is fine.

💡 Now imagine this at 500 hosts. Behind eight load-balancer nodes, a rollout that fails on one node leaves a certificate that is stale for one connection in eight. Users report an intermittent, unreproducible error; your monitoring is green because it keeps landing on a good node; and the only way anyone finds it is by checking every node individually. Build that into the sweep from day one — it is very hard to retrofit during an incident.

Read the expiry from the live connection, not from files on disk, because what is deployed and what is being served can differ. In practice that means Prometheus with blackbox_exporter, alerting on probe_ssl_earliest_cert_expiry - time() < <threshold>, or an equivalent scheduled check that connects and reports days remaining.

Alert on days remaining rather than on failure, and set the threshold longer than your renewal cycle plus your worst-case deployment time — seven days is a common default and is too short if a change needs a release window.

The detail worth adding: say clearly what expiry monitoring does not catch, because that is the part interviewers are usually probing for. A broken chain, a certificate for the wrong name, an obsolete protocol and a certificate that was never actually reloaded all report perfectly healthy to an expiry check, forever. Expiry is one of five common failure modes and it is the only one most teams watch.

Almost always the monitor and the users are not talking to the same thing. The likely causes, in the order I would check them: the monitor is not sending SNI, so it gets the default vhost's certificate rather than the real one; the monitor resolves the DNS name and hits one node while users hit another that was missed by a rollout; or the monitor tests the CDN edge while the failing certificate is on the origin behind it.

I would reproduce with openssl s_client -connect <node-ip>:443 -servername <hostname> against each node individually, and compare the certificate fingerprints.

The detail worth adding: the fix is a monitoring design rule, not a one-off diagnosis — always send SNI, and check every node by IP rather than only the DNS name. A stale certificate on one node out of eight produces exactly this signature: intermittent user reports, green dashboards, and nothing reproducible, because your check keeps landing on a healthy node.

1. serving now       : A0:65:35:22:6C:A3:0E:21:61:CF
2. new file on disk  : D4:DF:64:05:5C:9C:A3:26:1D:5B
3. STILL serving     : A0:65:35:22:6C:A3:0E:21:61:CF   <-- files replaced, nothing reloaded yet
4. reload:
   [emerg] SSL_CTX_use_PrivateKey("/tmp/m13/good.key") failed
   (SSL: error:05800074:x509 certificate routines::key values mismatch)
5. after reload      : A0:65:35:22:6C:A3:0E:21:61:CF
6. what nginx -t says:
   [emerg] SSL_CTX_use_PrivateKey("/tmp/m13/good.key") failed
   (SSL: error:05800074:x509 certificate routines::key values mismatch)
   nginx: configuration file .../nginx.conf test failed

What to read out of this — the rotation failed twice, in two different ways, and the service never stopped working.

• Step 3 is the first failure, and it is the ordinary one. The file on disk is new. The server is serving the old certificate. Any check that reads good-fullchain.crt would report success. Only a check that connects sees the truth.
• Step 4 is the second failure, and it is much more interesting. The reload was rejected. Port 9005 in our config points at good-leafonly.crt — still the old certificate — paired with good.key, which is now the new key. That pair does not match, so nginx refused the whole configuration.
• Step 5 is the part that makes this dangerous. nginx kept running on the old configuration. That is deliberate and correct — it will not take your site down over a bad config — but it means a failed rotation and a successful rotation look identical from outside. Same fingerprint, same 200s, no user impact, no alert.
• key values mismatch is worth memorising. It means the certificate and key in one pair do not belong together. Same check as Module 02's key↔certificate match, applied by nginx at load time.

⚠️ The real-world shape of this bug: one key, referenced twice, updated once. A certificate or key used by more than one server block — a redirect vhost, a second port, a stale block someone left behind — will fail exactly this way. The rotation script reports success, the reload silently does not happen, and nobody finds out until the next restart, which is when the config is finally re-read and the server refuses to start. That is usually weeks later, in the middle of unrelated work, and it looks like the restart caused it.

🔑 Three rules, and the third is the one people miss. Always nginx -t before -s reload — it catches this while you are still watching. Always verify by connecting, never by reading the file. And grep the whole config for every reference to the file you are replacing before you replace it.

7. the second reference nobody updated:
   21-  ssl_certificate .../good-leafonly.crt; ssl_certificate_key .../good.key;
8. fixed. nginx -t   :
   nginx: configuration file .../nginx.conf test is successful
9. 9001 now serving  : D4:DF:64:05:5C:9C:A3:26:1D:5B
   9005 now serving  : D4:DF:64:05:5C:9C:A3:26:1D:5B

What to read out of this — the fingerprint on the wire now matches the fingerprint on disk. That comparison is the verification step.

• grep found the second reference in one line. Make that a step in the runbook, not something you remember. grep -rn "$(basename $KEYFILE)" /etc/nginx/ before any rotation.
• Both ports now serve the same new certificate. Comparing fingerprints across every endpoint that should share a certificate is the cheapest possible consistency check.
• nginx -t succeeded before the reload was attempted. That ordering is the entire discipline: test, then reload, then verify on the wire.

🔑 The three-line rotation runbook, worth writing on the wall:

grep -rn "mycert" /etc/nginx/           # 1. find EVERY place it is referenced
nginx -t && nginx -s reload             # 2. test, then reload, never reload blind
echo | openssl s_client -connect host:443 -servername host 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256 -enddate    # 3. verify ON THE WIRE

💡 Now imagine this at 500 hosts. Step 3 becomes "compare the fingerprint from every node against the expected one" — and it must run on every node individually, for the reason C2 gave. A rotation that lands on 499 of 500 hosts is a rotation that will page someone at an unpredictable time, and the only way to know is to have checked all 500.

The server has not re-read them. nginx, Apache and most TLS servers load the certificate and key once at start-up and keep them in memory, so replacing the files on disk changes nothing until a reload. nginx -s reload or apachectl graceful picks them up without dropping connections.

The way to confirm is always to connect, not to look at the file: compare openssl s_client ... | openssl x509 -noout -fingerprint -sha256 against the fingerprint of the file you deployed.

The detail worth adding: the failure worth describing is a reload that was rejected. If any server block in the config still pairs an old certificate with the new key, nginx fails the whole configuration with key values mismatch and keeps running on the old one. Nothing breaks, nothing alerts, and the site only fails at the next full restart — often weeks later, during unrelated work, which sends everyone looking in the wrong place. So: nginx -t before every reload, and grep the config for every reference to the file you are replacing.

Find every reference to the certificate and key across the configuration first — a grep, not a memory. Deploy the new certificate and a new key. Validate the configuration before applying it. Reload rather than restart, so connections are not dropped. Then verify by connecting to each node and comparing the certificate fingerprint and expiry against what you deployed. Keep the previous certificate until verification passes, so rollback is possible.

The order matters: new certificate in place and verified before anything is removed or revoked.

The detail worth adding: the two additions that separate a runbook that works from one that reads well. Verify on the wire, per node, by IP with explicit SNI — checking the DNS name only tests wherever you happened to land. And rotate the key, not just the certificate: reissuing on the same key means a key compromised once stays valid indefinitely, which is the same rotationPolicy: Never trap cert-manager has by default.

=== did the key actually change between the two rotations?
good-new   pubkey 620c73cb754dcc760c6093f2
good-r2    pubkey a3925d991558874f232334d5

=== now the mistake: reissue on the SAME key
same-key   pubkey a3925d991558874f232334d5
serial=1006

What to read out of this — compare the last two public key hashes. They are identical.

• good-new and good-r2 have different public keys. That is a real rotation: new key, new certificate, and the old private key is now worthless to anybody who has it.
• same-key has the same public key as good-r2. A brand new certificate, a fresh serial number 1006, a new expiry date — and the exact same private key. If that key had leaked, it is still leaked. Nothing was contained.
• Nothing about this looks wrong. Every dashboard shows a new certificate. The expiry date moved. The rotation "succeeded". This is why it happens so often.

🔑 Put this one line in your rotation verification, permanently:

openssl x509 -in new.crt -noout -pubkey | openssl sha256

Compare it with the old certificate's. If the hashes match, you renewed a certificate and did not rotate a key — which is fine for a routine renewal and completely inadequate for a compromise.

💡 The same check works against a live server, which is how you audit whether a fleet-wide emergency rotation genuinely happened:

echo | openssl s_client -connect host:443 -servername host 2>/dev/null \
  | openssl x509 -noout -pubkey | openssl sha256

⚠️ And check whether the key was used anywhere else. A key copied between services — a shared wildcard, a load balancer and an origin, a container image baked with a key in it — means one leak is several certificates. Module 11's CT search finds the public ones: search crt.sh for your domain and look for certificates you do not recognise.

Contain the leak first — whatever exposed the key, close it, or the rotation is pointless. Then issue a new certificate with a new key, deploy it, and verify on the wire node by node. Only once the replacement is confirmed live do I revoke the old certificate with reason keyCompromise, publish the CRL where one is used, and make sure every checker has re-read it. Then search Certificate Transparency for other certificates on that key, and finally investigate how it leaked.

The order matters: revoking before the replacement is live takes the service down.

The detail worth adding: two things separate a good answer here. First, a new certificate on the leaked key achieves nothing — and most tooling does exactly that by default, since certbot renew reuses the key and cert-manager's rotationPolicy defaults to Never. Verify the public key hash actually changed. Second, be honest about revocation's limits: on the public web, soft-fail means revocation may reach almost nobody (Module 09), so replacement is the control and revocation is the paperwork. If the certificate is short-lived, it may genuinely be right to skip the revocation dance entirely.

Renewal produces a new certificate — new serial, new dates, new signature — but it can be issued over the same public key, in which case the private key on the server is unchanged. Rotation means generating a new key pair, so the old private key stops being usable for anything.

For a routine expiry renewal, reusing the key is acceptable. For a compromise it is useless, because the thing that leaked is still valid.

The detail worth adding: name the defaults, because they are the trap. certbot renew reuses the key unless you pass --reuse-key=false or force a new one; cert-manager reuses it unless privateKey.rotationPolicy: Always; and any CSR sitting on disk already contains the old public key. The verification is one command — hash the certificate's public key and compare — and it is worth having in the runbook precisely because everything else about a same-key reissue looks like a success.

#!/usr/bin/env bash
# tlsaudit — one pass over one endpoint, asking every question Modules 03-12 taught us to ask.
# usage: tlsaudit.sh host:port [ca-bundle]      IP=127.0.0.1 overrides where we connect
host_port="$1"; ca="${2:-}"; host=${host_port%%:*}; port=${host_port##*:}
[ -z "$host_port" ] && { echo "usage: $0 <host:port> [ca-bundle]"; exit 2; }
epoch() { date -d "$1" +%s 2>/dev/null || date -j -f "%b %e %T %Y %Z" "$1" +%s; }
F=0
note() { printf "  %-6s %s\n" "$1" "$2"; case "$1" in FAIL|WARN) F=$((F+1));; esac; }

# ---------- one connection, reused for everything ----------
vopt=""; [ -n "$ca" ] && vopt="-CAfile $ca"
raw=$(echo | openssl s_client -connect "${IP:-$host}:$port" -servername "$host" \
      -verify_hostname "$host" $vopt 2>/dev/null)
[ -z "$raw" ] && { echo "== $host_port"; note FAIL "no TLS response at all - wrong port, or not TLS"; exit 1; }
crt=$(mktemp); echo "$raw" | openssl x509 -out "$crt" 2>/dev/null

echo "== $host_port"
echo "  subject  $(openssl x509 -in "$crt" -noout -subject | cut -d= -f2-)"
echo "  issuer   $(openssl x509 -in "$crt" -noout -issuer  | cut -d= -f2-)"

# ---------- 1. did the chain verify?                          (Module 07) ----------
vr=$(echo "$raw" | grep -m1 "Verify return code" | cut -d: -f2- | sed 's/^ *//')
[ "$vr" = "0 (ok)" ] && note ok "chain verified" || note FAIL "verify: $vr"

# ---------- 2. how many certificates did the server send?     (Module 08) ----------
n=$(echo "$raw" | grep -c "^ *[0-9] s:")
[ "$n" -le 1 ] \
  && note WARN "server sent $n certificate - the intermediate is probably missing" \
  || note ok "server sent $n certificates (leaf + intermediates, root correctly omitted)"

# ---------- 3. expiry ----------
end=$(openssl x509 -in "$crt" -noout -enddate | cut -d= -f2)
days=$(( ( $(epoch "$end") - $(date +%s) ) / 86400 ))
if   [ "$days" -lt 0 ];  then note FAIL "EXPIRED $((0-days)) days ago ($end)"
elif [ "$days" -lt 14 ]; then note WARN "expires in $days days ($end)"
else                          note ok "expires in $days days"; fi

# ---------- 4. does the name match?                     (Modules 03, 07) ----------
echo "$raw" | grep -q "Verify return code: 62" \
  && note FAIL "hostname mismatch - no SAN covers $host" \
  || note ok "name ok: $(openssl x509 -in "$crt" -noout -ext subjectAltName 2>/dev/null | tail -1 | xargs)"

# ---------- 5. protocol and forward secrecy                   (Module 06) ----------
line=$(echo "$raw" | grep -m1 "^New, TLS")     # New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
proto=$(echo "$line" | awk -F', ' '{print $2}'); ciph=$(echo "$line" | awk '{print $NF}')
case "$proto" in
  TLSv1.3|TLSv1.2) note ok "protocol $proto, cipher $ciph" ;;
  *)               note FAIL "protocol ${proto:-unknown} is obsolete" ;;
esac
case "$ciph" in TLS_*|ECDHE*|DHE*) : ;; *) note WARN "cipher $ciph has no forward secrecy" ;; esac

# ---------- 6. key size and signature algorithm         (Modules 02, 03) ----------
bits=$(openssl x509 -in "$crt" -noout -text | grep -m1 -oE "Public-Key: \([0-9]+ bit\)" | grep -oE "[0-9]+")
alg=$(openssl x509 -in "$crt" -noout -text | grep -m1 "Signature Algorithm" | awk '{print $NF}')
case "$alg" in *sha1*|*md5*) note FAIL "signed with $alg" ;;
               *)            note ok "signed with $alg, key $bits bits" ;; esac

# ---------- 7. Certificate Transparency                       (Module 11) ----------
scts=$(openssl x509 -in "$crt" -noout -text | grep -c "Signed Certificate Timestamp")
[ "$scts" -eq 0 ] && note info "no SCTs - private CA, or TLS interception" \
                  || note ok "$scts SCTs embedded"

# ---------- 8. CAA, climbing the tree                         (Module 11) ----------
if command -v dig >/dev/null 2>&1; then
  n2="$host"; rec=""
  while [ -n "$n2" ]; do
    rec=$(dig +short CAA "$n2" 2>/dev/null); [ -n "$rec" ] && break
    n2="${n2#*.}"; [ "$n2" = "${n2#*.}" ] && break
  done
  [ -n "$rec" ] && note ok "CAA set at $n2" \
                || note info "no CAA - any public CA may issue for $host"
fi

# ---------- 9. is mTLS being requested?                       (Module 12) ----------
echo "$raw" | grep -q "Acceptable client certificate CA names" \
  && note ok "server requests a client certificate (mTLS)" \
  || note info "server does not request a client certificate"

rm -f "$crt"
echo "  ---> $F finding(s)"

Run it against the healthy endpoint and two broken ones:

chmod +x ~/tls-lab/m13/tlsaudit.sh
cd ~/tls-lab/m13
for t in good.lab.test:9001 dead.lab.test:9003 good.lab.test:9005; do
  IP=127.0.0.1 ./tlsaudit.sh $t root.crt; echo
done

== good.lab.test:9001
  subject  CN = good.lab.test
  issuer   CN = Estate Issuing CA
  ok     chain verified
  ok     server sent 2 certificates (leaf + intermediates, root correctly omitted)
  ok     expires in 89 days
  ok     name ok: DNS:good.lab.test, DNS:localhost, IP Address:127.0.0.1
  ok     protocol TLSv1.3, cipher TLS_AES_256_GCM_SHA384
  ok     signed with sha256WithRSAEncryption, key 2048 bits
  info   no SCTs - private CA, or TLS interception
  info   no CAA - any public CA may issue for good.lab.test
  info   server does not request a client certificate
  ---> 0 finding(s)

== dead.lab.test:9003
  subject  CN = dead.lab.test
  issuer   CN = Estate Issuing CA
  FAIL   verify: 10 (certificate has expired)
  ok     server sent 2 certificates (leaf + intermediates, root correctly omitted)
  FAIL   EXPIRED 3 days ago (Aug 24 07:35:58 2026 GMT)
  ok     name ok: DNS:dead.lab.test, DNS:localhost, IP Address:127.0.0.1
  ok     protocol TLSv1.3, cipher TLS_AES_256_GCM_SHA384
  ok     signed with sha256WithRSAEncryption, key 2048 bits
  info   no SCTs - private CA, or TLS interception
  info   no CAA - any public CA may issue for dead.lab.test
  info   server does not request a client certificate
  ---> 2 finding(s)

== good.lab.test:9005
  subject  CN = good.lab.test
  issuer   CN = Estate Issuing CA
  FAIL   verify: 21 (unable to verify the first certificate)
  WARN   server sent 1 certificate - the intermediate is probably missing
  ok     expires in 89 days
  ok     name ok: DNS:good.lab.test, DNS:localhost, IP Address:127.0.0.1
  ok     protocol TLSv1.3, cipher TLS_AES_256_GCM_SHA384
  ok     signed with sha256WithRSAEncryption, key 2048 bits
  info   no SCTs - private CA, or TLS interception
  info   no CAA - any public CA may issue for good.lab.test
  info   server does not request a client certificate
  ---> 2 finding(s)

1. One connection, nine checks. The script connects once and reuses $raw for everything. That is not tidiness — at five hundred endpoints, nine connections each is four and a half thousand handshakes, and some of them will be against things that rate-limit you.

2. The finding count is what makes it usable at scale. ---> 0 finding(s) versus ---> 2 is sortable, greppable and alertable. Run it across an estate, sort descending, and read from the top. Human-readable output that a machine can also rank is the whole design goal.

3. info deliberately does not count as a finding. No SCTs and no CAA are true of every internal service; if they counted, every line would be noisy and the count would be meaningless. Getting the severity levels right is what decides whether a tool gets used or ignored, and it is a judgement call, not a technical one.

4. Check 7 says "private CA, or TLS interception" rather than reporting a fault. Module 11 established that zero SCTs on a public host is a strong interception signal, while on an internal host it is completely normal. A tool that cannot tell the difference should say so rather than guess — a false alarm that fires on every internal host trains people to ignore it.

5. Compare 9005's two lines with the whole of Part A. FAIL verify: 21 and WARN server sent 1 certificate are the error code and the count, side by side, in the same output. That is the A3 diagnosis, automated.

What is still missing, honestly: it checks one endpoint, not every node behind a name — the C2 lesson is not built in, and it should be, by resolving the name and looping over the addresses. It does not check the CRL, does not verify SCT signatures, does not compare fingerprints against an expected value, and reports only the leaf's expiry when an intermediate expiring sooner would break things just as thoroughly. It replaces eight tools with one; it does not replace testssl.sh.

🔑 Wrap it in the sweep from C1 and you have an estate audit: read host:port from a file, run this for each, sort by finding count, and alert on anything above zero. That is roughly what commercial certificate-management tools do, and now you know exactly what they are doing.

error:05800074:x509 certificate routines::key values mismatch
error:0A000086:SSL routines::certificate verify failed

What to read out of this — you have seen both of these already in this module.

• 05800074 is the D1 failure. nginx printed exactly this hex when the reload was rejected. The certificate and key in one server block did not belong together.
• 0A000086 is the generic wrapper around every chain failure — expired, wrong name, broken chain, all of them. It tells you that verification failed and never why, which is precisely why you need the verify code, not the library error.
• The two numbering systems are unrelated. 21 is a verify code. 0A000086 is a library error. Mixing them up sends people searching for the wrong thing.

🔑 This is one of the least-known useful OpenSSL subcommands. Logs from nginx, Apache, HAProxy, Python and Node frequently contain a bare hex code and nothing else. openssl errstr translates it instantly, offline, with no search engine and no guessing — and it works for every OpenSSL error, not just TLS ones.

A single curl against the endpoint, reading the error sentence. Almost every TLS failure is one of five — expired, wrong name, broken chain, wrong certificate served, or not TLS at all — and curl names which one in plain English in about ten seconds.

Not -k, which disables the failing check and always succeeds. And record the error text before changing anything, because the fastest way to lose a diagnosis is to start fixing before anyone wrote down the symptom.

20, unable to get local issuer certificate, is yours: your trust store lacks the issuer, typically a missing internal root or a container without ca-certificates. 21, unable to verify the first certificate, is theirs: the server sent a leaf without its intermediate.

Tell them apart by counting certificates in the s_client chain listing. Note that curl reports both with the same message, which is exactly why you escalate to s_client.

Most likely you did not pass -verify_hostname, which is not a default — without it a certificate for an entirely different name validates as ok while every browser rejects it.

Otherwise: something the browser enforces and OpenSSL does not. Certificate Transparency policy, a CA in the OS store that the browser's own root store distrusts, or an HSTS policy removing the click-through. s_client validates a chain; browsers enforce a policy.

They answer different questions. s_client reports what the two ends negotiated — normally the strongest mutually supported option, so it looks good almost regardless of configuration. sslscan enumerates everything the server is willing to accept.

Default nginx, for instance, accepts static-RSA suites like AES256-GCM-SHA384 with no forward secrecy alongside the ECDHE ones. An attacker never picks your preferred option; they pick your weakest permitted one.

Run testssl.sh locally — it implements the published SSL Labs rating guide and prints the same weighted breakdown and letter grade, labelled experimental. Pass --add-ca with your internal root or nothing validates.

It also avoids the disclosure problem: submitting a hostname to the public grader publishes it on an indexed results board by default, which leaks internal service names exactly as CT does.

TLS 1.0 or 1.1 support, no forward secrecy, no AEAD ciphers, or DH parameters under 2048 bits. Certificate problems are an outright F; a name mismatch or untrusted certificate gives M or T instead of a grade.

The caps matter more because one legacy protocol or a single non-forward-secret suite holds the grade down no matter how the score is tuned. And the grade measures configuration, not security — an A+ with an unmonitored certificate is worse off than a B that renews automatically.

Everything that is not a date. A broken chain, a certificate for the wrong name, an obsolete protocol, a weak cipher, and a certificate that was deployed but never reloaded all report perfectly healthy to an expiry check — forever.

In the lab sweep, the endpoint with the missing intermediate showed ok, 89 days while being completely unusable. Expiry is one of five failure modes and it is the only one most estates watch.

Because one IP and port can serve any number of certificates, chosen by the name the client asks for. A check that sends no SNI silently receives the default_server certificate, which may belong to a completely different service and be perfectly healthy while the real one has expired.

The related rule is to check each node by IP with explicit SNI, not just the DNS name — otherwise you only ever test wherever DNS happened to send you.

Either the server has not been reloaded — nginx and Apache read certificates once at start-up and hold them in memory — or the reload was attempted and rejected, in which case nginx keeps running the old configuration and nothing looks wrong.

The second is the dangerous one: a certificate or key referenced by more than one server block and updated in only one produces key values mismatch, the reload fails silently, and the site only breaks at the next full restart, often weeks later.

Connect and compare fingerprints. openssl s_client ... | openssl x509 -noout -fingerprint -sha256 against the fingerprint of the file you deployed, on every node, by IP, with SNI. A file on disk proves what was deployed; only the wire proves what is served.

And run nginx -t before every reload, plus grep the whole configuration for every reference to the file you are replacing.

Contain the leak, issue a new certificate with a new key, deploy and verify on the wire, and only then revoke the old one with reason keyCompromise, publish the CRL and reload the checkers. Then search CT for other certificates on that key, then investigate the leak itself.

The most common mistake is reissuing on the same key — certbot renew and cert-manager both reuse it by default — which changes the serial and contains nothing. Verify by hashing the public key.

On a schedule, not during an incident — scanners answer "what is wrong but not yet broken", which is not the question when the site is down. And only against infrastructure you are responsible for: a scan is thousands of probing connections, indistinguishable from an attack in someone else's logs and unlawful in many jurisdictions.

Tell whoever runs the monitoring before a full run, since scans trip IDS rules and fill logs. --sneaky reduces traces; it is not permission.

Question selection was cross-referenced against publicly published 2026 SSL/TLS, SRE and PKI interview question sets, then rewritten and deepened:

• NovelVista — Top SRE interview questions and answers 2026
• JavaInUse — Top OpenSSL interview questions 2026
• ClimbTheLadder — PKI interview questions

Every command and expected output in this module was executed on OpenSSL 3.0.13, nginx 1.24.0, curl 8.5.0, sslscan 2.1.2 and testssl.sh 3.3dev — including all five broken endpoints and their verify codes (0, 10, 21, 62), the three distinct curl error sentences, the sslscan cipher enumeration showing the non-forward-secret suites a default nginx accepts, the testssl.sh chain-incomplete and serial-entropy findings, the SNI experiment, and the full rotation sequence.

Two things in this module were discovered by running it rather than planned, and both were kept because they are more instructive than what was intended. The D1 rotation was supposed to demonstrate only that a replaced file is not served until reload; in the actual run the reload was additionally rejected with key values mismatch, because the new key was referenced by a second server block still pointing at the old certificate — and nginx carried on serving the old configuration with no outward sign of failure. That is a better lesson than the one planned, so the exercise was rebuilt around it. Similarly, the sslscan output revealed that a default nginx accepts static-RSA cipher suites with no forward secrecy, which was not the point of the exercise and is the most useful thing in it.

Current-state claims were verified against primary sources: the SSL Labs Server Rating Guide for the 30/30/40 weights, the score thresholds and every capping rule; the testssl.sh command-line help, read from the installed binary, for the option list including --severity, --add-ca, --ip, --jsonfile, --sneaky and the beta --mtls; the sslscan repository and its local --help for its flags; and PromLabs for the probe_ssl_earliest_cert_expiry - time() < 7*24*3600 alert expression. The SSL Labs public grader itself was not used, in keeping with the module's own advice about scanning only what you are responsible for.

Answers were rewritten and deepened rather than reproduced — published versions are usually correct but shallow, and the added operational detail is what actually differentiates a candidate in the room.