⛓️Module 05 — Chain of Trust & Running Your Own CA

=== how many trust anchors does this machine have? ===
152
305

=== why are there more entries than files? ===
lrwxrwxrwx 1 root root   21 Aug 20 11:30 002c0b4f.0 -> GlobalSign_Root_CA.pem
lrwxrwxrwx 1 root root   25 Aug 20 11:30 02265526.0 -> Entrust_Root_Certification_Authority_-_G2.pem
lrwxrwxrwx 1 root root   20 Aug 20 11:30 0b1b94ef.0 -> Certigna.pem

=== the hash a certificate gets filed under ===
subject_hash=8d33f237
subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1

What to read out of this.

• 152 files but 305 entries. The extra ones are symlinks named after a hash of the subject, with a .0 suffix. That is the indexing mechanism, and it is the answer to "how does OpenSSL find a root among 152 without opening all of them?"
• The lookup is: hash the Issuer name you are looking for, open <hash>.0. One filesystem operation instead of 152 file reads. The .0 is a collision counter — if two different certificates hash the same, the second becomes .1.
• This is why openssl rehash exists. If you drop a certificate into a -CApath directory by hand and do not rehash it, OpenSSL will never find it, and will report unable to get local issuer certificate while the file sits right there. update-ca-certificates does the rehash for you, which is why you should use it rather than copying files around.
• -CAfile versus -CApath: -CAfile is one file that OpenSSL reads completely; -CApath is a hashed directory it looks into. The hash mechanism only applies to the second.

=== 1. against the SYSTEM trust store: our CA is unknown ===
error 20 at 1 depth lookup: unable to get local issuer certificate
error app.crt: verification failed

=== 2. against a trust store we supply ourselves ===
app.crt: OK

=== 3. a directory-based store, built in our own lab ===
total 8
lrwxrwxrwx 1 zaeem zaeem   8 Aug 20 15:02 f0d8335d.0 -> root.crt
-rw------- 1 zaeem zaeem 1830 Aug 20 15:02 root.crt
app.crt: OK

What to read out of this — you have learned the whole lesson without changing your machine.

• error 20 at 1 depth, not at 0. Depth 1 is the intermediate. OpenSSL got past the leaf, then could not find the intermediate's issuer. The depth number tells you which link is broken, which is far more useful than the error text alone.
• The certificate did not change between 1 and 2. Only the store did. That is Module 03's D3 lesson made concrete: trust is a property of the store, not of the file.
• Step 3 builds a real hashed trust store in your lab directory. openssl rehash created the symlink f0d8335d.0 — the subject hash from Exercise A3.1. This is exactly what /etc/ssl/certs is, just in a folder you own. -CApath then uses it.
• That is the whole mechanism. The system store is not magic; it is a directory of certificates plus hash symlinks, and update-ca-certificates is a script that copies files and runs rehash.

🔑 And this is how you should do it in production code anyway. Applications that need to trust an internal CA should be pointed at a CA bundle — --cacert, RootCAs, NODE_EXTRA_CA_CERTS, REQUESTS_CA_BUNDLE, SSL_CERT_FILE — rather than relying on the machine's global store. It is explicit, it is visible in configuration, it survives OS upgrades, and it does not grant that CA authority over everything else on the host.

=== before: our own CA is not trusted ===
error 20 at 1 depth lookup: unable to get local issuer certificate
error app.crt: verification failed

Adding debian:zaeem-labs-root.pem
done.
done.

=== after ===
app.crt: OK

=== what actually happened on disk ===
lrwxrwxrwx 1 root root 19 Aug 20 11:30 f0d8335d.0 -> zaeem-labs-root.pem
lrwxrwxrwx 1 root root 52 Aug 20 11:30 zaeem-labs-root.pem -> /usr/local/share/ca-certificates/zaeem-labs-root.crt
f0d8335d

What to read out of this.

• error 20 at 1 depth, not at 0. Depth 1 is the intermediate. OpenSSL got past the leaf fine, then could not find the intermediate's issuer. The depth number tells you which link is broken, which is far more useful than the error text alone.
• The certificate did not change. Only the list did. This is Module 03's D3 lesson, made concrete: trust is a property of the store.
• Two symlinks were created. One with a friendly name pointing at your file, and one named f0d8335d.0 — the subject hash from A3.1. Both are needed: the hash link is how OpenSSL finds it, the named link is how humans find it.
• Adding debian:zaeem-labs-root.pem — note the debian: prefix and the .pem extension in the output. The source file must be .crt, but the installed copy is renamed. That asymmetry is exactly why the .crt requirement catches people.

⚠️ To undo this, delete the source file and run the update again — do not delete the symlinks in /etc/ssl/certs/, which will be recreated:

sudo rm /usr/local/share/ca-certificates/zaeem-labs-root.crt
sudo update-ca-certificates --fresh

Now imagine this at 500 hosts. Doing this by hand does not scale and does not survive. The root must be delivered by configuration management or baked into your base image — and separately into every runtime trust store that matters (Java's cacerts, Node's NODE_EXTRA_CA_CERTS, Python's REQUESTS_CA_BUNDLE). Teams that install roots by hand rediscover this at every OS upgrade, when the package manager replaces the bundle.

Distribute the root certificate only — never the intermediate, and never a private key — through configuration management or the base image, so it survives rebuilds and OS upgrades.

On Debian that is /usr/local/share/ca-certificates/name.crt plus update-ca-certificates; on RHEL, /etc/pki/ca-trust/source/anchors/ plus update-ca-trust extract. The file must be PEM and, on Debian, must end in .crt or it is silently skipped.

The part people forget: the OS store is only one of several. Java reads cacerts, Node reads NODE_EXTRA_CA_CERTS, Python's requests reads certifi unless told otherwise, and Firefox has its own store entirely. A root installed for the OS does nothing for a Java service on the same host — which is the classic "curl works, the app does not" symptom from Module 02.

And the design point: distribute the root, not the intermediate. Trusting the root means you can rotate intermediates without touching a single client, which is the whole reason for a two-tier hierarchy.

So the root key can stay offline. A key used for every issuance cannot live in a safe; a key used four times a decade can.

It also bounds damage. A compromised intermediate is revoked and replaced, affecting only the certificates beneath it. A compromised root has no recovery path — it cannot revoke itself, and removing it requires every browser and OS vendor to ship an update, which takes years to reach the long tail of devices.

And it enables segmentation: separate intermediates per product, region or validation level, each constrained by EKU, pathlen and name constraints.

The design detail worth volunteering: issuing intermediates carry pathlen:0, so a compromised intermediate can mint leaf certificates but cannot build a hierarchy beneath itself. The root delegates once and no further.

subject=C = MY, O = Zaeem Labs, CN = Zaeem Labs Root CA
issuer=C = MY, O = Zaeem Labs, CN = Zaeem Labs Root CA
X509v3 Basic Constraints: critical
    CA:TRUE
X509v3 Key Usage: critical
    Certificate Sign, CRL Sign

What to read out of this, and the config choices behind it.

• Subject equals Issuer — self-signed, as every root is (Module 03, D3).
• CA:TRUE with no pathlen. Unlimited depth, which is correct for a root: its job is to delegate. Its protection is that the key is offline, not a path limit.
• rand_serial = yes makes OpenSSL generate random serials instead of counting upward from the serial file — the CA/B Forum entropy requirement from Module 03 (A2), and good practice even privately.
• copy_extensions = none is the setting that matters most for security. It is Module 04's -copy_extensions trap in configuration form: the CA decides the extensions, never the requester.
• policy_loose with commonName = supplied means a CSR must carry a CN, and country and organisation are optional. policy_match (the other common choice) would additionally force the CSR's O to equal the CA's — useful inside one organisation, annoying otherwise.
• 7300 days is 20 years. Long, deliberately. A root that expires is an estate-wide outage, and replacing one is a project measured in months.

⚠️ In production this key would not be here. It would be generated and used on an air-gapped machine or in an HSM, and this directory would only ever hold the certificate. Everything else in this module is realistic; that part is a lab shortcut.

commonName            :ASN.1 12:'Zaeem Labs Issuing CA 1'
Certificate is to be certified until Aug 19 11:28:58 2031 GMT (1825 days)

Write out database with 1 new entries
Database updated

=== the certificate ===
subject=C = MY, O = Zaeem Labs, CN = Zaeem Labs Issuing CA 1
issuer=C = MY, O = Zaeem Labs, CN = Zaeem Labs Root CA
serial=15ACDDBF1E08B89A8CAE1A6539694E22165F757F
X509v3 Basic Constraints: critical
    CA:TRUE, pathlen:0
X509v3 Key Usage: critical
    Certificate Sign, CRL Sign

=== the ledger ===
V	310819112858Z		15ACDDBF1E08B89A8CAE1A6539694E22165F757F	unknown	/C=MY/O=Zaeem Labs/CN=Zaeem Labs Issuing CA 1

=== new_certs_dir ===
15ACDDBF1E08B89A8CAE1A6539694E22165F757F.pem

What to read out of this — the ledger line is the important part.

index.txt is tab-separated with six fields:

• Field 1 and field 3 are what revocation is. Revoking a certificate changes V to R and writes a timestamp into field 3. That is the entire mechanism, and the CRL you will generate in Module 09 is just a signed rendering of this file.
• Write out database with 1 new entries / Database updated — openssl ca refuses to lose track of anything it signs. This is exactly what openssl x509 -req does not do, and why Module 04's approach cannot support revocation.
• new_certs_dir now holds a copy filed by serial. If you ever need to know exactly what you issued three years ago, it is there.
• -batch skipped the confirmation prompts and -notext kept the human-readable dump out of the output file. Without -notext your .crt contains the full text dump above the PEM block — valid, but ugly and larger.
• pathlen:0 came from the v3_intermediate section, chosen with -extensions. The CSR asked for nothing; the issuer decided everything.

openssl x509 -req signs a CSR and forgets it. openssl ca maintains state: a database (index.txt) of everything issued, a serial counter, and a copy of each certificate filed by serial.

That state is not bookkeeping for its own sake — it is what makes revocation possible. A CRL is generated from index.txt; a certificate the CA has no record of cannot be revoked, because there is nothing to mark as revoked.

The practical rule: x509 -req is fine for a throwaway test certificate. Anything you will have to operate — anything that might need revoking, auditing or reissuing — goes through openssl ca or a real CA product.

What you would actually use in production: not raw OpenSSL. step-ca, HashiCorp Vault's PKI engine, EJBCA, or a cloud private CA. Raw openssl ca is how you learn the moving parts and how you run a root ceremony offline; it is not how you run an issuing CA at scale.

Write out database with 1 new entries
Database updated

=== 1. leaf against the root only ===
C = MY, O = Zaeem Labs, CN = app.internal.test
error 20 at 0 depth lookup: unable to get local issuer certificate
error app.crt: verification failed

=== 2. leaf with the intermediate supplied ===
app.crt: OK

=== 3. and the hostname ===
app.crt: OK

=== 4. show the path ===
app.crt: OK
Chain:
depth=0: C = MY, O = Zaeem Labs, CN = app.internal.test (untrusted)
depth=1: C = MY, O = Zaeem Labs, CN = Zaeem Labs Issuing CA 1 (untrusted)
depth=2: C = MY, O = Zaeem Labs, CN = Zaeem Labs Root CA

What to read out of this.

• Step 1 fails even though you own the root. Trusting the root is not enough — the client still needs the intermediate to bridge the gap. This is exactly the situation your servers will be in, and exactly why fullchain.pem exists.
• error 20 at 0 depth here, versus at 1 depth in Exercise A3.2. The depth tells you where the search stopped. Depth 0 means it could not find the leaf's issuer; depth 1 means it found the intermediate but not its issuer. Two different fixes.
• -show_chain proves the three-level structure you built. depth=2 has no (untrusted) marker because it came from -CAfile — it is the anchor.
• The SAN came from v3_server in the config, not from the CSR. The CSR only carried a CN. The issuer decided the names, which is the Module 04 lesson enforced by configuration.

💡 A real CA would validate those names first. Our config hard-codes them, which is fine for one service and obviously wrong for a general-purpose CA. Real CA software takes the requested names, checks them against an authorisation policy, and then writes them. That validation step is the entire difference between a CA and a signing script.

Write out database with 1 new entries
Database updated

=== the intermediate issued it without complaint ===
subject=C = MY, O = Zaeem Labs, CN = Rogue Sub CA
X509v3 Basic Constraints: critical
    CA:TRUE

=== now ask a client to accept the 4-deep chain ===
C = MY, O = Zaeem Labs, CN = Zaeem Labs Issuing CA 1
error 25 at 2 depth lookup: path length constraint exceeded
error victim.crt: verification failed

What to read out of this — the two halves say opposite things and both are true.

The intermediate issued the sub-CA happily. openssl ca did not check its own pathlen. It signed the request, wrote a line in its ledger, and reported success. An issuer does not police itself.

The verifier refused the chain. error 25 ... path length constraint exceeded, reported at depth 2 — which is the intermediate, the certificate that carried the constraint. The client walked up the chain, reached the intermediate, read pathlen:0, counted the CAs below it, and found one too many.

• www.google.com was never at risk, because nothing outside your lab trusts your root. But inside an organisation whose machines do trust your root, that certificate would be accepted by everything — which is precisely the scenario pathlen bounds.
• This is the same duality as Module 04 (D1.2), where a CSR could request CA:TRUE and a careless issuer granted it. Issuance and validation are separate systems, and security requires both to be right.

🔑 Add error 25 to your table. Alongside 20 (missing issuer), 21 (cannot verify first certificate), 18/19 (self-signed), 62 (hostname mismatch) and 47 (name constraint), this is the set that covers nearly every real chain failure you will meet.

X509v3 Name Constraints: critical
    Permitted:
      DNS:.internal.test
      DNS:internal.test

=== issue a name INSIDE the permitted space ===
ok.crt: OK

=== now issue a name OUTSIDE it ===
CN = www.google.com
error 47 at 0 depth lookup: permitted subtree violation
error badnc.crt: verification failed

What to read out of this.

• error 47 — permitted subtree violation. The CA signed the certificate perfectly happily; the verifier rejected it because the name falls outside what the CA above was permitted to issue for. Same issuer/verifier split as pathlen.
• The error is reported at depth 0 — the leaf — because that is where the offending name lives, even though the constraint lives at depth 1. Compare with error 25 in C1.1, reported at depth 2 where the constraint was. The depth points at the certificate that made the check fail, which is not always the one carrying the rule.
• Both permitted;DNS: lines are needed. .internal.test with the leading dot covers subdomains; internal.test without it covers the bare domain. It is the same trap as the wildcard rule from Module 03 (B3) — the shorthand does not include the apex.
• Marked critical, as RFC 5280 requires. That means a client that does not understand name constraints must reject the certificate rather than ignore the restriction — which is exactly the behaviour you want from a control like this.

⚠️ Two real-world caveats worth knowing before you rely on this.

1. Support is good but not universal. OpenSSL, Go, modern browsers, macOS and Windows all enforce name constraints. Some older embedded stacks and a few Java versions historically did not. It is a strong control, not a guaranteed one — so treat it as defence in depth rather than the only fence.
2. Constraints apply to the whole subtree below. Adding one to an existing CA invalidates every certificate it has already issued for names outside the new limits. Introduce them with a new intermediate, not by re-issuing an existing one.

🔑 If you take one thing from Part C into an interview, make it this. "Our internal CA is name-constrained to our own domains, so even a full compromise of the issuing CA cannot be used to impersonate an external service to our fleet." That sentence demonstrates that you have thought about your own CA as an attack surface, which most people have not.

By default, nothing — and that is the point of the question. An internal root installed on every machine is, to those machines, exactly as authoritative as any public CA. Compromise the issuing CA and you can impersonate anything on the internet to your entire estate.

The control is name constraints (RFC 5280 §4.2.1.10): a critical extension on the CA certificate listing permitted and excluded namespaces, for example permitted;DNS:.internal.example.com. A leaf with a name outside that space fails validation with permitted subtree violation, no matter how correctly it was signed.

Pair it with pathlen:0 so the CA cannot create sub-CAs that might carry different constraints, and with EKU restrictions so an intermediate limited to serverAuth cannot mint code-signing certificates.

The caveats that show you have actually deployed this: enforcement is done by the client, and while OpenSSL, Go, browsers, macOS and Windows all honour it, a few older stacks do not — so it is defence in depth. And constraints apply retroactively to everything beneath the CA, so they must be introduced with a new intermediate rather than bolted onto an existing one.

The client, during path validation. A CA will happily sign something that violates its own constraints — openssl ca does not check its own pathlen before issuing.

That split is deliberate and it is the same design as basicConstraints: the certificate carries the rule, and every verifier along the way is obliged to enforce it. It means a compromised or misconfigured issuer cannot quietly grant itself more authority, because the authority was written down and signed by someone above it.

The practical consequence: you cannot test these controls by trying to issue a bad certificate — issuance will succeed. You test them by trying to verify the result, which is why every exercise in this part ends with openssl verify rather than with the signing step.

=== A. leaf only - the broken deployment ===
1
Verify return code: 21 (unable to verify the first certificate)
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

=== B. leaf + intermediate - the fix ===
2
HTTP 200

What to read out of this.

• grep -c 'BEGIN CERTIFICATE' on -showcerts is your fastest diagnostic. One certificate from a server whose leaf was issued by an intermediate means the chain is incomplete. Full stop, no further investigation needed.
• Verify code 21, not 20. Both mean "missing intermediate", and which you get depends on context: openssl verify on files gives 20 (unable to get local issuer certificate), while s_client against a live server tends to give 21 (unable to verify the first certificate). Treat 20 and 21 as the same diagnosis.
• curl reports error 60 with unable to get local issuer certificate — the same underlying condition, a third wording. Three tools, three phrasings, one cause.
• The fix added one file and nothing else changed.

⚠️ A genuine s_server trap, and it caught me while writing this module. openssl s_server -cert fullchain.pem reads only the first certificate from that file. Unlike nginx, it does not treat a bundle as a chain — you must pass the rest with -cert_chain. So testing a bundle with s_server can show a broken chain even when the same file works perfectly in nginx.

How real servers want it:

🔑 The rule to remember: cat leaf.crt intermediate.crt > fullchain.pem. Leaf first, then upward, never the root.

=== correct order: leaf first ===
subject=C = MY, O = Zaeem Labs, CN = app.internal.test
issuer=C = MY, O = Zaeem Labs, CN = Zaeem Labs Issuing CA 1

subject=C = MY, O = Zaeem Labs, CN = Zaeem Labs Issuing CA 1
issuer=C = MY, O = Zaeem Labs, CN = Zaeem Labs Root CA

app.crt: OK

=== reversed ===
subject=C = MY, O = Zaeem Labs, CN = Zaeem Labs Issuing CA 1
issuer=C = MY, O = Zaeem Labs, CN = Zaeem Labs Root CA

subject=C = MY, O = Zaeem Labs, CN = app.internal.test
issuer=C = MY, O = Zaeem Labs, CN = Zaeem Labs Issuing CA 1

app.crt: OK

What to read out of this — and the answer is more nuanced than most people say.

• openssl verify accepted both orders. As a bag of candidate certificates, order is irrelevant — the verifier searches it either way.
• But the file's order still matters, for two reasons:

1. The web server reads it positionally. nginx takes the first certificate in ssl_certificate as the leaf and everything after as the chain. Put the intermediate first and nginx serves the intermediate as your server certificate — the handshake then presents a certificate whose SAN does not include your hostname, and every client fails with a name mismatch. The file is valid; the deployment is broken.

2. TLS itself specifies the order. RFC 8446 §4.4.2 says the sender's certificate comes first, each subsequent one certifying the previous. Modern clients tolerate a mis-ordered chain on the wire, but tolerating something is not the same as it being correct, and older and embedded stacks are less forgiving.

🔑 So the honest answer in an interview is: "Order does not matter to path building, because the client searches. It matters to the server, which reads the file positionally, and it is what the TLS spec requires on the wire. Leaf first, then up, never the root." That is more accurate than either "order doesn't matter" or "order is critical", and the nuance is the part worth having.

First establish whether the server is sending its chain at all:

openssl s_client -connect host:443 -servername host -showcerts </dev/null 2>/dev/null | grep -c 'BEGIN CERTIFICATE'

One certificate from a publicly issued site means the chain is incomplete — that is the answer, immediately. Confirm with the verify code: 20 or 21 both mean the same thing.

Then get the missing piece from the leaf's AIA CA Issuers URL (Module 03, C5), convert it from DER, and rebuild fullchain.pem with the leaf first.

The two mistakes to name:

1. Fixing it on the client. Adding the intermediate to a client's trust store makes that one client work and leaves every other one broken, forever. The fault is on the server.
2. Trusting the browser. Chrome on Windows and macOS fetches missing intermediates via AIA and caches ones it has seen, so it hides the fault entirely. Test with openssl s_client or curl.

The other possibility worth checking: the server might be sending the wrong intermediate — for instance after a CA key rotation. Compare the leaf's AKI with the intermediate's SKI to confirm they belong together.

=== same subject, different issuer ===
int/int.crt      subject=... CN = Zaeem Labs Issuing CA 1  issuer=... CN = Zaeem Labs Root CA
int-cross.crt    subject=... CN = Zaeem Labs Issuing CA 1  issuer=... CN = Zaeem Labs Legacy Root

=== same public key? ===
IDENTICAL KEY

=== the SAME leaf now verifies under EITHER root ===
app.crt: OK
app.crt: OK

What to read out of this.

• app.crt was signed exactly once, and it validates under two completely unrelated roots. That is the whole trick: the leaf's issuer is identified by the intermediate's key, and both intermediate certificates carry that same key.
• Nothing about the leaf changed. No reissue, no redeploy. You extended trust to a new population of clients purely by handing out a different intermediate.
• This is why AKI/SKI matter (Module 03, C4). With two certificates sharing a Subject DN, the DN alone is ambiguous. The Authority Key Identifier points at the key, which is what makes the search unambiguous.

The real-world story worth being able to tell. Let's Encrypt launched in 2015 with a root — ISRG Root X1 — that no trust store contained. To be usable immediately, its intermediates were cross-signed by IdenTrust's DST Root CA X3, which had been trusted for years.

For six years, Let's Encrypt certificates chained to DST Root CA X3 on older clients and to ISRG Root X1 on newer ones. Then:

• 30 September 2021 — DST Root CA X3 expired, and a long tail of old Android devices, OpenSSL 1.0.x systems and appliances broke. Sites that had done nothing wrong went down.
• February 2024 — Let's Encrypt stopped serving the cross-signed chain by default.
• June 2024 — it became unavailable entirely.
• September 2024 — the cross-sign expired for good.

Shortening the chain cut certificate bytes per handshake by over 40%. And the reason it became safe to do was simply that ISRG Root X1's presence in Android trust stores had risen from 66% to 93.9%.

🔑 The interview-ready summary: "Cross-signing lets a new CA be trusted immediately by clients that only know an old root. Two certificates, same subject and key, different issuers — the client picks whichever path reaches a root it has. The cost is a longer chain and a hard dependency on the old root's expiry date, which is exactly what bit everyone in September 2021."

💡 The same mechanism is running right now: Let's Encrypt announced a new "Generation Y" hierarchy in November 2025 — roots ISRG Root YR (RSA 4096) and ISRG Root YE (ECDSA P-384) with intermediates YR1–YR3 and YE1–YE3 — which will be cross-signed by the existing X-generation roots while the new ones work their way into trust stores. Exactly the same play, ten years later.

Issuing a second certificate for the same CA key and subject, signed by a different root. The CA ends up with two certificates that are interchangeable from a signature standpoint, so a leaf it issued validates under either root depending on which the client trusts.

It exists because getting a new root into trust stores takes years — browser and OS update cycles, plus devices that will never be updated at all. Cross-signing lets a new CA be usable on day one by borrowing an established root's reach.

The concrete example: Let's Encrypt's ISRG Root X1 was cross-signed by IdenTrust's DST Root CA X3 from 2015. When X3 expired on 30 September 2021, older clients that had been relying on that path broke — a well-known outage that hit sites which had changed nothing. Let's Encrypt fully retired the cross-sign in 2024, which cut certificate bytes per handshake by over 40%.

The operational lesson: a cross-signed chain has two expiry dates that matter — your certificate's and the cross-signing root's — and the second one is not visible in your own monitoring. It is a good example of why chain-aware monitoring beats leaf-only monitoring.

[OK  ] depth=0  CN = letsencrypt.org                                Nov 12 08:14:59 2026 GMT
[OK  ] depth=1  C = US, O = Let's Encrypt, CN = E7                  Sep  2 00:00:00 2028 GMT
[OK  ] depth=0  CN = example.com                                    Jan 15 23:59:59 2027 GMT
[OK  ] depth=1  C = US, O = DigiCert Inc, CN = DigiCert Global G3... Sep 23 00:00:00 2030 GMT

What to read out of this.

• The script walks every certificate the server sent, not just the leaf. That single change is what makes it useful — an expired intermediate behind a healthy leaf is invisible to almost every off-the-shelf expiry check.
• It reads what the server is actually serving, over the network, rather than a file on disk. Those two things diverge the moment a renewal writes a new file that nothing reloaded — which is the single most common certificate outage there is.
• It does not check the root, because the root is not sent and is not supposed to be. To catch an expiring root you would monitor your trust store, which is a different job with a much longer time horizon.
• Exit-code semantics from Module 03 (A3.2) are reused: -checkend N returns 0 for "safe" and 1 for "expiring or expired". Getting that backwards produces a monitor that reports healthy forever.

🔑 Three things separate real certificate monitoring from the naive kind, and all three are worth naming in an interview:

1. Check the live endpoint, not the file — catches "renewed but never reloaded".
2. Check every certificate in the chain — catches expiring intermediates.
3. Alert with escalating lead time — 30 days, then 7, then daily — so a single missed email is not the last line of defence.

Now imagine this at 500 hosts. Run it from a central place against your public endpoints rather than as an agent on each host, because that also catches load balancers and CDNs terminating TLS with a certificate you did not know about. Feed the results into whatever you already alert on. Module 13 extends this into a full auditing tool.

Several possibilities, roughly in order of likelihood:

1. An expired intermediate. Every certificate in the chain must be in date, and leaf-only monitoring never sees this coming.
2. An expired root, or a cross-signing root reaching its end date — the Let's Encrypt / DST Root CA X3 event of September 2021 is the canonical example.
3. A renewal that was never reloaded. The new certificate is on disk; the running process still holds the old one in memory. Monitoring the file rather than the live endpoint misses this entirely.
4. The wrong certificate being served — an SNI or virtual-host misconfiguration where a default server block answers instead.
5. A revoked certificate, or an OCSP responder failing in a hard-fail configuration (Module 09).

The diagnostic that separates them in one command:

openssl s_client -connect host:443 -servername host -showcerts </dev/null 2>/dev/null \
  | openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -print_certs -noout

That prints every certificate the server is actually serving, so you can check each one's dates rather than assuming.

mkdir -p ~/tls-lab/m05/capstone && cd ~/tls-lab/m05/capstone
umask 077
mkdir -p root/{certs,db,private} sub/{certs,db,private}
touch root/db/index.txt sub/db/index.txt
openssl rand -hex 8 > root/db/serial; openssl rand -hex 8 > sub/db/serial

# ---------- 1. the root ----------
cat > root.cnf <<'EOF'
[ ca ]
default_ca = CA_root
[ CA_root ]
dir             = ./root
database        = $dir/db/index.txt
serial          = $dir/db/serial
new_certs_dir   = $dir/certs
certificate     = $dir/root.crt
private_key     = $dir/private/root.key
default_md      = sha256
policy          = pol
rand_serial     = yes
unique_subject  = no
copy_extensions = none
email_in_dn     = no
[ pol ]
countryName      = optional
organizationName = optional
commonName       = supplied
[ req ]
distinguished_name = dn
prompt = no
[ dn ]
C = MY
O = Corp Test
CN = Corp Test Root CA
[ v3_issuing ]
basicConstraints       = critical,CA:TRUE,pathlen:0
keyUsage               = critical,keyCertSign,cRLSign
extendedKeyUsage       = serverAuth
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always,issuer
nameConstraints        = critical,permitted;DNS:.corp.test,permitted;DNS:corp.test,excluded;IP:0.0.0.0/0.0.0.0
EOF

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:4096 -out root/private/root.key
openssl req -x509 -config root.cnf -key root/private/root.key -out root/root.crt -days 7300 \
  -addext "basicConstraints=critical,CA:TRUE" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" \
  -addext "subjectKeyIdentifier=hash"

# ---------- 2. the issuing CA ----------
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out sub/private/sub.key
openssl req -new -key sub/private/sub.key -out sub/sub.csr \
  -subj "/C=MY/O=Corp Test/CN=Corp Test Issuing CA 1"
openssl ca -config root.cnf -extensions v3_issuing -days 1825 -notext -batch \
  -in sub/sub.csr -out sub/sub.crt

# ---------- 3. the leaf ----------
sed 's|CA_root|CA_sub|; s|./root|./sub|; s|root.crt|sub.crt|; s|root.key|sub.key|' root.cnf > sub.cnf
cat >> sub.cnf <<'EOF'
[ v3_leaf ]
basicConstraints       = critical,CA:FALSE
keyUsage               = critical,digitalSignature
extendedKeyUsage       = serverAuth
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid,issuer
subjectAltName         = DNS:shop.corp.test,DNS:www.shop.corp.test
[ v3_evil ]
basicConstraints = critical,CA:FALSE
keyUsage         = critical,digitalSignature
extendedKeyUsage = serverAuth
subjectAltName   = DNS:evil.example.com
[ v3_subca ]
basicConstraints = critical,CA:TRUE
keyUsage         = critical,keyCertSign,cRLSign
EOF

openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out shop.key
openssl req -new -key shop.key -out shop.csr -subj "/C=MY/O=Corp Test/CN=shop.corp.test"
openssl ca -config sub.cnf -extensions v3_leaf -days 90 -notext -batch -in shop.csr -out shop.crt

# ---------- 4. prove the four controls ----------
echo "== 4a. valid chain + hostname =="
openssl verify -CAfile root/root.crt -untrusted sub/sub.crt -verify_hostname shop.corp.test shop.crt

echo "== 4b. a name outside the constraint =="
openssl req -new -key shop.key -out evil.csr -subj "/CN=evil.example.com"
openssl ca -config sub.cnf -extensions v3_evil -days 90 -notext -batch -in evil.csr -out evil.crt
openssl verify -CAfile root/root.crt -untrusted sub/sub.crt evil.crt

echo "== 4c. a sub-CA beneath a pathlen:0 issuer =="
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out rogue.key
openssl req -new -key rogue.key -out rogue.csr -subj "/C=MY/O=Corp Test/CN=Rogue CA"
openssl ca -config sub.cnf -extensions v3_subca -days 365 -notext -batch -in rogue.csr -out rogue.crt
openssl req -new -key shop.key -out r2.csr -subj "/CN=deep.corp.test"
printf 'basicConstraints=critical,CA:FALSE\nsubjectAltName=DNS:deep.corp.test\n' > r2.cnf
openssl x509 -req -in r2.csr -CA rogue.crt -CAkey rogue.key -out deep.crt -days 90 -extfile r2.cnf 2>/dev/null
cat sub/sub.crt rogue.crt > deepchain.pem
openssl verify -CAfile root/root.crt -untrusted deepchain.pem deep.crt

echo "== 4d. no intermediate supplied =="
openssl verify -CAfile root/root.crt shop.crt

# ---------- 5. the bundle ----------
cat shop.crt sub/sub.crt > fullchain.pem
echo "== 5. bundle order (leaf must be first) =="
openssl crl2pkcs7 -nocrl -certfile fullchain.pem | openssl pkcs7 -print_certs -noout | grep subject
head -1 <(openssl x509 -in fullchain.pem -noout -subject)   # what a server reads as the leaf

# ---------- 6. the ledger ----------
echo "== 6. the issuing CA ledger =="
cat sub/db/index.txt

Expected output at the key points:

== 4a. valid chain + hostname ==
shop.crt: OK

== 4b. a name outside the constraint ==
CN = evil.example.com
error 47 at 0 depth lookup: permitted subtree violation
error evil.crt: verification failed

== 4c. a sub-CA beneath a pathlen:0 issuer ==
C = MY, O = Corp Test, CN = Corp Test Issuing CA 1
error 25 at 2 depth lookup: path length constraint exceeded
error deep.crt: verification failed

== 4d. no intermediate supplied ==
C = MY, O = Corp Test, CN = shop.corp.test
error 20 at 0 depth lookup: unable to get local issuer certificate
error shop.crt: verification failed

== 5. bundle order (leaf must be first) ==
subject=C = MY, O = Corp Test, CN = shop.corp.test
subject=C = MY, O = Corp Test, CN = Corp Test Issuing CA 1
subject=C = MY, O = Corp Test, CN = shop.corp.test

== 6. the issuing CA ledger ==
V	261118...Z		4A1F...	unknown	/C=MY/O=Corp Test/CN=shop.corp.test
V	261118...Z		7B2C...	unknown	/CN=evil.example.com
V	270820...Z		9E44...	unknown	/C=MY/O=Corp Test/CN=Rogue CA

The six things this capstone is really testing.

1. The issuing CA signed everything it was asked to. Look at the ledger: evil.example.com and Rogue CA are both recorded as V — valid, issued, no complaint. Every one of your controls was enforced by the verifier, not the issuer. If you only tested by trying to issue, you would have concluded the controls did not work.

2. Three different error codes, three different controls. 47 for the name constraint, 25 for pathlen, 20 for the missing intermediate. Being able to name which control fired from the code alone is the practical skill.

3. The depths differ, and they point at different things. Error 47 at depth 0 (the leaf carries the bad name); error 25 at depth 2 (the CA carries the constraint). The depth identifies the certificate that made the check fail, which is not always the one holding the rule.

4. excluded;IP:0.0.0.0/0.0.0.0 forbids all IP SANs. Worth including on an internal CA: it means the CA cannot issue certificates for bare IP addresses, which closes off a class of internal impersonation that DNS-based constraints miss entirely.

5. The leaf is first in fullchain.pem — confirmed by openssl x509 -in fullchain.pem -noout -subject, which reads only the first certificate and is therefore exactly what nginx sees.

6. What revocation would look like. In index.txt, revoking evil.example.com changes field 1 from V to R and writes a timestamp into field 3. Nothing else in the file changes. That two-field edit is revocation — everything in Module 09 is about publishing it and getting clients to read it.

What is still missing, honestly: the root key is on the same disk (it should be air-gapped or in an HSM), there is no CRL, no OCSP responder, no automated issuance, and no separation of duties. That is the gap between this and step-ca or Vault PKI — and it is a fair thing to say out loud in an interview.

Usage: verify [options] cert.pem...
General options:
 -help                       Display this summary
 -engine val                 Use engine, possibly a hardware device
Certificate chain options:
 -CAfile infile              A file of trusted certificates
 -CApath dir                 A directory of files with trusted certificates
 -no-CAfile                  Do not load the default certificates file
 -no-CApath                  Do not load certificates from the default certificates directory
 -untrusted infile           A file of untrusted certificates
 -trusted infile             A file of trusted certificates
 -show_chain                 Display information about the certificate chain
...
Verification options:
 -verify_hostname val        Expected hostname
 -verify_ip val              Expected IP address
 -verify_email val           Expected email address
 -x509_strict                Disable certificate compatibility work-arounds
 -partial_chain              Accept partial certificate chain if at least one is a trusted certificate

What to read out of this — three flags here are worth knowing about.

• -untrusted versus -trusted. -untrusted supplies intermediates that still have to chain to an anchor. -trusted says "treat these as anchors directly". Using -trusted where you meant -untrusted makes a broken chain appear to verify, which is a genuinely dangerous mistake in a test script.
• -partial_chain accepts a chain that stops at any trusted certificate rather than requiring a self-signed root. That is how you pin trust to a specific intermediate rather than a root — occasionally useful, and occasionally the reason someone's verification is looser than they think.
• -x509_strict turns off the compatibility workarounds OpenSSL applies by default. Worth running once against your own CA's output: if a certificate passes normally but fails under -x509_strict, you have a standards violation that some other client will eventually reject.

💡 -verify_ip and -verify_email are the siblings of -verify_hostname, for checking IP Address: and email: SAN entries. Same idea, different SAN type (Module 03, B3).

A sequence of certificates, each signed by the next, ending at a trust anchor in the client's store. At every link: signature, validity dates, name chaining (Issuer = next Subject), CA:TRUE, keyCertSign, pathlen, name constraints and revocation.

Two things people miss: every certificate must be in date, not just the leaf; and hostname matching is not part of chain validation — it is a separate check on the leaf's SAN.

The client builds and validates it, searching upward from the leaf using Issuer names and AKI until it reaches a trusted anchor. The server's only job is to supply the intermediates.

This tells you where to fix things: a path-building failure is almost always a server problem. Adding intermediates to a client's trust store fixes one client and leaves every other one broken.

No. Clients that trust it already have it; clients that do not will never trust it because a server offered it. Sending it wastes bytes on every handshake — Let's Encrypt cut certificate bytes over 40% by shortening its chain.

Send leaf + intermediates. Distribute the root to clients separately.

So the root key can stay offline — used a handful of times per decade, in an HSM or air-gapped machine. A key used for every issuance cannot be protected that way.

A compromised intermediate is revoked and replaced. A compromised root cannot revoke itself, and removal requires every browser and OS vendor to ship an update — years, in practice.

State: index.txt (the ledger of everything issued), a serial counter, and a copy of each certificate filed by serial.

That ledger is what makes revocation possible — a CRL is generated from it. A certificate the CA has no record of cannot be revoked.

It forbids any CA beneath this one. The client enforces it during path validation — the issuing CA will happily sign a sub-CA anyway, and openssl ca does not check its own pathlen.

Violating it produces error 25: path length constraint exceeded, reported at the depth of the certificate carrying the constraint.

By default nothing — and an internal root is as powerful as a public CA on every machine that trusts it. The control is name constraints: a critical extension listing permitted and excluded namespaces, e.g. permitted;DNS:.corp.test.

A leaf outside that space fails with error 47: permitted subtree violation. Pair it with pathlen:0 and EKU limits. Caveat: enforcement is client-side and a few older stacks ignore it, so treat it as defence in depth.

A second certificate for the same CA subject and public key, signed by a different root. A leaf then validates under either root, because the issuer is identified by key.

It exists because getting a new root into trust stores takes years. Let's Encrypt's ISRG Root X1 was cross-signed by DST Root CA X3 from 2015; when X3 expired in September 2021, clients relying on that path broke. The cross-sign was fully retired in 2024.

Not to path building — the client searches, so a bag of certificates works in any order. It matters for two other reasons.

The server reads the file positionally: nginx treats the first certificate as the leaf, so a reversed file makes it serve the intermediate as the server certificate. And RFC 8446 specifies leaf-first on the wire. Leaf first, then up, never the root.

An expired intermediate, an expired or retiring cross-signing root, a renewal that was never reloaded, the wrong certificate being served via SNI, or a revocation/OCSP failure.

One command settles most of it — print every certificate the server is actually serving and read their dates, rather than checking the file on disk.

20 unable to get local issuer certificate — missing intermediate or untrusted root. 21 unable to verify the first certificate — the same cause, seen from s_client. 25 path length constraint exceeded. 47 permitted subtree violation — a name constraint. 62 hostname mismatch — chain fine, wrong name.

Match on the number in scripts; the text changes between versions. The depth in the message tells you which link failed.

The root goes to clients — it is the trust anchor, delivered by config management or the base image, and into every runtime store (OS, Java cacerts, Node, Python) separately.

The intermediate goes to servers, served in fullchain.pem. Getting this backwards — distributing intermediates to clients — builds a hierarchy you can never rotate without touching every machine.

Question selection was cross-referenced against publicly published 2026 SSL/TLS and PKI interview question sets, then rewritten and deepened:

• ClimbTheLadder — 15 PKI Interview Questions and Answers
• InterviewPrep — Top 25 SSL Certificate Interview Questions and Answers
• JavaInUse — Top OpenSSL Interview Questions (2026)
• ClimbTheLadder — 10 SSL Certificate Interview Questions and Answers
• Devinterview.io — Web Security interview questions for 2026

Every command and every expected output in this module was executed on OpenSSL 3.0.13, including the complete two-tier CA build, the index.txt ledger format, and all four failure demonstrations — error 20 (missing issuer), error 21 (live server, missing chain), error 25 (path length constraint exceeded) and error 47 (permitted subtree violation). The openssl s_server -cert fullchain.pem behaviour noted in Exercise D1.1 — that it reads only the first certificate and needs -cert_chain for the rest — was discovered while testing this module rather than recalled.

Standards and history were verified against primary sources: RFC 5280 §6 and §4.2.1.9–4.2.1.10, the CA/B Forum Baseline Requirements, and Let's Encrypt's own documentation of its chains of trust, the cross-sign retirement, the DST Root CA X3 expiry and the Generation Y hierarchy.

Answers were rewritten and deepened rather than reproduced — published versions are usually correct but shallow, and the added operational detail is what actually differentiates a candidate in the room.