🤝Module 12 — mTLS & Internal PKI

no client cert -> HTTP 200

What to read out of this — the server let in a complete stranger, and that is the normal, correct behaviour of every website you have ever used.

• The server never asked who you were. It proved itself, and that was the end of the conversation about identity.
• --resolve is doing a small but important job. It tells curl to send api.lab.test to 127.0.0.1 without touching /etc/hosts. Module 07 introduced it; every exercise in this module uses it, so nothing on your machine is edited.
• -www makes openssl s_server behave like a tiny web server that replies with a status page. It is not a real web server — it exists so you can see a real HTTP response.

🔑 This is the baseline to compare everything against. Right now, anyone who can reach port 4433 gets a 200. In A3 the same command will fail during the handshake — and understanding why it fails, and where, is the whole point of the module.

💡 Now imagine this at 500 hosts. Five hundred services, all reachable by anything inside the network, all relying on the application to check a token. One service that forgets the check, or checks it wrongly, is a door into everything. mTLS makes the check a property of the connection rather than of each application's code — which means it cannot be forgotten in one place.

In normal TLS only the server presents a certificate, so only the server's identity is proved. The client is anonymous at the TLS layer, and the application authenticates it afterwards with a password, API key or token.

In mTLS the server also asks the client for a certificate, and the client proves it holds the matching private key. Both identities are established during the handshake, before any application data is sent.

The detail worth adding: the point people miss is when the check happens. With a bearer token the attacker still gets a completed connection and can hammer your endpoint; with mTLS an unknown client is rejected in the handshake and never reaches the application at all. That also means your application logs will not show the attempt — the evidence lives in the proxy or load balancer, which is something to plan for before you deploy it.

An API key is a bearer secret: it must be transmitted to be used, it must be stored by both sides, and anyone who obtains it becomes the client from anywhere in the world. A client certificate is not transmitted as a secret at all — the certificate is public, and the client proves possession of the private key by signing something. The private key never leaves the machine.

That changes the theft model completely. Stealing an API key can be as easy as reading a log file, a URL, or an environment variable in a crash dump. Stealing a certificate identity means getting a file off a specific host.

The detail worth adding: the honest caveat is that a private key sitting in a file on disk is still a file on disk, so mTLS raises the bar rather than removing the problem. What removes most of it is short lifetimes plus automatic rotation — a stolen key that expires in an hour is worth far less than an API key that has been valid since 2021. That combination is Part D, and it is the answer that shows you have run this rather than read about it.

Acceptable client certificate CA names
CN = Lab Client CA
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:Ed25519:...
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:...

What to read out of this — the server told you, in plain text, exactly which CA it trusts for clients.

• Acceptable client certificate CA names is the CertificateRequest message, decoded for you. The server is advertising CN = Lab Client CA.
• This list is public. Anyone who connects can read it, with no certificate of their own. That is worth knowing: your internal CA's name is visible to anyone who can reach the port.
• Requested Signature Algorithms is the server saying which key types and hashes it will accept for the CertificateVerify signature. A client with an old key type that is not on this list cannot authenticate even if its certificate is perfect.

🔑 This one command answers the most common mTLS support question there is. When somebody says "my client certificate is being rejected", run this first. If the CA name on the server does not match the issuer of their certificate, you have found the problem in ten seconds — and it is the problem about half the time.

💡 In nginx this list comes from the ssl_client_certificate file. Put a 200-certificate bundle there and the server advertises all 200 names on every handshake — which bloats the handshake and, in a browser, produces an enormous certificate-picker dialogue. Part C covers the directive that avoids this.

--- (a) no client certificate at all:
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.13: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0
--- (b) a certificate from an unrelated CA:
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.13: error:0A000418:SSL routines::tlsv1 alert unknown ca, errno 0
--- (c) the proper client certificate:
Client certificate
Certificate:
--- what the server logged:
depth=0 CN = rogue-svc, O = Elsewhere Ltd
verify error:num=20:unable to get local issuer certificate
depth=1 CN = Lab Client CA
depth=0 CN = payments-svc, O = Lab Ltd

What to read out of this — the two failures fail for completely different reasons, and the error text tells you which.

• certificate required — you sent nothing and the server insisted. This is the alert you will see most often in real life, and it means the client did not present a certificate, not that the certificate was bad.
• unknown ca — you sent something, and it did not chain to Lab Client CA. The rogue certificate is perfectly valid and correctly formed; it is simply from the wrong CA for this purpose. Wrong issuer, not wrong maths. The server log names the real reason — verify error:num=20, the same unable to get local issuer certificate from Module 07, now applied to a client.
• Client certificate followed by the full certificate — success. s_server echoes back what it received, which makes it an excellent debugging tool.
• ⚠️ The -verify_return_error flag is not optional, and leaving it out is a real trap. Without it, openssl s_server prints verify error:num=20 to its log and completes the handshake anyway. Your test then looks like it proves mTLS is working, when the server is in fact letting everyone in. If you build a lab and every certificate seems to be accepted, this flag is almost always the reason.

🔑 Learn these two alert strings. certificate required and unknown ca cover the large majority of mTLS failures, and they point at completely different fixes — the client is not configured versus the client is configured with the wrong certificate. Being able to tell them apart from a single line of curl output is genuinely most of mTLS troubleshooting.

💡 Notice that case (b) failed at the CA check, not the EKU check. The server never got as far as looking at Extended Key Usage, because the chain failed first. Validation stops at the first failure, so the error you see is the earliest problem, not necessarily the only one.

The same path validation as any certificate: the signature chains to a CA the server trusts for client authentication, the dates are valid, and the chain is well-formed. On top of that it checks Extended Key Usage allows clientAuth, and if a CRL is configured it checks the certificate has not been revoked.

The one thing it does not do is a hostname check. There is no hostname to check — a client is not reached by name, so there is nothing to compare a SAN against.

The detail worth adding: say clearly that the trust list for clients is separate from the server's own trust store, and that it should be. Trusting the public web PKI for client authentication would mean any certificate from any public CA gets in, which is a catastrophic misconfiguration and one that has happened in production. In nginx the directive is ssl_client_certificate, and it should contain your internal client CA and nothing else.

No. It gives you a verified identity, and nothing more. If the server trusts a CA, every certificate that CA has issued will pass — including ones issued to services that have no business talking to this endpoint.

Authorisation is a separate decision you have to implement: map the certificate's identity (the subject DN, a SAN, or an OU) to a set of permissions, and enforce it in the proxy or the application.

The detail worth adding: the failure mode is subtle and common. A team enables mTLS, sees that unknown clients are rejected, and concludes access control is done — but internally every service can now reach every other service, because they all share one client CA. The fix is either an explicit allow-list keyed on the identity, or issuing from per-purpose CAs or per-namespace identities so that the trust relationship itself carries meaning. SPIFFE in Part D exists largely to make that second option practical.

Compare the issuer of the client's certificate with the CA list the server advertises. openssl x509 -in client.crt -noout -issuer gives you one side; connecting with openssl s_client and reading the Acceptable client certificate CA names block gives you the other. If they do not match, that is the answer.

If they do match, the next suspect is a missing intermediate: the client is sending only its leaf certificate and the server cannot build the chain. The error moves to unable to verify the first certificate, OpenSSL error 21.

The detail worth adding: mention that this is the mirror image of Module 08's chain-order trap. There, the server had to send its intermediates; here the client does, and the same rule applies — each side sends everything except the root. It is a satisfying answer in an interview because it shows you see the symmetry rather than treating client certificates as a separate topic.

curl: (56) OpenSSL SSL_read: OpenSSL/3.0.13: error:0A000413:SSL routines::sslv3 alert unsupported certificate, errno 0
--- what the server logged:
depth=0 CN = api.lab.test
verify error:num=26:unsuitable certificate purpose

What to read out of this — the chain was perfect and it was still refused.

• verify error:num=26 — unsuitable certificate purpose. Add this to the verify-code table from Module 07. It means the maths is fine, the issuer is trusted, and this certificate is not allowed to do this job.
• Nothing about the chain was wrong. server.crt really was issued by server-ca.crt, and the server really did trust that CA. The single line that stopped it was extendedKeyUsage = serverAuth in the certificate.
• The alert on the wire is unsupported certificate, which is unhelpfully vague. The useful detail is always on the server side, in the log. Remember that when someone sends you a screenshot of a client error.

🔑 This is why you should always set EKU explicitly. A certificate with no EKU extension is treated as valid for every purpose — so leaving it out silently removes this protection. One line in your extensions file is the difference between a guardrail and no guardrail.

💡 Now imagine this at 500 hosts. Suppose someone points ssl_client_certificate at the public CA bundle "to make it work". Every certificate issued by every public CA on earth now chains successfully — but almost all of them are serverAuth only, so EKU quietly refuses them and the mistake looks like it is working. The day someone obtains a real clientAuth certificate from a public CA, they walk straight in. Do not rely on the guardrail; get the trust list right.

No. Keep them separate. The list of CAs a server trusts for client authentication is effectively the list of everyone who may connect, so it should contain your client CA and nothing else. If it also signs server certificates, every one of those becomes a potential credential.

They also have completely different lifecycles: server certificates come from a public CA and renew via ACME, while client certificates are yours and are issued as part of deployment.

The detail worth adding: mention extendedKeyUsage as the second line of defence. A serverAuth-only certificate offered for client authentication is refused with verify error 26, unsuitable certificate purpose — so EKU catches the mistake even when the trust configuration is wrong. But say clearly that it is a guardrail, not the design: a certificate with no EKU extension is valid for all purposes, so the protection disappears the moment someone omits it.

Pointing the client trust store at the public CA bundle. It usually happens because someone is debugging a unknown ca error and swaps in /etc/ssl/certs/ca-certificates.crt to make it go away. The moment they do, the server trusts every publicly-trusted CA in the world for client authentication.

In practice most public certificates are serverAuth only and get refused by EKU, so it appears to keep working — which is what makes it so dangerous. It is a hole that does not announce itself.

The detail worth adding: the general lesson is that in mTLS the trust file is the access control list, and it deserves to be reviewed like one. A good habit is to assert its contents in configuration management — this file contains exactly these CA certificates, and a diff is a change that needs approval, not a routine edit.

=== the same DN, three ways:
subject=CN = payments-svc, O = Lab Ltd
subject=O=Lab Ltd,CN=payments-svc
=== now a certificate that carries its identity in a URI SAN instead:
subject=
X509v3 Basic Constraints: critical
    CA:FALSE
X509v3 Extended Key Usage: 
    TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Alternative Name: critical
    URI:spiffe://lab.test/ns/payments/sa/checkout
notBefore=Aug 26 13:19:14 2026 GMT
notAfter=Aug 27 13:19:14 2026 GMT
=== pull the identity out in one line:
spiffe://lab.test/ns/payments/sa/checkout

What to read out of this — look at line 2 and line 3, then look at the empty Subject.

• CN = payments-svc, O = Lab Ltd versus O=Lab Ltd,CN=payments-svc. Reversed order, no spaces. Both describe the same certificate. Use -nameopt RFC2253 whenever you are building a rule that a proxy will evaluate.
• subject= is empty on the SVID, and that is deliberate. There is no CN at all. The identity is entirely in the URI SAN — which is why the SAN is marked critical: a client that cannot understand the SAN must not accept the certificate, because without it the certificate identifies nobody.
• serverAuth, clientAuth together. A workload is both. payments-svc calls other services and is called by them, so its one certificate has to work in both directions.
• One day of validity. That is not a typo, and it is the norm for workload identity. Part D explains how anything can be operated that way.

🔑 The URI SAN is easier to work with than a DN, and that is the whole argument for it. One flat string, one obvious way to write it, no ordering question, no escaping question, and it can encode structure — namespace, service account, environment — that a rule can pattern-match on. If you are designing an internal PKI now, put the identity in a URI SAN.

💡 Now imagine this at 500 hosts. With a DN scheme, an allow-list of 500 services is 500 exact strings that must each be spelled correctly in the proxy's config. With a URI scheme it can be one pattern — spiffe://lab.test/ns/payments/*. That difference is the difference between a rule set humans can maintain and one nobody dares touch.

In a URI SAN, if you are designing it today — spiffe://trust-domain/path. It is one unambiguous string, it can encode structure like namespace and service account, and it pattern-matches cleanly in proxy rules. The Subject can be left empty, in which case the SAN must be marked critical.

The traditional alternative is the CN, sometimes with O or OU for grouping. It works everywhere and is what most existing systems use, but a DN is a structured object being handled as a string, which invites bugs.

The detail worth adding: the concrete bug is worth naming because it is so common. openssl x509 -subject prints CN = payments-svc, O = Lab Ltd, while nginx's $ssl_client_s_dn prints the RFC 4514 form O=Lab Ltd,CN=payments-svc — reversed, with the spaces removed. Allow-lists built by copying OpenSSL's output silently never match. -nameopt RFC2253 gives you the string the proxy will actually see, and knowing that flag is a good sign you have debugged this for real.

Because in service-to-service traffic a workload is both. payments-svc receives calls, so it needs serverAuth; it also makes calls to other services, so it needs clientAuth. Issuing one certificate that does both is simpler than managing two, and it is what SPIFFE's X.509-SVID specification recommends.

For a certificate issued to a person, or to something that only ever initiates connections, clientAuth alone is the correct and tighter choice.

The detail worth adding: the trade-off is that a dual-purpose certificate cannot be used to enforce direction. If a workload's key is stolen, the attacker can both impersonate the service and call other services as it. Some high-security designs deliberately split them for that reason, at the cost of doubling the certificates to manage. Knowing the trade-off exists — rather than treating serverAuth,clientAuth as the default answer — is the mark of someone who has thought about it.

=== the ledger:
V	260925131828Z		1000	unknown	/CN=payments-svc/O=Lab Ltd
V	260925131828Z		1001	unknown	/CN=reporting-svc/O=Lab Ltd
=== the ledger after revoking one:
V	260925131828Z		1000	unknown	/CN=payments-svc/O=Lab Ltd
R	260925131828Z	260826131828Z,keyCompromise	1001	unknown	/CN=reporting-svc/O=Lab Ltd
=== the CRL:
Certificate Revocation List (CRL):
        Version 2 (0x1)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = Lab Client CA
        Last Update: Aug 26 13:18:28 2026 GMT
        Next Update: Sep  2 13:18:28 2026 GMT
        CRL extensions:
            X509v3 CRL Number: 
                4096
Revoked Certificates:
    Serial Number: 1001
        Revocation Date: Aug 26 13:18:28 2026 GMT

What to read out of this — your dates and serial hex will differ; the structure will not.

• V became R, with a revocation date and the reason. That single character is the entire state change, exactly as Module 09 described.
• Serial numbers are now sequential and recorded — 1000, 1001. With openssl x509 -req -CAcreateserial they were random and written nowhere useful. Revocation works on serial numbers, so a serial nobody recorded cannot be revoked.
• Next Update is seven days out, from default_crl_days = 7. Module 09's warning applies here too: a CRL past its Next Update is treated as broken, and a broken CRL will lock out every one of your clients at once.
• The subject came out as /CN=payments-svc/O=Lab Ltd, following the order in the [ pol ] section rather than the order you typed. This is where the RFC 4514 reversal from B2 comes from — check your allow-list strings after switching to openssl ca.

⚠️ copy_extensions = none is deliberate and important. With copy set, anything a requester puts in their CSR — including basicConstraints CA:TRUE — is copied into the issued certificate. That would let anyone who can request a client certificate mint themselves a CA. Module 05 covered this; it matters even more here, because client CSRs often come from other teams.

💡 Now imagine this at 500 hosts. index.txt is a flat text file with no locking. Two issuances at the same second corrupt it. That is not a criticism of OpenSSL — it was never meant to be a service — it is the exact point at which you stop hand-rolling and adopt one of the tools in Part D.

Because openssl ca keeps a database. Every issuance is recorded in index.txt with its serial number, expiry, subject and status, and serial numbers come from a counter rather than being random. That record is what makes revocation possible — a CRL lists serial numbers, so a certificate whose serial was never recorded cannot meaningfully be revoked.

openssl x509 -req just signs. It is fine for a lab or a one-off, and it is the wrong tool for anything you will later need to manage.

The detail worth adding: also mention copy_extensions. It defaults to none for good reason: with it set to copy, extensions from the CSR are carried into the certificate, so a requester could include basicConstraints=CA:TRUE and have your CA issue them a CA certificate. In an internal PKI where CSRs arrive from other teams, that is a privilege-escalation path, and it is a detail interviewers use to separate people who have read a tutorial from people who have run a CA.

As soon as issuance stops being a human action. index.txt is a flat file with no locking, so concurrent issuance corrupts it; the CA private key sits on disk with whatever permissions someone remembered to set; there is no audit trail beyond the file; and renewal is entirely manual.

Roughly, OpenSSL is fine while a person issues certificates occasionally and remembers each one. Once certificates are created by a deployment pipeline, or once lifetimes are short enough that renewal must be automatic, you need a service.

The detail worth adding: frame it as the same transition Module 10 described for public certificates. Once lifetimes drop below what a human can track, automation is not an efficiency improvement, it is the only way the system can work at all. That is why cert-manager, Vault and SPIRE exist, and it is why they all issue short-lived certificates by default rather than offering it as an option.

nginx: the configuration file .../nginx.conf syntax is ok
nginx: configuration file .../nginx.conf test is successful
=== (a) no client certificate:
HTTP 400
<html>
<head><title>400 No required SSL certificate was sent</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>No required SSL certificate was sent</center>
<hr><center>nginx/1.24.0 (Ubuntu)</center>
=== (b) with a valid client certificate:
verify=SUCCESS
subject=CN=payments-svc,O=Lab Ltd
issuer=CN=Lab Client CA
serial=38B99DC9309CBA95F8BD9531734E6A142D4A4609
expires_in=29 days

What to read out of this — three things, and the middle one is a surprise.

• 400 No required SSL certificate was sent. nginx completes the TLS handshake and then returns an HTTP 400, rather than rejecting at the TLS layer as openssl s_server did. That is a deliberate nginx design choice so it can give a readable error, and it means a browser shows a page rather than a blank connection failure.
• subject=CN=payments-svc,O=Lab Ltd — this run used the certificate from A3, issued with openssl x509 -req. After you switch to openssl ca in B3 the same certificate prints as O=Lab Ltd,CN=payments-svc, because the CA reordered the DN. The string your rules must match depends on how the certificate was issued. That is the B2 trap, appearing in the wild.
• expires_in=29 days is free monitoring. Log $ssl_client_v_remain on every request and you will never again be surprised by a client certificate expiring — which, in mTLS estates, is the single most common outage.

🔑 ssl_verify_client defaults to off. Setting ssl_client_certificate alone changes nothing at all — the server will happily serve everybody. This is the "we configured mTLS but never enabled it" failure, and it is common enough that the capstone in Part E is built to detect it.

💡 The certificate serial here is a long random hex string because -CAcreateserial generated it. After B3 it becomes 1000, 1001 and so on. Sequential serials make a CRL readable by a human; random ones do not.

=== (a) client sends its leaf ONLY:
HTTP 400
client SSL certificate verify error: (21:unable to verify the first certificate)
=== (b) same client, with the intermediate attached:
verify=SUCCESS
subject=CN=deep-svc,O=Lab Ltd
issuer=CN=Lab Client Issuing CA
serial=6073B53D919A264D6E6D2DF8E6F374B2CADB190C
expires_in=29 days

What to read out of this — the certificate was identical in both runs. Only what was sent alongside it changed.

• Error 21, unable to verify the first certificate. The same code Module 07 introduced and Module 08 saw on the server side. The server has the root; it does not have the intermediate; it cannot join the two.
• The fix is cat leaf.crt intermediate.crt > chain.crt, in that order. Leaf first, then up the chain, root omitted. Precisely the Module 08 rule, applied to the client.
• The error only appears in the server's log. The client sees a bare 400. When someone reports "mTLS is broken", the useful information is always on the server — this is worth saying out loud before anyone spends an hour reading client-side output.

⚠️ The second trap: ssl_verify_depth defaults to 1 in nginx and 10 in Apache. A depth of 1 allows exactly one intermediate, which covers the common root → issuing-CA → client shape. Add a second tier and nginx starts refusing chains that Apache accepts, with no obvious clue as to why. If you have a three-tier internal PKI, set ssl_verify_depth 2 explicitly and do not rely on the default.

🔑 The rule to memorise, because it covers both directions. Each side sends its own certificate plus every intermediate above it, and never the root. Servers do it with fullchain.pem (Module 08). Clients do it with a concatenated --cert file. One rule, two places.

💡 Now imagine this at 500 hosts. When an internal PKI rotates its intermediate, every client's chain file must be rebuilt — not just the leaf. Automation that renews only the leaf produces a fleet of certificates that are individually valid and collectively unusable, and it fails all at once on the day the old intermediate is removed from the server's trust file.

Two directives. ssl_client_certificate points at the CA certificate you trust to sign client certificates, and ssl_verify_client on requires a valid client certificate. Optionally ssl_verify_depth if your PKI has more than one intermediate tier, and ssl_crl for revocation.

After that, $ssl_client_verify, $ssl_client_s_dn and the other $ssl_client_* variables carry the identity into your configuration or your application.

The detail worth adding: two defaults are worth naming because both cause real incidents. ssl_verify_client defaults to off, so configuring the CA file alone silently does nothing — the classic "we deployed mTLS" that never enforced anything. And ssl_verify_depth defaults to 1 in nginx while Apache defaults to 10, so a chain that works behind Apache can be rejected by nginx after a migration.

Most likely the client is sending only its leaf certificate and not the intermediate that issued it. The server has the root but cannot bridge the gap, so path building fails with OpenSSL error 21, unable to verify the first certificate. The fix is to concatenate leaf then intermediate into the file passed to --cert.

The other candidate is ssl_verify_depth, which defaults to 1 in nginx — a PKI with two intermediate tiers exceeds it.

The detail worth adding: the diagnostic habit matters as much as the answer. The client sees a bare 400; the reason only appears in nginx's error_log, and only at info level or lower for some of these messages. Saying "I'd raise the error log level and read the server side first" is a better answer than any amount of certificate theory, because it is what actually resolves the ticket.

=== /public, no certificate:
public: verify=NONE
=== /private, no certificate:
401 no valid client certificate
=== /private, payments-svc:
200 welcome O=Lab Ltd,CN=payments-svc
=== /private, reporting-svc:
403 O=Lab Ltd,CN=reporting-svc is authenticated but not authorised

What to read out of this — the last two lines are the entire point of the section.

• 401 and 403 mean different things and you should keep them different. 401 is I do not know who you are. 403 is I know exactly who you are, and no. Collapsing both into one response makes every future support conversation harder.
• reporting-svc had a perfectly valid certificate from a trusted CA and was still refused. That is authorisation working. Without the map, it would have been let in.
• verify=NONE on /public shows what optional does: the check ran, found nothing, recorded NONE, and carried on.
• The map key is "O=Lab Ltd,CN=payments-svc". If you get 403 for a client you meant to allow, print $ssl_client_s_dn and copy the string from there — do not type it from the CSR. This is B2's trap, and it is the single most common reason this pattern does not work first time.

🔑 map is the right tool here and if is not. nginx's if inside location is notoriously surprising, and it is used above only because it keeps the example short. In production, do the decision in map — which is evaluated once, cheaply, and predictably — and use it to select an upstream, a return, or an auth_request. "Use map, avoid if" is a good thing to be able to say about nginx generally.

💡 Now imagine this at 500 hosts. A map with 500 exact DN strings is unmaintainable and will drift. This is the practical argument for the URI-SAN identities from B2: one regex over spiffe://lab.test/ns/payments/* replaces fifty lines and keeps working when a new service is added to that namespace.

When one server block serves both public and protected content, or when you need different rules for different identities rather than one rule for the whole server. optional requests and verifies the certificate but does not reject anyone, recording the outcome in $ssl_client_verify so you can act on it per location.

on is simpler and safer where it fits, because nginx does the enforcement and you cannot forget to.

The detail worth adding: the danger is that optional moves responsibility to you, and any protected path where you forget the check is simply open. The specific bug to name is writing the check as if ($ssl_client_verify = NONE) — with optional_no_ca an untrusted certificate produces FAILED:..., not NONE, so that test passes an attacker straight through. Always test positively for SUCCESS.

Extract a stable identity from the certificate — the subject DN, an OU, or ideally a URI SAN — and map it to permissions. In nginx that is a map block keyed on $ssl_client_s_dn feeding a decision; at larger scale it is a policy engine, or a service mesh doing it centrally.

The key design choice is what the identity looks like. Exact DN strings mean one rule per service and constant maintenance; structured URI identities let you write rules over namespaces and environments.

The detail worth adding: say where the decision should live. Enforcing in each application means every team must implement it correctly, which is exactly the problem mTLS was supposed to solve. Enforcing at the proxy or mesh gives you one implementation, one audit point and one place to change — and it is why service meshes ended up owning this. If the interviewer pushes on defence in depth, the honest answer is that critical services should do both, and that most estates do neither consistently.

=== 1. through the proxy, with a real certificate:
app sees identity: O=Lab Ltd,CN=payments-svc
=== 2. through the proxy, spoofing the header, no certificate:
HTTP 400  (the proxy rejected it)
=== 3. straight to the application, past the proxy entirely:
app sees identity: O=Lab Ltd,CN=payments-svc

What to read out of this — lines 1 and 3 are identical, and one of them required no certificate at all.

• Case 2 shows the proxy doing its job. You cannot smuggle the header past it: proxy_set_header overwrites whatever the client sent, and in any case the connection is rejected before it gets that far.
• Case 3 is the real vulnerability, and there was no flaw in the TLS anywhere. The mTLS was correct, the certificate check was correct, the header handling was correct. The application was simply reachable another way.
• The application cannot tell the two apart. From its side both requests are identical plain HTTP with the same header. There is nothing it could check.

⚠️ This is the most common real-world way an mTLS deployment is defeated, and it is almost never found by testing the mTLS. It is found by asking a different question: "what else can reach this port?" Container networks, Kubernetes services, cloud load balancers with a second listener, a debug port someone opened in 2024 — all of them produce case 3.

🔑 The two-part rule, worth saying in an interview. Identity headers are only as trustworthy as the network path. Set them unconditionally at the proxy, and make the backend unreachable except through it. If you can only do one, do the second — a backend nobody can reach directly is safe even with sloppy header handling, whereas perfect header handling on a publicly reachable backend protects nothing.

💡 Now imagine this at 500 hosts. Verifying "nothing can reach the backend directly" 500 times, by hand, forever, is not realistic. This is the argument for a service mesh, where mTLS runs all the way to each workload's sidecar rather than terminating at one shared edge — there is no plain-HTTP segment left to sneak into. Part D.

=== payments-svc (not revoked):
HTTP 200
=== reporting-svc (revoked in B3):
HTTP 400
client SSL certificate verify error: (23:certificate revoked)

What to read out of this — compare it with Module 09 and notice how much simpler this is.

• Error 23, certificate revoked. Another entry for the Module 07 table. Note it is a verification error like any other — revocation is just one more gate in the same path validation.
• It took effect immediately. No OCSP responder, no soft-fail, no waiting for a cache to expire. nginx reads the CRL file and applies it.
• reporting-svc's certificate is still cryptographically perfect. Nothing about the file changed. The only thing that changed is a line in your CA's ledger and a serial number in a file on the server.

⚠️ The failure mode is the opposite of the public web's, and it is worse. Public revocation fails open — nobody notices. Internal CRL checking fails closed: nginx will not start, or will reject every client, if the CRL file is missing, malformed, or past its Next Update. Module 09's warning applies with full force — monitor Next Update, not whether the file exists. A CRL that expires at 3am takes down every client at once.

🔑 nginx reads the CRL at load time, not per request. Regenerating the file changes nothing until you reload nginx (nginx -s reload). Any revocation runbook must include the reload, on every server, or the revocation is decorative.

💡 Now imagine this at 500 hosts. Distributing a CRL file to 500 servers and reloading each of them, within your promised revocation window, is a real engineering problem. This is precisely why modern internal PKI leans on short lifetimes instead of revocation — a one-hour certificate does not need a CRL, because it is gone before you could have distributed one. That is the central idea of Part D.

Because the reasons it fails publicly do not apply. On the web, the client is a browser you do not control, checking with a CA you do not control, over a network that may not reach it — so browsers soft-fail and ignore the answer. Internally, you own the CA, you own every server doing the checking, and the CRL is a local file. There is no network call to fail, so there is nothing to soft-fail on.

In nginx it is one directive, ssl_crl, and a revoked client is refused with verify error 23.

The detail worth adding: flip the risk around, because that is the part people miss. Internal CRL checking fails closed — a missing, malformed or expired CRL rejects every client simultaneously. Public revocation risks letting a bad certificate in; internal revocation risks a total outage. So the thing to monitor is the CRL's Next Update, and the thing to automate is regenerating and redistributing it well before then.

Issue a replacement certificate with a brand-new key and deploy it first, so the service keeps working. Then revoke the compromised certificate in the CA, regenerate the CRL, distribute it to every server that checks, and reload each one — a revocation that has not reached the servers has not happened. Then find out how the key leaked, because that is the actual incident.

Rotate in that order: replace, then revoke. Revoking first takes the service down.

The detail worth adding: the answer that shows range is to ask how long the certificate had left to live. If your PKI issues one-hour certificates, the honest response may be to do nothing but rotate — the leaked credential expires before a CRL could realistically be distributed to the fleet, and skipping the revocation dance is faster and less risky than performing it. Recognising when revocation is not worth doing is a more senior answer than reciting the procedure.

lifetime  per service/day   fleet of 500/day   per minute
  1 year            0.005                  3         0.00
 90 days            0.022                 11         0.01
 30 days            0.067                 33         0.02
24 hours            2.000               1000         0.69
  1 hour           48.000              24000        16.67

What to read out of this — find the row where the number stops being a task and becomes a rate.

• At 1 year, a 500-service fleet renews about 3 times a day. A person could genuinely do that. Badly, and with the occasional outage, but they could.
• At 30 days it is 33 a day. Now it is somebody's job, and that person will make mistakes.
• At 1 hour it is 17 per minute, forever. There is no version of this that a human touches. The renewal path must be as reliable as the service itself, because it is the service.
• The middle rows are the dangerous ones. A 30-day certificate is short enough to renew constantly and long enough that people convince themselves a calendar reminder will do. That is where outages live.

🔑 The insight to carry into the rest of Part D: automation is not an optimisation, it is a precondition. You do not shorten lifetimes and then automate. You build the issuing service first, and the short lifetime becomes something you can simply switch on. Every tool in this part is built that way round — they all default to short lifetimes because they assume nothing manual is involved.

💡 The same reasoning drove the public web. Module 10 covered the CA/Browser Forum lifetime schedule dropping to 47 days by 2029, and Module 11 covered the Chrome Root Program requiring every subordinate CA to be integrated with automation by 15 March 2027. Internal and public PKI reached the identical conclusion independently: short lifetimes plus automation beats long lifetimes plus revocation.

=== the three files a cert-manager Secret would hold:
tls.key  <- cm.key
tls.crt  <- cm.crt
ca.crt   <- client-ca.crt
=== and what is inside tls.crt:
notBefore=Aug 26 13:40:11 2026 GMT
notAfter=Aug 27 13:40:11 2026 GMT
X509v3 Subject Alternative Name: critical
    URI:spiffe://lab.test/ns/payments/sa/checkout
X509v3 Extended Key Usage: 
    TLS Web Client Authentication, TLS Web Server Authentication

What to read out of this — there is no magic in cert-manager. It runs these commands for you, on a loop.

• The Secret is three files with fixed names: tls.crt, tls.key, ca.crt. Anything that can read a certificate from disk can consume it; nothing needs to know about cert-manager.
• duration: 24h is -days 1. usages is the EKU from B1. uris is the URI SAN from B2. Every field maps onto something you have already built by hand.
• A new key was generated, matching rotationPolicy: Always. Reusing cm.key on the next issuance would be the default behaviour you do not want.

🔑 This is the mental model worth keeping for every tool in Part D. cert-manager, Vault and SPIRE are not doing anything you have not now done manually. They are a control loop around it: keep this thing true, notice when it stops being true, fix it. The value is entirely in the loop, and that is exactly the part you cannot build with a cron job and hope.

💡 If you have a cluster to hand, the real version is helm install cert-manager jetstack/cert-manager --set crds.enabled=true, then a ClusterIssuer of kind: CA pointing at a Secret holding client-ca.crt and client-ca.key. Do not do that on a work laptop's cluster without asking — it installs cluster-wide CRDs and a controller with broad permissions.

With cert-manager. You define a ClusterIssuer describing where certificates come from — an internal CA, Vault, or an ACME server — and a Certificate resource declaring what should exist. A controller reconciles that continuously, writing tls.crt, tls.key and ca.crt into a Secret and renewing before expiry without anyone asking.

It is declarative in the Kubernetes sense: you state the desired end state and the controller keeps reality matching it, including after a failure.

The detail worth adding: name two fields, because they are where real incidents come from. privateKey.rotationPolicy defaults to Never, so renewals reuse the same key — which quietly removes most of the benefit of short lifetimes, since a stolen key stays valid indefinitely. And renewBefore must exceed your worst-case deployment or outage window; a 24-hour certificate with a one-hour renewBefore gives you one hour of slack for the entire cluster.

The Secret was updated and the process did not reload. Most TLS libraries read the certificate file once at startup and hold it in memory, so a new file on disk changes nothing. There is also a delay before Kubernetes refreshes a projected volume — often a minute or more — so even the file is not instantly current.

The fix is either a server that watches and reloads (nginx and Envoy do), a sidecar that signals it, or accepting that renewal means a rolling restart.

The detail worth adding: the reason this bites so hard with short lifetimes is that the gap between "renewed" and "actually serving" used to be irrelevant. With a one-year certificate, a restart happens naturally long before expiry. With a 24-hour certificate the reload path has to be deliberate and tested — and it is the single most common reason a first cert-manager rollout causes an outage rather than preventing one.

=== a well-behaved request:
  short-svc -> notAfter=Aug 26 14:34:32 2026 GMT
=== a request that exceeds the role's ceiling:
  (asked for 720h, role ceiling is 24h)
  greedy-svc -> notAfter=Aug 27 13:34:32 2026 GMT

# for reference, the clock at the time of the run:
Wed Aug 26 13:34:32 UTC 2026

What to read out of this — the greedy request succeeded. It just did not get what it asked for.

• Nothing failed. The caller asked for 30 days and received a valid certificate for 24 hours. That is exactly how Vault behaves, and it is why the trap above matters: a silently shortened lifetime looks like success.
• -startdate and -enddate are how you set an exact validity window with openssl ca. Module 07 introduced them when openssl x509 -req turned out not to support -not_before on OpenSSL 3.0. Same technique, different use.
• The policy lives with the issuer, not the caller. The issue function decides; the caller only asks. That single inversion is most of what a PKI service gives you over a shared script.

🔑 The habit this should leave you with: verify the certificate you got, not the request you sent. Check notAfter on the first certificate from any new role, issuer or automation. Every PKI service in this part will quietly give you something smaller than you asked for under some configuration, and none of them will tell you.

💡 Now imagine this at 500 hosts. The ceiling is what stops one team's ttl=8760h from creating 500 year-long credentials nobody tracks. Without it, the first person who finds renewals annoying will simply ask for a longer certificate, and your short-lifetime design quietly evaporates over about six months.

Policy and identity. A role constrains what can be issued — allowed domains and URI SANs, key type, EKUs, and a maximum TTL — so the rules live with the CA rather than with each caller. And Vault's own authentication decides who may reach which role, so a workload gets a certificate by proving it is that workload, using whatever identity it already has: a Kubernetes service account, AppRole, or cloud IAM.

It also handles the CA key properly: audited access, a real storage backend, and no key file sitting on a build server.

The detail worth adding: the capability worth naming is ephemeral certificates. Vault can generate the private key and return it over the API, so the key need never touch disk — the workload holds it in memory for an hour and it is gone. That closes the last gap in the mTLS story, because a private key in a file is still a file someone can steal. Vault's docs also make the operational point directly: short TTLs mean revocation is rarely needed, which keeps CRLs small.

At the mount's max_lease_ttl, set when the PKI engine was enabled. It is a hard ceiling above every role beneath it, so a role with max_ttl=24h under a mount enabled with -max-lease-ttl=1h issues one-hour certificates and reports no error at all.

Confirm it by reading notAfter on an issued certificate rather than trusting the role definition.

The detail worth adding: generalise it, because the same class of bug appears everywhere in PKI. Requested lifetimes are ceilings negotiated at several layers — the CA's own expiry caps everything beneath it, an intermediate caps its leaves, a mount caps its roles, a role caps its callers, and public CAs cap by policy. When a lifetime is not what you asked for, work down that stack. And the reason it goes unnoticed is that every layer silently truncates rather than refusing.

== svid.crt
  exactly one URI SAN     : ok
  SPIFFE ID               : spiffe://lab.test/ns/payments/sa/checkout
  basicConstraints CA:FALSE: ok
  keyUsage digitalSignature: ok
  EKU clientAuth           : ok
  empty Subject + critical SAN: ok

== payments-svc.crt
  exactly one URI SAN     : FAIL (0 found)
  SPIFFE ID               : FAIL (no spiffe:// URI)
  basicConstraints CA:FALSE: ok
  keyUsage digitalSignature: ok
  EKU clientAuth           : ok
  Subject                  : CN = payments-svc, O = Lab Ltd (allowed, but the SPIFFE ID is what counts)

What to read out of this — payments-svc.crt is a perfectly good client certificate that is not an SVID, and the difference is only where the identity lives.

• Everything about payments-svc.crt is valid. It works in every exercise in this module. It simply carries its name in a DN instead of a URI SAN, so it cannot be used in a SPIFFE trust domain.
• svid.crt has an empty Subject and a critical SAN. Those two go together: if the only identity is in the SAN, nothing may be allowed to ignore the SAN.
• The checker is about 25 lines of shell. That is worth noticing — an SVID is not a new certificate format. It is an ordinary X.509 certificate with a naming convention and a handful of rules.

🔑 The idea to take away even if you never deploy SPIRE: identity should be derived from what a workload is, not given to it as a secret. Any scheme where the credential is baked into an image, injected at deploy time, or stored in a file the workload reads has a long-lived secret at the bottom of it. Attestation replaces that with a fresh observation each time, which is why SVIDs can safely last an hour.

💡 Now imagine this at 500 hosts. Structured identities are what makes 500 tractable. spiffe://lab.test/ns/payments/* is one authorisation rule covering every service in a namespace, present and future. The DN equivalent is 500 exact strings that must each be maintained by hand — the same point C3 reached from the other direction.

SPIFFE is a standard for workload identity. A workload gets a SPIFFE ID — a URI like spiffe://trust-domain/path — and proves it with an SVID, which for X.509 is an ordinary certificate carrying that ID in a URI SAN. SPIRE is the common implementation: a server holding the CA, and an agent on each node that hands out SVIDs over a local socket.

The problem it solves is bootstrapping. Normally a workload needs a credential to obtain a credential, and teams break the loop by baking a long-lived secret into an image or dropping one on disk.

The detail worth adding: explain attestation, because it is the actual idea. The SPIRE agent identifies a workload by observing what it demonstrably is — its process, container, Kubernetes service account, the node's cloud identity — and issues on that basis. The workload never holds a secret it was given; it receives a fresh, short-lived one through a socket. That is what makes one-hour certificates practical, and it means there is no long-lived credential at the bottom of the stack to steal.

So that "who is this?" has exactly one answer. A certificate with two SPIFFE IDs would let a verifier and an authoriser reach different conclusions about the same connection, which is a security hole rather than a flexibility feature. The specification requires validators to reject certificates with more than one URI SAN.

The related rule is that if the Subject is empty — which is normal for SVIDs — the SAN must be marked critical, so nothing can accept a certificate while ignoring the only field that identifies it.

The detail worth adding: contrast it with a server certificate, where multiple SANs are entirely normal because a server legitimately answers to many names. The asymmetry is the point: a server is a set of names, a workload is one identity. Noticing that the two ends of a connection have genuinely different naming needs is a good sign you understand what SANs are actually for.

X509v3 Name Constraints: critical
    Permitted:
      DNS:.lab.test
      URI:.lab.test
--- inside the constraint:
ok.crt: OK
--- outside the constraint:
CN = www.google.com
error 47 at 0 depth lookup: permitted subtree violation
error bad.crt: verification failed

What to read out of this — the CA happily signed the google.com certificate. It is the verifier that refuses it.

• error 47 — permitted subtree violation. One more for the Module 07 table. The certificate is correctly signed by a CA in the chain, and it is still rejected, because the CA was not permitted to sign that name.
• Signing succeeded and verification failed. That distinction is worth holding on to: name constraints are enforced by whoever checks the chain, not by the CA. So the protection works even if your CA software is compromised, as long as the constraint is baked into the CA certificate — which is exactly why it must be set at creation and cannot be added afterwards.
• DNS:.lab.test with a leading dot means "this domain and everything under it". svc.lab.test matches; lab.test.evil.com does not.
• pathlen:0 stops this CA from creating further CAs, which would otherwise be a way around the constraint on some clients.

🔑 This is the highest-value line in the whole module, per character typed. One nameConstraints line turns a CA that could impersonate the entire internet into one that can only impersonate you. If you take a single practical action away from Module 12, make it adding name constraints to every internal CA you create from now on.

⚠️ Support is good but not universal. OpenSSL, Firefox, Chrome, Go and macOS enforce name constraints. Some older embedded stacks and a few Java versions historically did not. Treat it as a strong extra layer rather than your only defence — but there is no reason not to have it.

💡 Now imagine this at 500 hosts. The constraint is set once, on one CA certificate, and every one of the 500 clients enforces it independently with no configuration. That is a rare shape in security: a control you apply in one place that is checked everywhere, and that keeps working even if the thing you are protecting against is the CA itself.

For internal traffic only, and only when nothing else fits. Anything facing the public internet should use a public CA via ACME. Inside Kubernetes, cert-manager or a service mesh should own it. If you already run Vault or your cloud provider offers a private CA, use those — you are paying them to hold the CA key safely, which is the genuinely hard part.

Running your own makes sense when you are air-gapped, under a regulatory constraint, or need identities that no public CA will issue — SPIFFE IDs, for example.

The detail worth adding: the cost people underestimate is not issuance, it is custody over time. The key must stay safe for a decade, through staff turnover and at least one migration, and somebody must know how to rotate it. Saying "I would want a written, tested rotation procedure before I created the CA, not after" is the answer of somebody who has watched a CA expire in production.

A nameConstraints extension on a CA certificate limits the names that CA may issue for — for example permitted;DNS:.lab.test. Any certificate outside that set fails validation with error 47, permitted subtree violation, on every client that enforces it.

It matters because an internal CA installed in your machines' trust stores can otherwise issue a valid certificate for any domain on earth, including your bank, and those machines will accept it. A constraint reduces that from "the whole internet" to "our own names".

The detail worth adding: the property that makes it powerful is that it is enforced by the verifier, not the issuer. The CA will happily sign a certificate for google.com; every client will reject it. So the protection survives the CA itself being compromised — which is the threat you actually care about. The catch is that it must be set when the CA certificate is created and cannot be added later without reissuing the CA and everything under it, so it is a day-one decision.