☁️Module 08 — DNS in the Cloud & in Kubernetes

Not with a CNAME — the apex must hold SOA and NS, and a CNAME excludes all other types at the same name. In Route 53 you use an alias record; other providers call it ALIAS, ANAME or CNAME flattening.

The provider resolves the target internally and returns a genuine A or AAAA, so nothing non-standard goes on the wire.

Two consequences worth volunteering. These records are invisible from outside — dig shows a plain A — so on an inherited estate the only way to find them is the provider's API, and they must be re-created by hand during any provider migration. And the flattening happens from the provider's network position, which for a geo-distributed target can hand your users a badly chosen address.

Yes, and it works — but not quickly, and the distinction matters more than the mechanism. Route 53 failover routing swaps the answer when a health check fails; weighted and multivalue policies do related jobs.

The limit is caching. Resolvers hold the old answer for the TTL, resolver-side min-cache-ttl floors can extend that, serve-stale can extend it further, and application-level caches ignore TTLs entirely — a JVM caches for the process lifetime by default.

So DNS moves traffic eventually, on the order of the TTL and often longer. Anything needing sub-second recovery belongs in an anycast address, a load balancer, or client-side retry logic. Designing a failover whose correctness depends on clients honouring a 30-second TTL is the mistake this question is testing for.

Serving different answers for the same name depending on who asks — internal clients get private addresses, external clients get public ones or nothing. Route 53 private hosted zones scoped to a VPC, BIND views, or separate internal resolvers.

It is the right way to keep internal names off the public internet, because the names are absent from the public zone rather than merely obscured. NSEC3 in a signed public zone only raises the cost of enumeration; split-horizon removes the data.

The operational cost is the part worth naming: it makes "it works for me" meaningless. Every DNS test result now depends on where it was run, so a name that resolves inside a VPC and NXDOMAINs from a laptop is not a broken record — it is the design working. Reproducing from inside the correct view is the first step in any such investigation.

$ cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

api.example.com.       4 ms      <- trailing dot: ONE query
api.example.com        31 ms     <- no dot: four queries, three of them wasted

$ getent hosts kubernetes.default.svc.cluster.local
10.96.0.1       kubernetes.default.svc.cluster.local

Run this in a real cluster — the exact numbers vary, the ratio does not.

One character changed the latency by roughly 8×, and it is the same trailing dot from Module 01 A3. Three NXDOMAIN round trips to CoreDNS were eliminated by making the name absolute.

Multiply it out. A service doing 1,000 external lookups a second sends 4,000 queries per second to CoreDNS instead of 1,000 — and 3,000 of them exist only to be told NXDOMAIN. At cluster scale this is the largest single source of CoreDNS load, and the reason teams scale CoreDNS up when the real problem is a search list.

It also ties three earlier modules together. Module 06 B3 gave you the ndots rule. Module 03 B2 explained that those NXDOMAINs are negatively cached, which is the only reason this is survivable at all. And Module 01 A3 gave you the fix.

Two things combine. Pods get options ndots:5, so any name with fewer than five dots — essentially every external name — goes through the search list first: three NXDOMAIN queries before the real one, and since most clients send A and AAAA in parallel, eight packets instead of two.

Then a known conntrack race in the kernel can drop one of those parallel UDP packets. The client cannot detect loss, so it waits out the default 5-second resolv.conf timeout before retrying.

The framing that shows real understanding: it is not a CoreDNS fault. ndots:5 multiplies the query count until a rare race becomes routine. The cheapest fix is a trailing dot on external names, making them absolute so the search list is skipped; then ndots:2 via dnsConfig; then NodeLocal DNSCache, which also moves the traffic to TCP and removes the race entirely.

A ClusterIP Service resolves to one stable virtual address, with kube-proxy load-balancing behind it. A headless Service — clusterIP: None — resolves to the actual addresses of the ready pods, one A record each. StatefulSets add stable per-pod names on top.

Headless is what you need when clients must address individual instances: a database with a primary, Kafka brokers, anything doing its own sharding.

The operational catch: a headless RRset changes on every scale event and rollout, so it inherits every caching problem in DNS. Kubernetes serves a 30-second TTL, but a client that ignores TTLs — a JVM with default settings, or a connection pool — keeps addressing pods that no longer exist. With ClusterIP that cannot happen, because the address is stable and the balancing is elsewhere.

CoreDNS forwards unresolved queries to the node's /etc/resolv.conf. If that file points at a local stub listener — 127.0.0.53 from systemd-resolved, or 127.0.0.1 — then inside the CoreDNS pod that address is the pod itself, so CoreDNS forwards to itself. The loop plugin detects this at startup and exits deliberately.

Fix: point the kubelet at the file holding the real upstream nameservers, --resolv-conf=/run/systemd/resolve/resolv.conf, or set explicit forwarders in the Corefile.

What makes this a good answer rather than a recalled one: it is the same misunderstanding as "why does my resolv.conf say 127.0.0.53" — a stub address naming a local process rather than a server — surfacing two layers away as a cluster outage. Naming that connection shows you understand the system, not just the error string.

1. cat /etc/resolv.conf shows options ndots:5 and a three-entry search list. api.example.com has two dots — fewer than five — so it is treated as relative: three suffixed queries returning NXDOMAIN, then the real one. Four queries, and eight packets once A and AAAA are sent in parallel. The five seconds is the default timeout:5 being waited out when the conntrack race drops one of those parallel UDP packets.

2. A trailing dot: api.example.com. It makes the name absolute, so ndots never applies and the search list is skipped. Module 01 A3 taught it, in the very first module, as "the trailing dot means send this name as written".

3. dnsConfig with ndots: "2" — fixes the amplification cluster-wide rather than name-by-name, so it covers third-party code you cannot edit. NodeLocal DNSCache — addresses something different: it moves the CoreDNS conversation to TCP, which removes the UDP conntrack race itself rather than merely reducing how often it is hit.

4. Because CoreDNS is probably not the bottleneck. The load is self-inflicted: 75% of the queries exist only to receive NXDOMAIN. Scaling CoreDNS adds capacity to serve queries that should never have been sent, and does nothing about the packet-loss race, which is where the five seconds comes from. Fix the query count first, then measure.

5. Split-horizon, or a private hosted zone not attached to the production VPC. Both produce "works here, NXDOMAIN there".

The command that distinguishes them: run the same query from inside the production VPC, and compare with the provider's API export of the private zone. If the record exists in the zone but NXDOMAINs from prod, the zone is not attached to that VPC. If it does not exist in the zone at all, it was only ever in the staging view.

dig from your laptop proves nothing here — Module 06 D1's "ask from where" rule.

6. Build: a Route 53 alias record at the apex pointing at the load balancer — a CNAME is illegal there — plus failover routing with a health check, and a low TTL on any non-alias record involved.

Refuse to promise: 30-second failover. DNS moves traffic at the pace of the slowest cache holding the old answer, and that is not bounded by your TTL: resolver-side min-cache-ttl floors, serve-stale, and application caches that ignore TTLs entirely all extend it. Sub-minute recovery must come from the load balancer or an anycast address, with DNS as the slow, eventual layer.

7. 10.0.48.20 is RFC 1918 — a private address in a public zone. It is useless to external clients and it leaks internal topology: subnet ranges, and by implication how the network is laid out. It is a private record that escaped its horizon, usually because someone added it to the wrong zone.

Fix: move it to a private hosted zone attached to the VPCs that need it, and add a CI check that rejects RFC 1918 addresses in any public zone. This is exactly the finding from Module 02 D5.

The five things most people miss:

1. Requirement 1's arithmetic — saying "ndots is wrong" without counting to four and then eight
2. Requirement 3's distinction. NodeLocal is not "the same fix but bigger": it removes the race, while ndots removes the amplification
3. Requirement 4 at all. "Scale it up" is the reflex, and resisting it with an argument about where the load comes from is the answer
4. Requirement 6's refusal. Saying what you cannot promise is the senior half of the answer
5. Requirement 7's second sentence. "It's a private IP" is the observation; "it leaks internal topology and is useless externally" is the finding

The apex must carry SOA and NS, and a CNAME excludes all other types at the same name. Route 53 offers alias records; other providers call it ALIAS, ANAME or CNAME flattening.

The provider resolves the target internally and returns a real A or AAAA, so nothing non-standard goes on the wire.

Consequences: invisible to dig, non-portable between providers, and resolved from the provider's network position rather than the user's.

The resolver's address, not the user's. A user in Kuala Lumpur on a Singapore-based ISP resolver looks like a Singapore user.

EDNS Client Subnet partially fixes this by forwarding a truncated client prefix, but it is optional and not universally supported.

This is why geolocation and latency routing are approximate, and why they behave oddly for anyone using a large public resolver.

Because resolvers hold the old answer for the TTL, resolver-side min-cache-ttl floors can extend it, serve-stale can extend it further when your servers are unreachable, and application caches ignore TTLs entirely.

So DNS moves traffic eventually, on the order of the TTL and often longer. Sub-second recovery has to come from a load balancer, an anycast address, or client-side retry.

<service>.<namespace>.svc.cluster.local — the Service name, its namespace, the literal svc for Services, and the cluster domain.

A ClusterIP Service resolves to one virtual address; a headless Service resolves to one address per ready pod. Named ports also generate SRV records at _port._proto.<service>.<ns>.svc.cluster.local.

ndots:5 in the pod's resolv.conf makes almost every external name relative, so each lookup becomes four queries — three of them NXDOMAIN — and eight packets once A and AAAA go in parallel.

A conntrack race in the kernel can drop one of those parallel UDP packets. The client cannot detect loss, so it waits out the default 5-second timeout before retrying.

It is not a CoreDNS fault: ndots:5 multiplies the query count until a rare race becomes routine.

Trailing dot on external names — makes them absolute, one query instead of four, costs nothing, but must be applied name by name.

ndots:2 via dnsConfig — fixes the amplification cluster-wide, including in code you cannot edit.

NodeLocal DNSCache — addresses something different: it moves the CoreDNS conversation to TCP, removing the conntrack race itself rather than reducing how often it is hit, and absorbs cache hits on the node.

When clients must address individual pods: a database with a primary and replicas, Kafka brokers, anything doing its own sharding or leader election. StatefulSets add stable per-pod names.

The cost is that the RRset changes on every scale event and rollout, so it inherits every caching problem in DNS. Kubernetes serves a 30-second TTL, and any client that ignores TTLs will keep addressing pods that no longer exist.

CoreDNS forwards unresolved queries to the node's /etc/resolv.conf. If that points at a local stub — 127.0.0.53 or 127.0.0.1 — then inside the CoreDNS pod that address is the pod itself, so it forwards to itself. The loop plugin detects it at startup and exits.

Fix: point kubelet's --resolv-conf at the file with the real upstream servers, or set explicit forwarders in the Corefile.

Split-horizon — the name exists only in a private view that production's resolver does not see. Or a private hosted zone not attached to the production VPC.

Distinguish them by querying from inside the production VPC and comparing with the provider's API export of the zone. If the record is in the zone but NXDOMAINs from prod, the zone is not attached; if it is not in the zone at all, it only ever existed in the staging view.

A test from your laptop proves nothing — always ask from where a result was obtained.

They are useless to external clients — nobody outside can route to 10.0.48.20 — and they leak internal topology: subnet ranges, addressing conventions, and by implication how the network is laid out.

It is almost always a private record added to the wrong zone. Move it to a private hosted zone and add a CI check that rejects RFC 1918 addresses in any public zone.

A note on this module's transcripts. The Route 53 and Kubernetes behaviour described here is drawn from the vendor documentation linked above rather than captured live — this environment has neither an AWS account nor a cluster. The resolv.conf contents, name shapes and Corefile are the documented defaults; the latency figures in B2.1 are labelled in place as illustrative. Run B2.1 in your own cluster — the ratio between the two lines is the point, and it will reproduce.

The one thing verified live is the ndots mechanism itself: Module 06 B3's demonstration against seamless.se was captured on 20 August 2026 and is the same rule that produces the Kubernetes behaviour here.

Documentation verified directly: Kubernetes DNS for Services and Pods, NodeLocal DNSCache, Using CoreDNS for Service Discovery, Customizing DNS Service, CoreDNS kubernetes plugin, the Route 53 developer guide pages, and RFC 9499.

Question selection cross-referenced against publicly published 2026 DNS, Kubernetes and DevOps interview question sets:

• Kubernetes CoreDNS: Configuration, ndots Tuning, and Production Optimization (2026) — Coding Protocols
• Kubernetes DNS Troubleshooting: CoreDNS, ndots, and the 5-Second Timeout — kubenatives
• How Kubernetes DNS-Based Service Discovery Works — OneUptime
• Top 25 DNS Interview Questions and Answers for 2026 — nitizsharma.com — split-horizon DNS, DNS load balancing, forwarders vs conditional forwarders
• Top 30 Most Common DNS Interview Questions — Verve AI — Anycast and GeoDNS for load balancing

Answers were rewritten and deepened rather than reproduced. The published Kubernetes write-ups describe the five-second delay well but usually stop at "set ndots to 2". The parts that matter in an interview — that the amplification and the race are two different problems, that scaling CoreDNS treats neither, and that a trailing dot fixes the first for free — appear in almost none of them.