CoreDNS Failures in Kubernetes

Diagnose and fix CoreDNS failures that cause cascading service-to-service connectivity failures in Kubernetes.

Overview

Overview

CoreDNS is the cluster DNS server: it resolves service names (myservice.namespace.svc.cluster.local), pod DNS names and external hostnames for every pod. Since best practice is to call services by name, a failing or overloaded CoreDNS breaks nearly all internal communication and looks like a full outage, even while the pods and control plane are healthy.

Where it lives

  • CoreDNS — A Deployment in kube-system, typically two replicas for redundancy
  • kube-dns Service — Exposes CoreDNS; keeps the historical name kube-dns whichever DNS server you run
  • Pod resolv.conf — The kubelet's clusterDNS setting injects the kube-dns ClusterIP as every pod's DNS server

Diagnosis flow

  • kubectl exec -it pod -- nslookup kubernetes.default.svc.cluster.local — Fails: CoreDNS is the problem. Works but one service fails: that name is not in DNS
  • Same service by ClusterIP vs by name — ClusterIP works, name fails: a DNS problem

Cause 1: CPU throttling

> [!WARNING] CPU throttling is the most common cause of CoreDNS failures: many cluster configurations give CoreDNS a 100m (0.1 CPU) limit.

Cause 2: ndots:5 amplification

The default pod /etc/resolv.conf has ndots:5 and a list of search domains. A name with fewer than 5 dots is tried with each search suffix before being tried as-is, so one lookup of an unqualified name can become 3-6 queries to CoreDNS.

  • 1 — myservice.mynamespace.svc.cluster.local
  • 2 — myservice.svc.cluster.local
  • 3 — myservice.cluster.local
  • 4 — myservice as-is

The resolver stops at the first answer, so a name in the pod's own namespace resolves on try 1; names in other namespaces and external hostnames walk further down the list. At hundreds of pods making hundreds of lookups per second, the amplification is significant.

The fix: NodeLocal DNSCache

> [!TIP] NodeLocal DNSCache cuts CoreDNS query volume by 70-90% in practice, ending throttling and shrinking the blast radius of a CoreDNS failure.

Key concepts

CoreDNS
The cluster DNS server deployed as a Deployment in kube-system — resolves Kubernetes service names, pod DNS names, and external hostnames for all cluster pods.
ndots:5
The default Kubernetes resolver configuration that causes short hostnames to be tried with 5 search domain suffixes before being resolved as absolute names, amplifying DNS query volume.
NodeLocal DNSCache
A DaemonSet that runs a DNS cache on every node, serving cached responses locally and reducing CoreDNS query volume by 70-90% in high-scale clusters.
Search Domains
The DNS suffix list in /etc/resolv.conf (namespace.svc.cluster.local, svc.cluster.local, cluster.local, etc.) that the resolver appends to unqualified hostnames before querying DNS.
FQDN
Fully Qualified Domain Name — a DNS name ending with a dot (myservice.mynamespace.svc.cluster.local.) that bypasses ndots search domain expansion and sends one DNS query instead of many.

When to use it

  • Diagnosing cluster-wide service connectivity failures that resolve when using ClusterIP instead of service names
  • Investigating intermittent DNS timeouts that correlate with high pod counts or frequent deployments
  • Explaining the ndots:5 amplification effect and how to mitigate it
  • Configuring NodeLocal DNSCache to reduce CoreDNS load at scale
  • Scaling CoreDNS replicas and adjusting CPU limits to handle peak cluster DNS load

Interview questions

  • How do you diagnose whether CoreDNS is causing a service connectivity failure?
  • What is the ndots:5 amplification problem and how do you fix it?
  • How does NodeLocal DNSCache work and what are the operational steps to deploy it?

Sign up free to read the full topic

More Troubleshooting topics

Start free on Cariara · Coding problems · System design studio