DNS Failures

Overview

Overview

DNS failures show up as connection errors that look like network failures but are really name resolution failures. They are among the most common causes of production incidents because they stay invisible until they cause a cascading failure.

The two failure modes

• NXDOMAIN — The name definitively does not exist; the authoritative server confirmed it — Record not created yet, deleted, or negatively cached
• SERVFAIL — The resolver hit an error — Nameserver unreachable, DNSSEC validation failed, or the resolver itself is broken

> [!IMPORTANT] NXDOMAIN and SERVFAIL need very different fixes, so read the response code first.

Caching failure mechanisms

• Propagation delay — After a record change, resolvers keep returning the old value until their cached TTL expires; some clients see the new record, some the old — At most the TTL of the old record
• Negative caching — A lookup for a missing record caches the NXDOMAIN, so clients cannot resolve the record even after it is created — The SOA minimum TTL (typically 1-5 minutes)

CoreDNS in Kubernetes

If CoreDNS pods crash or are overloaded, all pod-to-pod communication by service name breaks at once. It shows up as SERVFAIL responses for .svc.cluster.local names.

> [!WARNING] Intermittent failures during a propagation window look like a network issue; compare answers from several resolvers before blaming the network.

Key concepts

NXDOMAIN

Authoritative denial — the name definitively does not exist. Cached for the SOA minimum TTL. Fix: the record is missing or was deleted.

SERVFAIL

Resolution failure — the resolver could not complete the query. Fix: the upstream nameserver is unreachable, DNSSEC is failing, or the resolver itself is broken.

Negative caching TTL

NXDOMAIN responses are cached. Duration is the SOA minimum TTL (often 300-3600 seconds). New records are not visible until this cache expires across all resolvers.

CoreDNS

Kubernetes cluster DNS server. All service name resolution goes through CoreDNS pods. Pod failure → all .svc.cluster.local resolution fails. Scale CoreDNS replicas for production.

ndots

Kubernetes /etc/resolv.conf sets ndots:5. Names with fewer than 5 dots are treated as relative, triggering multiple search-domain queries before trying absolute. Can cause excessive CoreDNS load.

When to use it

• Diagnosing intermittent "could not resolve host" errors in production services
• Explaining why a DNS record change is not immediately visible to all users
• Debugging Kubernetes service discovery failures where pods cannot reach each other
• Tracing a split-horizon DNS issue where internal and external clients see different records

Interview questions

• After updating a DNS record, some services are working and some are not. Why and how do you fix it?

More Troubleshooting topics

• CPU Runaway Process Investigation
• CrashLoopBackOff
• Database Deadlock Diagnosis
• Database Migrations in Production
• Deployment Rollback Strategies
• Disk IOPS Saturation
• EC2 Instance Unresponsive
• ECS Task Failures and Restarts
• ELB 5xx Errors
• Flaky CI/CD Pipeline Failures
• HPA Not Scaling
• IAM Permission Denied Errors