Internal DNS Cache Poisoning That Redirected API Traffic

Overview

An internal DNS resolver left without DNSSEC validation cached a forged record for payments-api.internal, and every service sent payments traffic to a dead IP for 2 hours.

What happened

The shared resolver was a self-hosted BIND instance on EC2. DNSSEC validation was turned off during an upgrade and never turned back on.

How it spread

Every microservice used the same resolver, so one bad cache entry redirected all of them. The failure was quiet: plain connection refused, not an obvious DNS error.

Impact

• Forged record TTL — 7,200 seconds
• Time the record persisted after the rogue resolver was blocked — 2 hours
• Time before drift detection caught DNSSEC being off — 6 weeks

The fix

• Resolver cache — Holding the forged record — Flushed with rndc flush
• payments-api.internal A record — Stale, decommissioned IP — Repointed to the correct IP
• DNSSEC validation — Disabled since the upgrade — Flagged by drift detection

> [!WARNING] Blocking the source of a bad record does not remove it. A cached answer lives until its TTL runs out or you flush the cache.

Key concepts

DNS cache poisoning

Inserting malicious records into a DNS resolver's cache, causing clients to resolve legitimate hostnames to attacker-controlled IPs.

DNSSEC

DNS Security Extensions — cryptographic signatures on DNS records that allow resolvers to verify that records are authentic and have not been tampered with.

DNS TTL

Time to live — how long a DNS record is cached. Long TTLs amplify the impact of poisoning; short TTLs increase resolver load but limit exposure duration.

Service discovery with health checks

Using a service registry (Consul, AWS Cloud Map) that actively health-checks instances and removes unhealthy IPs from DNS responses rather than relying on static TTLs.

When to use it

• Discussing DNS security (DNSSEC, resolver hardening)
• Explaining why internal DNS infrastructure requires the same security scrutiny as external DNS
• Designing service discovery with health checking and short TTLs

Interview questions

• How do you design internal service-to-service DNS to minimize the impact of DNS failures or poisoning?

More War Stories topics

• BGP Route Leak That Rerouted Traffic Through an Unexpected AS
• Blue-Green Deployment That Resulted in Split-Brain Sessions
• Cache Stampede on a Flash-Sale Product Page
• Connection Pool Exhaustion That Brought Down 12 Microservices
• DynamoDB Hot Partition That Throttled a Flash Sale
• Failover to a Lagging Replica Lost 4 Minutes of Orders
• Malicious npm Package Exfiltrated Environment Variables at Build Time
• N+1 Query Pattern That Collapsed Under Load
• NAT Gateway Port Exhaustion Under Bursty Lambda Traffic
• Node.js Memory Leak That Required Daily Restarts
• Public S3 Bucket Exposed Customer PII for 8 Months
• Silent Data Corruption from an Off-By-One Migration