Service Networking Errors — DNS, Timeouts, Port Mismatch and Mesh Sidecars

Healthy pods that cannot talk: dropped versus refused packets, DNS failures and five-second delays, port mismatches, rollout errors and mesh denials, with the commands and the EKS, AKS and GKE view.

Overview

At this layer every pod says Running and Ready, and requests between them still fail. Service networking is really three separate systems stacked on top of each other, and each fails on its own: name resolution (the pod's resolver and CoreDNS), connectivity (the CNI plugin, NetworkPolicy, node firewalls and any mesh sidecar) and port mapping (the Service, its EndpointSlices and whatever programs the Service virtual IP: kube-proxy in iptables, IPVS or nftables mode, or an eBPF data plane such as Cilium).

Work outside-in and test the layers in order: resolve the name, then open a socket to the pod IP, then open a socket to the Service. Each step either passes or tells you exactly which system to fix. The error text helps too. A refusal means the packet arrived and nothing was listening. A timeout means the packet vanished, which almost always means something filtered it.

The one idea to keep: a healthy pod proves nothing about the path to it. Test from where the traffic really starts, ideally from inside the calling pod's own network namespace with kubectl debug, because NetworkPolicy, DNS settings and sidecars all depend on which pod is asking.

Symptom to first command

dial tcp ...: i/o timeout between pods — NetworkPolicy or firewall dropping packets — kubectl get networkpolicy -n payments
Could not resolve host / ENOTFOUND — Wrong name or namespace, CoreDNS down, DNS blocked by policy — kubectl exec -n prod deploy/orders -- cat /etc/resolv.conf
Lookups sometimes take exactly 5 seconds — Conntrack race on UDP DNS, amplified by ndots:5 — kubectl exec -n prod deploy/orders -- getent hosts api.example.com (timed)
Connection refused on the Service, pod answers on localhost — targetPort does not match the listener, or app bound to 127.0.0.1 — kubectl get svc orders-svc -n prod -o jsonpath='{.spec.ports}'
502s and resets only during a rollout — App exits before endpoint removal reaches every proxy — kubectl get endpointslices -n prod -l kubernetes.io/service-name=orders-svc -w
RBAC: access denied, resets, pod stuck at 1/2 — Mesh AuthorizationPolicy, strict mTLS, or sidecar start order — istioctl analyze -n prod
LoadBalancer drops part of traffic, or app sees node IPs — externalTrafficPolicy Local versus Cluster — kubectl get svc orders-lb -n prod -o jsonpath='{.spec.externalTrafficPolicy}'

What you will be able to do

Tell a timeout from a refusal and name the layer each one points at before touching a log.
Diagnose DNS failures and the five-second DNS delay, and explain ndots:5, NodeLocal DNSCache and the conntrack race.
Find a port mapping error in one command and make it impossible with named ports.
Make rolling updates invisible to clients with a preStop wait, a grace period and EndpointSlice conditions.
Debug an Istio sidecar or ambient mesh denial, and say how each problem looks on EKS, AKS and GKE.

Key concepts

Timeout versus refusal: A refusal means the packet reached a host that had nothing listening. A timeout means the packet was dropped on the way, usually by a NetworkPolicy or firewall.
NetworkPolicy isolation: A pod is isolated for ingress or egress once any policy selects it for that direction. Allowed traffic is the union of all policies, and a connection needs both the source egress and the destination ingress to allow it. Policies do nothing without a CNI that enforces them.
ndots:5 and the search list: Names with fewer than five dots are tried with each cluster search suffix first, so one external lookup can become several failed queries before the real one. A trailing dot makes a name absolute.
NodeLocal DNSCache: A DNS cache on every node that pods query without Service DNAT. It avoids the UDP conntrack race and talks to CoreDNS over TCP.
port, targetPort, containerPort: The Service port, the pod port traffic is sent to (number or name), and the container port declaration, which mostly documents the port but is required for named ports.
EndpointSlice conditions: The ready, serving and terminating flags per endpoint. Proxies send new traffic only to ready endpoints and fall back to serving-and-terminating ones only when nothing else is left. EndpointSlices replace the Endpoints API, deprecated since 1.33.
Service proxy modes: kube-proxy can program Services with iptables (the default), nftables (stable since 1.33) or IPVS (deprecated since 1.35). eBPF data planes such as Cilium and GKE Dataplane V2 replace kube-proxy entirely.
Native sidecar: An init container with restartPolicy Always, stable since 1.33. It starts before the app, stays running, and stops after it, which fixes mesh start order and Jobs that never complete.
externalTrafficPolicy: Cluster sends external traffic to any ready pod and hides the client IP. Local keeps the client IP and uses only node-local pods, with a health-check node port that tells the load balancer which nodes to use.

When to use it

Pods are Running and Ready but calls between them fail or time out.
Name lookups fail, or requests show a delay of about 5 seconds that nobody can explain.
A Service answers with connection refused while the pod answers on localhost.
Every rollout causes a short burst of 502s or connection resets.
A service mesh returns 403 RBAC errors, resets, or pods sit at 1/2 Ready.
A LoadBalancer Service marks most nodes unhealthy, or the app cannot see client IPs.

Interview questions

The orders service in prod times out calling payments in the payments namespace. DNS resolves. Walk me through it.
Some outbound API calls from our pods take just over 5 seconds, most take milliseconds. What is going on?
After every deploy we see 20 seconds of 502s from the ingress, even though readiness probes are correct. Explain and fix.
curl to orders-svc gives connection refused, but curl localhost:8080 inside the pod works. What do you check?
We enabled Istio STRICT mTLS in payments and some callers started getting connection resets while others get 403. Why the difference?
How would the same DNS failure look different on EKS, AKS and GKE, and where would you look on each?
A LoadBalancer Service with externalTrafficPolicy: Local shows 7 of 9 nodes unhealthy. Is that a problem?