Kubernetes Service Not Reachable

Overview

Overview

"Service not reachable" covers many failure modes with one user-visible symptom: connection refused or connection timeout when calling a Service. The root causes and fixes differ widely, so a layered diagnosis beats random investigation.

What must all be true

• The Service selector matches at least one pod's labels — A label typo
• That pod is Ready — The readiness probe fails
• kube-proxy is running and has programmed the rules — Rules not synchronized, e.g. after a kube-proxy restart
• NetworkPolicies, if any, allow the traffic — A missing allow rule
• The process listens on the configured port — Misconfiguration or a silent crash

Diagnosis flow

Order matters: each step assumes the one before it passed.

Commands and what each tells you

• kubectl get pods -l app=myapp -n mynamespace — Pods matching the selector and whether they are Ready
• kubectl describe pod pod-name / kubectl logs pod-name — Why a pod is not running or not Ready
• kubectl describe svc myservice -n mynamespace — The Service's selector
• kubectl get endpoints myservice -n mynamespace — The IPs and ports the Service actually routes to
• kubectl exec -it pod-name -- ss -tlnp or netstat -tlnp — Whether the process is listening on the port
• kubectl exec -it debug-pod -- nslookup myservice.mynamespace.svc.cluster.local — Whether CoreDNS resolves the Service

> [!IMPORTANT] A label typo, app=myApp vs app=myapp (case-sensitive), is the single most common cause of empty endpoint lists.

DNS vs ClusterIP

• Fails — Works — CoreDNS
• Fails — Fails — kube-proxy or the pod layer

Key concepts

Endpoints

A Kubernetes object automatically maintained by the endpoints controller, listing the IP addresses and ports of pods that match a Service's selector and are in Ready state.

kube-proxy

A per-node DaemonSet that watches Services and Endpoints and programs iptables or IPVS rules to forward traffic from ClusterIP to pod IPs.

Selector

A label matcher on a Service that determines which pods receive traffic — it must exactly match pod labels (case-sensitive) for Endpoints to be populated.

NetworkPolicy

A Kubernetes resource that restricts pod-to-pod and pod-to-external traffic by namespace, label selector, and port — when present, traffic is denied by default unless explicitly allowed.

ClusterIP

A virtual IP address assigned to a Kubernetes Service that is stable across pod restarts — traffic to this IP is forwarded to pod IPs by kube-proxy rules.

When to use it

• Diagnosing connection refused or connection timeout errors when calling a Kubernetes Service
• Explaining why kubectl get endpoints shows empty output for a Service
• Debugging selector mismatch between a Service and its target pods
• Investigating NetworkPolicy rules that may be blocking service-to-service traffic
• Troubleshooting CoreDNS resolution failures for Kubernetes service names

Interview questions

• Walk through the complete diagnostic sequence for a Kubernetes Service returning connection refused.
• How does a NetworkPolicy affect Service routing and what is the default behavior when policies exist?
• What are the most common selector mismatch patterns that cause empty Kubernetes Service endpoints?

More Troubleshooting topics

• ECS Task Failures and Restarts
• ELB 5xx Errors
• Flaky CI/CD Pipeline Failures
• HPA Not Scaling
• IAM Permission Denied Errors
• JVM Garbage Collection Pauses
• Lambda Cold Start Latency
• Log Volume Explosion and Cost Runaway
• Memory Leak Diagnosis
• Metrics Gaps in Dashboards
• Missing Traces in Distributed Tracing
• Multi-Region High Latency