Service Discovery

Overview

Overview Service discovery is the mechanism by which services in a distributed system locate each other's network endpoints without hardcoded IPs or configuration files. In dynamic environments where containers, pods, and instances start, stop, and move frequently, static service endpoints are impractical. Service discovery solves this by maintaining a real-time registry of healthy service endpoints and providing a consistent interface for clients to query it.

There are two primary discovery patterns. In client-side discovery, the client queries a service registry (such as Consul or Eureka) directly, receives a list of healthy endpoints, and applies its own load balancing logic to select an instance. Netflix Ribbon used this pattern extensively. In server-side discovery, the client sends requests to a fixed intermediary (a load balancer or service mesh proxy), which queries the registry and routes the request. Kubernetes services and AWS ECS use server-side discovery, where kube-proxy or the cloud load balancer handles the routing transparently.

Kubernetes implements service discovery through its built-in DNS system, powered by CoreDNS. Every Service object gets a DNS name in the format service-name.namespace.svc.cluster.local. A ClusterIP service returns a single virtual IP; the traffic is intercepted by kube-proxy (or eBPF-based solutions like Cilium) and load-balanced to healthy pods. Headless services (ClusterIP: None) return the individual pod IPs directly via DNS instead of a virtual IP, enabling client-side load balancing and direct pod addressing.

SRV records extend DNS-based discovery to include port and protocol information. A SRV record encodes the hostname, port, priority, and weight for a service instance, allowing clients to discover not just where a service runs but what port it listens on. etcd, Consul, and gRPC name resolvers all support SRV-based discovery.

Consul goes beyond DNS to provide health-checked service registration, service mesh (with Envoy sidecar proxies), key-value storage, and ACL-based access control. Its DNS interface integrates service discovery into standard DNS resolution while its HTTP API enables programmatic registry access.

Key concepts

Client-Side vs Server-Side Discovery

In client-side discovery, the client queries the service registry, gets a list of endpoints, and picks one using its own load balancing policy. This gives the client control but requires each client to implement discovery logic. In server-side discovery, the client sends requests to a stable endpoint (load balancer, service mesh proxy); the proxy queries the registry and forwards the request. Kubernetes services use server-side discovery — the client only knows the Service ClusterIP, not individual pod IPs.

CoreDNS in Kubernetes

CoreDNS is the cluster DNS server in Kubernetes, running as a Deployment in the kube-system namespace. It serves the cluster.local domain. Every Service gets an A record (service.namespace.svc.cluster.local) resolving to the ClusterIP. Pods get A records (pod-ip.namespace.pod.cluster.local). CoreDNS is configured via a Corefile, which specifies plugins for forwarding, caching, health checking, and custom zone handling. Cluster DNS is configured in /etc/resolv.conf of each pod via the kubelet.

Headless Service

A Kubernetes Service with spec.clusterIP: None. Instead of creating a virtual ClusterIP, Kubernetes DNS returns the individual pod IPs directly for the service's DNS name. This enables client-side load balancing, direct pod addressing, and StatefulSet pod discovery (each pod gets a stable DNS name like pod-0.service.namespace.svc.cluster.local). Headless services are used by stateful applications (Cassandra, Kafka, MongoDB) where clients need to address specific instances.

SRV Record

A DNS SRV record encodes service location information: priority, weight, port, and target hostname. Format: _service._proto.name. TTL class SRV priority weight port target. Example: _http._tcp.myservice.example.com SRV 10 20 8080 host1.example.com. Clients query SRV records to discover both hostname and port dynamically. gRPC uses SRV records for name resolution; Kubernetes creates SRV records for named ports in Services (e.g., _http._tcp.service.namespace.svc.cluster.local).

Consul Service Mesh

Consul operates as a service registry (agents register services with health checks), DNS resolver (consul.service.consul format), and service mesh (Consul Connect injects Envoy sidecars that handle mTLS, traffic policies, and observability). The Consul catalog stores service name, address, port, tags, and health status. Services are registered via API, CLI, or configuration files. Consul supports multiple datacenters with WAN gossip federation, enabling multi-region service discovery.

Service Mesh vs DNS Discovery

DNS discovery returns endpoints for a service name, leaving load balancing and health checking to the client or a proxy. A service mesh (Istio, Linkerd, Consul Connect) injects a sidecar proxy alongside each service instance that intercepts all traffic, enforces mTLS, applies traffic policies (circuit breaking, retries, rate limiting), and emits telemetry. The mesh control plane (Pilot/Istiod) distributes endpoint information to sidecars using xDS APIs, not DNS. DNS discovery is simpler; service meshes add security and observability at the cost of operational complexity.

When to use it

• Connecting microservices in Kubernetes without hardcoding service IPs or ports.
• Implementing zero-downtime deployments where service endpoints change as pods roll over.
• Building a service mesh where policy, mTLS, and observability are applied at the discovery layer.
• Enabling stateful service discovery where clients need to talk to specific instances (e.g., Kafka brokers, Cassandra nodes) rather than any healthy replica.
• Cross-cluster or multi-cloud service discovery where services span multiple Kubernetes clusters or cloud providers.
• Replacing static configuration files with dynamic service registries in legacy application migrations.

Interview questions

• Explain how Kubernetes service discovery works end-to-end, from a pod making a DNS request to receiving a response from another service.
• What is the difference between client-side and server-side service discovery, and what are the tradeoffs of each pattern?
• How would you set up Consul for service discovery in a multi-datacenter environment? What is the WAN gossip pool?

More Networking topics

• Packet Capture & Analysis
• QUIC Protocol & Transport
• Rate Limiting Algorithms
• Route 53 & DNS Routing Policies
• Router vs Switch
• SD-WAN
• Service Mesh
• Split-Horizon DNS
• Sticky Sessions
• TCP Congestion Control
• TCP vs UDP
• TCP/IP Stack & Handshake