VPC Connectivity Failures

Overview

Overview

VPC connectivity failures are among the most common issues in AWS architectures. They often look like network timeouts but come from misconfigured security groups, NACLs, missing routes or broken peering.

Security groups vs NACLs

• State — Stateful: return traffic is allowed automatically — Stateless: you must allow return traffic on ephemeral ports
• Scope — Instance (ENI) level — Subnet level
• Rules — Allow only — Allow and deny
• Evaluation — All rules are evaluated — Ascending order; the first matching rule wins

Reading the symptoms

• "Connection timed out" — Packets are silently dropped: a security group or NACL blocks them, or no route exists
• "Connection refused" — The host is reachable but nothing listens on the port, so the host's own TCP stack answers with a reset
• VPC Flow Logs REJECT — A security group or NACL blocked the flow

> [!IMPORTANT] Security groups and NACLs both drop blocked packets silently; neither sends a reset or an ICMP unreachable, so a block always looks like a timeout.

VPC Flow Logs record every flow at the ENI level with an ACCEPT or REJECT action. The action does not say which control blocked it; a request logged ACCEPT whose response is logged REJECT points to a stateless NACL, because a security group always lets return traffic out.

VPC peering requirements

• Peering connection accepted — Active, not Pending Acceptance
• Routes in both route tables — Each VPC needs a route to the other VPC's CIDR via the peering connection
• Security groups updated — Allow traffic from the peered VPC's CIDR

> [!WARNING] A route in only one VPC is the classic half-configured peering: requests leave, replies have no way back.

Key concepts

Security group (stateful)

Instance-level firewall. Return traffic automatically allowed for established connections. Only allow rules. All rules evaluated simultaneously.

NACL (stateless)

Subnet-level firewall. Must explicitly allow both inbound and outbound for each traffic direction including return traffic on ephemeral ports (1024-65535). Rules evaluated in order.

VPC Flow Logs

Per-ENI log of network flows with ACCEPT/REJECT decision. Critical for determining whether traffic is blocked by security group (instance-level) or NACL (subnet-level).

VPC peering route requirement

Both VPCs need a route pointing their CIDR to the peering connection gateway (pcx-xxxx). Security groups in each VPC must allow traffic from the other VPC's CIDR.

Ephemeral ports

Client-side ports 1024-65535 used for TCP return traffic. NACLs must explicitly allow outbound 1024-65535 for return traffic to clients.

When to use it

• Diagnosing why a service in one VPC cannot reach a service in a peered VPC
• Distinguishing a security group rule issue from a NACL rule issue
• Using VPC Flow Logs to confirm whether traffic is reaching the instance
• Troubleshooting why Lambda functions cannot reach RDS in a private subnet

Interview questions

• You added a new microservice in VPC-B, but services in VPC-A (which is peered) cannot connect. What do you check?

More Troubleshooting topics

• RDS CPU at 100%
• RDS Read Replica Replication Lag
• S3 High Latency Spikes
• Slow Query Diagnosis
• Staging Passes, Prod Fails
• TLS Certificate Expiry Incidents
• Zero-Downtime Schema Changes