EC2 Instance Unresponsive

Overview

Overview

An EC2 instance that stops answering SSH can fail at three layers: network (security group, NACL, routing), OS (kernel panic, OOM kill of sshd, disk full) or hardware (the underlying host). Start with the layers you can reach without SSH: the AWS console and the EC2 Serial Console.

Tools you can use without SSH

• EC2 System Log — Actions > Monitor and troubleshoot > Get system log — Kernel boot messages and early userspace output: kernel panics, disk errors, fsck failures, even when the instance is unreachable
• EC2 Serial Console — Must be enabled for the account — An interactive terminal with no network dependency; works even if the network interface is broken
• System Status Check — Console, Status checks tab — The underlying hardware: power, network, hardware
• Instance Status Check — Console, Status checks tab — The OS: network config, kernel, sshd

> [!IMPORTANT] A failing system check usually needs a stop/start, not a reboot: only stop/start migrates the instance to a new host.

Common causes

• Security group change removed the port 22 rule — Network — CloudTrail, security group rules
• Disk full, so sshd cannot write /var/run/sshd.pid — OS — System log: "No space left on device"
• OOM kill terminated sshd — OS — System log: "Out of memory: Kill process"
• Kernel panic from a kernel update — OS — System log: "Kernel panic"
• CPU credit exhaustion on T-class instances — Compute — Instance throttled, sshd handshakes time out

> [!TIP] Enable the Serial Console at account level before you need it; you cannot turn it on for an instance you already cannot reach.

Key concepts

EC2 System Log

Kernel and early boot output captured by the hypervisor. Accessible in the AWS console without network access. Shows kernel panics, fsck, and disk errors.

EC2 Serial Console

Interactive terminal to the instance bypassing the network. Must be enabled at account level. Login requires OS user password (SSH key auth not available here).

Stop vs reboot

Reboot restarts on the same host. Stop+Start migrates to a new host — required for System Status Check failures that indicate underlying hardware issues.

CPU credit exhaustion (T-class)

T2/T3 instances have a baseline CPU and a burst credit pool. When credits run out, CPU is hard-throttled to baseline. Sshd handshakes time out under heavy throttling.

Rescue instance

Detach the root EBS volume from the broken instance, attach it to a working instance, mount it, fix the issue (disk full, corrupted fstab), re-attach, and restart.

When to use it

• SSH to an EC2 instance fails with "Connection timed out" or "Connection refused"
• An EC2 instance passed both status checks but applications are not responding
• Recovering a Linux instance after a failed kernel upgrade
• Instance entered an unreachable state after a disk was full

Interview questions

• An EC2 instance that was running fine is now unreachable via SSH. Walk through your complete diagnosis.

More Troubleshooting topics

• CrashLoopBackOff
• Database Deadlock Diagnosis
• Database Migrations in Production
• Deployment Rollback Strategies
• Disk IOPS Saturation
• DNS Failures
• ECS Task Failures and Restarts
• ELB 5xx Errors
• Flaky CI/CD Pipeline Failures
• HPA Not Scaling
• IAM Permission Denied Errors
• JVM Garbage Collection Pauses