ECS Task Failures and Restarts

Overview

Overview

ECS task failures fall into a few distinct categories, each with its own signal and fix. The first step is always the stopped task reason in the ECS console or CLI: ECS keeps it for a while after the task stops, and it narrows the problem immediately.

Failure categories

• Image pull failure — CannotPullContainerError with the ECR error embedded — ECR repository or image tag does not exist; task execution role lacks ecr:GetAuthorizationToken or ecr:BatchGetImage; no route to ECR (NAT Gateway or ECR VPC endpoints) — Fix the tag, role or route
• OOMKilled — Exit code 137, Linux sent SIGKILL — Container exceeded its memory limit — Increase the task memory limit or find the leak
• Health check failure — Task runs but is marked unhealthy and replaced — Slow start with no startPeriod, or a real app fault — Generous startPeriod (30–120 seconds) for slow starters such as Spring Boot
• Essential container exit — Whole task stops — An essential sidecar (log router, Envoy proxy, secrets fetcher) or the app exits — Mark as essential only what really must stop the task

Reading memory in Container Insights

• Climbs steadily — A leak
• Spikes on specific requests — An unbounded operation, e.g., loading a large file into memory

Where the health check lives

• Task definition — ECS kills the task after the configured number of failures
• Load balancer target group — ALB deregisters the target, new tasks replace it, and you loop if the cause is not fixed

Sources to correlate

• ECS console stopped reason — The category
• CloudWatch Logs for the task — The last output before exit
• CloudTrail — IAM permission failures

> [!TIP] Turn on the Deployment Circuit Breaker: it stops ECS from endlessly replacing failing tasks and surfaces the problem instead of masking it behind restart loops.

Key concepts

Stopped Task Reason

A string ECS preserves after a task stops, describing why. Available in the ECS console under stopped tasks or via aws ecs describe-tasks CLI. Expires after several hours.

OOMKilled (Exit Code 137)

The Linux kernel killed the container process because it exceeded the container memory limit. Exit code 137 = 128 + SIGKILL signal number (9).

startPeriod

A health check parameter that gives a container grace time to start before health check failures count toward the unhealthy threshold. Essential for slow-starting applications.

Essential Container

A container marked essential:true in the task definition. If any essential container exits for any reason, ECS stops the entire task including all other containers.

Deployment Circuit Breaker

An ECS service setting that detects when a deployment is repeatedly failing and rolls back to the previous stable version automatically, preventing endless restart loops.

When to use it

• Diagnosing ECS services that show tasks constantly starting and stopping
• Explaining ECS health check semantics and essential container behavior in interviews
• Designing fault-tolerant ECS services that surface failures rather than hiding them in restart loops

Interview questions

• An ECS service is stuck in a restart loop. Tasks start, run for 30 seconds, then stop, and new tasks immediately replace them. How do you diagnose this systematically?
• How does ECS health checking interact with ALB health checking, and how do misconfigured thresholds create phantom restart loops?

More Troubleshooting topics

• Database Deadlock Diagnosis
• Database Migrations in Production
• Deployment Rollback Strategies
• Disk IOPS Saturation
• DNS Failures
• EC2 Instance Unresponsive
• ELB 5xx Errors
• Flaky CI/CD Pipeline Failures
• HPA Not Scaling
• IAM Permission Denied Errors
• JVM Garbage Collection Pauses
• Kubernetes Service Not Reachable