ECS deployment stuck? The circuit breaker and automatic rollbacks

A deployment that sits at “in progress” for an hour is one of the most common ECS questions. Usually the new tasks keep failing and ECS keeps replacing them. By default it never gives up. The deployment circuit breaker is the setting that makes it give up and roll back.

A short recap

During a rolling deployment ECS starts tasks of the new revision and stops tasks of the old one. Two settings set the pace: minimum healthy percent is the floor of healthy tasks that must stay up, and maximum percent is the ceiling of tasks that may run at once. Part 1 explains both, with a simulator.

Why deployments get stuck

A deployment finishes when every desired task runs the new revision and is healthy. It gets stuck when that never happens:

1. New tasks never start. The image can’t be pulled, a secret can’t be read, the execution role is missing a permission, or the container exits right away.
2. New tasks start but fail health checks. The app listens on the wrong port, the health path returns a 401 or a redirect, or startup takes longer than the checks allow. See the health checks post.
3. There is no room. On EC2, no instance has enough free CPU, memory or ports for the new tasks.
4. The settings allow no step. A floor equal to the ceiling, like 100% / 100%, means ECS can neither stop nor start a task.

The service’s events tell you which one it is. Messages like was unable to place a task because no container instance met all of its requirements point to capacity. Tasks that keep stopping point to the first two causes, and the stopped reason of a failed task says which: CannotPullContainerError, ResourceInitializationError or Task failed ELB health checks. Bayplan’s rollout view shows every task a deployment started and stopped on one timeline, so a replace loop stands out at a glance.

The replace loop

With a broken revision, ECS starts new tasks, they fail, ECS stops them and starts replacements. With the default 100% / 200% the old tasks keep serving traffic, so nothing is down, but the deployment never ends. ECS slows its retries down after repeated failures, but it doesn’t give up.

Press Play below. The new revision is broken, and the circuit breaker is off.

Try 50% / 100% as well. ECS stops half the old tasks to make room, and the service runs at half capacity for as long as the loop lasts.

How the circuit breaker counts failures

The deployment circuit breaker watches a rolling deployment in two stages:

• Stage 1 counts new tasks that fail to reach RUNNING: image pull errors, missing secrets, containers that exit at once.
• Stage 2 starts once tasks are running. It counts tasks that fail their load balancer, Cloud Map or container health checks.

When the failure count reaches the threshold, the deployment is marked FAILED. The threshold is half the desired count, rounded up, but never less than 3 and never more than 200:

For small services that means 3 failures, which takes only a minute or two when tasks crash on start. A service with 100 tasks needs 50.

Rollback

On its own, the circuit breaker only marks the deployment as failed and stops launching new tasks. With rollback on, ECS also starts a new deployment of the last revision that completed successfully. If no deployment ever completed, there is nothing to roll back to and the deployment stays stalled.

Turn on the circuit breaker in the simulator above and press Play again. After the threshold is reached, ECS stops the failing tasks and goes back to revision 7.

Turn both on per service:

aws ecs update-service --cluster production --service web \
  --deployment-configuration "deploymentCircuitBreaker={enable=true,rollback=true}"

In CloudFormation it’s DeploymentCircuitBreaker inside the service’s DeploymentConfiguration, and in Terraform the deployment_circuit_breaker block. aws ecs list-service-deployments and describe-service-deployments show the failure count, the threshold and whether the circuit breaker triggered.

Alarms for what the circuit breaker can’t see

The circuit breaker only knows whether tasks start and pass health checks. A revision that starts fine but returns errors to half of its users passes both. For that, attach CloudWatch alarms to the deployment:

"deploymentConfiguration": {
  "deploymentCircuitBreaker": { "enable": true, "rollback": true },
  "alarms": {
    "alarmNames": ["web-5xx-rate", "web-p99-latency"],
    "enable": true,
    "rollback": true
  }
}

If an alarm goes into ALARM during the deployment, ECS fails it and rolls back the same way. Whichever detects a failure first wins.

Recommended settings

• Turn on the circuit breaker with rollback for every rolling deployment. It costs nothing, and a rollback is almost always better than an endless loop.
• Set a health check grace period that covers your real startup time, so slow starts don’t count as failures.
• Add alarms on error rate or latency for services where a bad release hurts.
• Remember that a brand-new service has nothing to roll back to. If its first deployment fails, it stalls instead.