How ECS rolling deployments work: minimum healthy percent and maximum percent

When you point an ECS service at a new task definition revision, ECS doesn’t swap all tasks at once. It starts tasks of the new revision and stops tasks of the old one in small steps, until every task runs the new one. Two numbers decide how big those steps are: minimum healthy percent and maximum percent.

Most services run with the defaults and never think about them. That works until a deployment takes twice as long as it should, needs more capacity than the cluster has, or doesn’t move at all.

What happens when you deploy

A deployment starts whenever the service’s configuration changes: a new task definition revision, aws ecs update-service --force-new-deployment, or a change to the network or capacity settings. ECS then repeats a simple loop:

1. Start tasks of the new revision, as many as the upper limit allows.
2. Wait until they are healthy.
3. Stop tasks of the old revision, as many as the lower limit allows.
4. Repeat until all desired tasks run the new revision.

The example in this post is a service called web with 4 desired tasks, moving from revision 7 to revision 8.

The two settings

Both live in the service’s deploymentConfiguration and are percentages of the desired count:

"deploymentConfiguration": {
  "minimumHealthyPercent": 100,
  "maximumPercent": 200
}

Minimum healthy percent is the floor. It’s the number of tasks that must stay running and healthy during the deployment, rounded up. With 4 tasks and 50%, ECS may stop 2 old tasks before new ones are ready. With 2 tasks and 75%, the floor is 1.5, rounded up to 2, so ECS may stop none.

Maximum percent is the ceiling. It’s the number of tasks that may run or be pending at once, old and new together, rounded down. With 4 tasks and 200%, ECS can start 4 new tasks next to the 4 old ones. With 3 tasks and 125%, the ceiling is 3.75, rounded down to 3, so ECS can’t start a single extra task.

For replica services the defaults are 100% and 200%. Daemon services default to a 0% minimum, because there is one task per instance and no room for a second one.

“Healthy” depends on what the task has:

• With a container health check on an essential container, the task is healthy once that check passes.
• Behind a load balancer, it also has to pass the target group’s health check.
• With neither, ECS treats a task as healthy 40 seconds after it reaches RUNNING.

The health checks post covers the checks and the grace period in detail.

Try it

Move the sliders and press Play. The floor and ceiling update as you go, and the chart below the tasks shows that the total never goes above the ceiling and the healthy tasks never drop below the floor.

The simulator is simplified. Real deployments don’t move in lockstep, and start and drain times vary from task to task. The decisions ECS makes at each step are the same, though.

Common settings compared

Minimum / maximum	What happens	Good for
100% / 200% (default)	All new tasks start next to the old ones, old ones stop once new ones are healthy	Most web services
100% / 150%	Half the tasks are replaced at a time, full capacity throughout	Large services where doubling is too much
50% / 100%	Half the old tasks stop first, no extra tasks ever run	Clusters without spare capacity
0% / 100%	All old tasks stop, then new ones start, with a short outage	Dev environments, queue workers that can pause
100% / 100%	Nothing can stop and nothing can start	Never: the deployment is stuck

Watch the rounding with small services. A single task at 100% / 150% has a ceiling of 1.5, rounded down to 1, so it’s stuck just like 100% / 100%. A service with one task needs a maximum of 200%, or a minimum of 0% and the brief outage that comes with it.

Capacity

On Fargate the extra tasks just cost money for a few minutes. With the default settings you briefly pay for twice the tasks.

On EC2, the cluster needs room for the extra tasks. If no instance has enough free CPU, memory or ports, the new tasks stay pending and the deployment waits for capacity. Two things make this worse:

• Static host ports. In bridge mode with a fixed host port, only one task of the service fits on each instance. The new task can’t start until an old one is gone. Dynamic host ports or awsvpc mode avoid this.
• A full cluster. With a capacity provider and managed scaling, ECS adds instances for the pending tasks, but that takes minutes. Without one, the tasks wait until something else frees up room.

If spare capacity is the problem, 50% / 100% deploys without any. The Fargate vs EC2 post has more on how capacity planning differs between the two.

The same two settings apply when you drain an EC2 instance. ECS starts replacement tasks elsewhere within the ceiling and stops the tasks on the draining instance within the floor.

Making deployments faster

Most of a deployment’s time is spent waiting, not deciding. The main levers:

• Deregistration delay. Before an old task stops, the load balancer drains its connections, for up to 300 seconds by default. Set it close to your longest request.
• Health check timing. A target group with the default 5 healthy checks every 30 seconds needs two and a half minutes per new task. The health checks post shows faster settings.
• A higher maximum. With 4 tasks, 200% replaces everything in one round. 125% takes four rounds, each with its own wait for health checks and draining.
• Early success criteria. A newer deploymentConfiguration option marks a deployment complete once a set percentage of new tasks is healthy, instead of waiting for all of them.

Other deployment strategies

Rolling updates are the default and the only strategy that uses these two settings. ECS also has built-in blue/green, linear and canary strategies. They start a full set of new tasks and shift load balancer traffic over all at once, in equal steps, or a small share first. They need a load balancer or Service Connect and roughly double the capacity during the shift. They’re worth a look when you need to switch back instantly. For most services, a rolling update with good health checks is enough.

Recommended settings

• Keep 100% / 200% for web services unless capacity is tight.
• Use 50% / 100% on EC2 clusters without headroom, and accept that the service runs at half capacity during deployments.
• Give single-task services a 200% maximum, or accept a short outage with 0% / 100%.
• Tune the deregistration delay and health checks before you change these settings for speed.

A deployment can also fail: the new revision crashes, fails its health checks or never starts. By default ECS keeps trying, sometimes for hours. Part 2 of this series covers why deployments get stuck and how the deployment circuit breaker rolls them back.