Home / server / Questions / 831137
Accepted
c4urself
Asked: 2017-02-08 08:57:35 +0800 CST

How to make AWS ASG do nothing when instance count is between min and max

  • 772

We're using Terraform to launch ASGs for most of our AWS EC2 instances. The problem is that once in awhile we want to do some extra work before terminating an instance; for example: decommissioning a node from a cluster before the EC2 instance it was running on is terminated. If we were just to lower min == max (our default) then an instance gets terminated and we can't run a graceful decommission.

Instead what I've tried is lowering min to the new desired value (example: 6) and keeping the max at the old value (example: 10), what happens in this case is that the desired value stays at 10 (the max) and terminating the EC2 instance causes a new one to be launched by the ASG. NOTE: we are not setting the Terraform desired_capacity setting at all.

If I set desired_capacity manually, I risk the ASG terminating a node that has not been gracefully decommissioned so I don't think that's an option for me.

What I'd ideally like is for the ASG to do nothing when the current EC2 Instance count for that ASG is between min and max and let me manually terminate instances. Obviously if the count goes below min I'd still like the ASG to launch a new EC2 Instance.

Is there any way to achieve this?

amazon-ec2 amazon-web-services terraform

2 Answers

  1. There are two possible ways to achieve what you want:

    Option 1: Suspend Auto Scaling Processes

    You can put the Auto Scaling group's processing "on hold" while you are making your adjustments.

    For example, you could try:

    1. Pause Auto Scaling processing: aws autoscaling suspend-processes --auto-scaling-group-name MyGroup
    2. Decommission and terminate your EC2 instances,
    3. Adjust the desired count manually,
    4. Resume Auto Scaling processing: aws autoscaling resume-processes --auto-scaling-group-name MyGroup

    http://docs.aws.amazon.com/cli/latest/reference/autoscaling/suspend-processes.html

    Option 2: Use Auto Scaling Lifecycle Hooks

    Using Lifecycle Hooks, your launched and/or terminating EC2 instances are given an opportunity to do an initial or pre-termination processing. For example, you can have the hook notify the terminating instnce that it is about to be terminated, and it can decommission itself from your cluster.

    http://docs.aws.amazon.com/autoscaling/latest/userguide/lifecycle-hooks.html

    • 3
  2. Best Answer

    Supposed solutions:

    Option 1: Your ASG should be created with instance protection ON - Terraform docs

    In this case, we could have next sequence of operations for instance decommission:

    1. Perform decommission of particular instance(s)(remove data from it(them))
    2. Reduce count of ASG's size by desirable value
    3. Apply Terraform state
    4. Remove protection flag from decommissioned instances: aws autoscaling set-instance-protection --instance-ids <instances_ids> --auto-scaling-group-name <asg_name> --no-protected-from-scale-in

    Option 2: Your ASG was not created with instance protection.

    In this case, we could have next sequence of operations for instance decommission:

    1. Add protection flag on all instances of ASG: aws autoscaling set-instance-protection --instance-ids <instances_ids> --auto-scaling-group-name <asg_name> --protected-from-scale-in
    2. Perform decommission of particular instance(s)(remove data from it(them))
    3. Reduce count of ASG's size by desirable value
    4. Apply Terraform state
    5. Remove protection flag from decommissioned instances: aws autoscaling set-instance-protection --instance-ids <instances_ids> --auto-scaling- group-name <asg_name> --no-protected-from-scale-in
    6. (Optionally) Wait until ASG shrinks to desirable size and remove protection flag from other instances
    • 3