Home / server / Questions / 356178
Accepted
Jeevan Dongre
Asked: 2012-02-03 05:50:59 +0800 CST

Instance has failed at least the Unhealthy Threshold number of health checks consecutively

  • 772

I am trying to autoscale my present instance, I am running a medium instance right now and auto scale with small instance. I used command line tool to config the settings, this is the configurations I have used to scale and I am running minimum of one instance apart from my regular instance, that means one more instance and I have attached to the load balancer.

s-create-auto-scaling-group groupname --launch-configuration launchconfig --availability-zones ap-southeast-1a --min-size 1 --max-size 5 --load-balancers prod

But when I have checked the load balancer it says "Out of Service" reason "Instance has failed at least the Unhealthy Threshold number of health checks consecutively". How can I solve this, using its public DNS, I am not able to get any response from the instance and also not able to ssh to it since the key value pair is not attached to the newly created instance.

what is the problem.How do I solve this.

Please help me out its little urgent, since I am struck up at this point from almost 2 days.

as-describe-launch-configs --show-long --headers



testLC,ami-e8c4bdba,t1.micro,(nil),(nil),(nil),(nil),default,2012-02-03T07:14:54.461Z,true,arn:aws:autoscaling:ap-southeast-1:346266270015:launchConfiguration:175a16db-1f6a-4514-9233-ac7cb34bca90:launchConfigurationName/testLC

as-describe-auto-scaling-groups --show-long --headers

testASG,testLC,ap-southeast-1a,2012-02-03T07:19:10.706Z,prod,EC2,1,5,1,300,0,(nil),(nil),arn:aws:autoscaling:ap-southeast-1:346266270015:autoScalingGroup:c4b584d0-bac4-4507-b972-4fc2b1bc53ac:autoScalingGroupName/testASG,(nil)

as-describe-auto-scaling-instances
i-43796716  testASG  ap-southeast-1a  InService  HEALTHY  testLC

elb-describe-lbs --headers --show-long

prod,prod-11719395.ap-southeast-1.elb.amazonaws.com,prod-11719395.ap-southeast-1.elb.amazonaws.com,Z1WI8VXHPB1R38,"{interval=120,target=HTTP:80/user/sign_in/,timeout=30,healthy-threshold=5,unhealthy-threshold=3}",ap-southeast-1a,(nil),(nil),"i-495dda1c, i-43796716","{protocol=HTTP,lb-port=80,instance-protocol=HTTP,instance-port=80,policies=AWSConsolePolicy-1}",(nil),"{policy-name=AWSConsolePolicy-1,expiration-period=180}","{owner-alias=amazon-elb,group-name=amazon-elb-sg}",(nil),2012-02-01T10:36:08.810Z

elb-describe-instance-health loadbalancername --headers --show-long

INSTANCE_ID,i-495dda1c,InService,N/A,N/A
INSTANCE_ID,i-43796716,OutOfService,Instance has failed at least the UnhealthyThreshold number of health checks consecutively.,Instance
amazon-ec2 amazon-web-services

1 Answers

  1. Best Answer

    There are a number of considerations to take into account here. Firstly, to resolve the most limiting problem - the lack of SSH access.

    Since your previous launch config did not specify a keypair, you will have no valid credentials with which to access the instance. Unfortunately, the initial keypair cannot be added after the instance is launched.

    To remedy this, you must create a new launch config, passing the --key and --group parameters, in addition to all the parameters you passed previously. --key takes the name of the keypair you want to use, while --group takes the security group name (if not in VPC) or ID.

    In cases where you can't access your instance, the console log can help you to verify the instance has in fact successfully booted. A common issue is boot failure due to missing volumes (especially trying to mount ephemeral volumes that exist only on the larger instance types, when booting a smaller instance type).

    An important point of mention is that an AMI is not updated if you change a running instance. You must explicitly create the new image. As such, if you try to launch a new instance using the same AMI you are currently using on a customized instance, there is a good chance that you will simply be launching one of the default AMIs, and not one with your customizations on it.

    Use ec2-describe-images to determine the block device mapping of your image - and the snapshot the volume is based on - this will verify that you will be mounting an EBS volume that has your customizations built into it.

    If you do not have an up to date AMI to use for autoscaling:

    • Create a snapshot of your EBS volume
    • Create an AMI with ec2-register -n IMAGE_NAME -s SNAPSHOT_ID
      • If you have additional EBS volumes to attach, specify those by adding the --block-device-mapping (-b) parameter (e.g. -b /dev/xvdf=SNAP_ID)
    • Verify that you have the correct block device mappings with ec2-describe-images

    Once you have an up to date AMI, you need to create a new launch config that will use this AMI. If desired, you can pass additional block device mappings to the command. Use as-create-launch-config, passing it your new AMI and all the parameters you used previously.

    Finally, you must update your autoscaling group. This group is associated with a particular launch config - the new launch config will not be automatically detected and has no effect on the autoscaling group until you explicitly associate it. Use as-update-auto-scaling-group GROUP_NAME --launch-configuration CONFIG_NAME to make this change.

    Once the changes have been made, you can simulate an autoscaling event using the as-execute-policy command.

    Remember to give your instances a few minutes to boot up - if your ELB is showing instances as unhealthy, you may want to increase the --grace-period of your autoscaling group.

    • 5