Home / server / Questions / 562327
In Process
SunSparc
Asked: 2013-12-19 15:08:37 +0800 CST

AWS Alarms in "ALARM" state not triggering Policy actions

  • 772

Today I created an AutoScaling Group with several policies, each with an Alarm associated with a CloudWatch metric. The AutoScaling Group launched two instances, because that the minimum that I required for that group. I turned on my benchmarking tests to hit the new group and their combine average CPUUtilization rose above the thresholds that I had set for the Alarms. I watched the History for each of those Alarms as the states changed from INSUFFICIENT_DATA, then to OK, then finally to ALARM. As expected. Then I waited for the Policies to be executed. The specified 300 seconds before execution came and went. Then I decided to see how long it would take before the Policies executed. It has now been over an hour and a half and those Policies have yet to execute.

I got frustrated and deleted the ASG and all the Policies and Alarms. I decided that perhaps I had made an error in creating the Policies and Alarms. So I recreated them all. Same problem. The Alarms are in ALARM state and never execute the Policies. I can execute the Policies manually without problem, but that defeats the point.

If it makes any difference, I created all these things in the AWS Console. In the past I have used the API to create ASGs and Policies and those groups execute policies just fine.

Is there something I am doing wrong? (Or perhaps AWS services are just having issues.)

UPDATE:

Tried this again today. Decided to use the API instead of the Console to see if there was any difference. The Alarm I am using just updated from OK to ALARM 20 minutes ago.

CPUUtilization >= 35 for 5 minutes
Actions: In ALARM: For group TEST use policy Scale Out API (Add 1 instance)
Namespace: AWS/EC2
Metric Name: CPUUtilization
Dimensions: AutoScalingGroupName = TEST
Statistic: Average
Period: 5 minutes

2013-12-19 15:13 UTC-7 - State update - Alarm updated from OK to ALARM

Current time: 15:43 UTC-7 (30 minutes since alarm)

Action has not triggered. Any idea why?

amazon-web-services

1 Answers

  1. I asked this question on the Amazon Forums and apparently there is a recent bug in the creation of Alarms that automatically sets the "ActionsEnabled" property to False.

    From AWS:

    We have identified an issue in the AutoScaling console regarding the binding of
AutoScaling policies to CloudWatch alarms and are working on a fix. We will post
an update to this thread once the fix is rolled out. Thanks for bringing this to
our attention.

    The workaround for now:

    In the meantime, please try calling the DescribeAlarms CloudWatch API. If the
alarms associated with your policies have ActionsEnabled=false, then this could
cause your policies to not be invoked when the alarm is triggered. Please try
calling the PutMetricAlarm CloudWatch API to update ActionsEnabled=true for the
affected alarms, and that should fix the issues you are experiencing.

    I have confirmed the bug and the workaround with my own Alarms through the API.

    • 1