I have an ECS cluster (on Fargate 1.4.0
) with many tasks and services that all log into Cloudwatch, and everything is fine. I have a couple of ECS scheduled tasks (via EventBridge), and I know they are running at the scheduled time as expected. I know it because a) I can see it in the EventBridge Rule monitoring tab, and b) one of the scheduled tasks job is to send an email, and I receive that email. So it is running but not logging into CloudWatch like the other tasks?
Before I go to my troubleshooting steps, let me give you more insights:
In each task definition, I have this log block:
logConfiguration = {
logDriver = "awslogs"
options = {
awslogs-group = aws_cloudwatch_log_group.ecs_log_group.name
awslogs-region = "us-east-1"
awslogs-stream-prefix = "prod-cron-engage"
}
}
I know everything is correct there because my other tasks that are not scheduled (running via services 24/7) are logging successfully there.
Each task has these two parameters:
execution_role_arn = aws_iam_role.ecs_task_execution_role.arn
task_role_arn = aws_iam_role.ecs_task_execution_role.arn
These are:
resource "aws_iam_role" "ecs_task_execution_role" {
name = "ecsTaskExecutionRole"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "ecs-tasks.amazonaws.com"
}
}
]
})
tags = {
"Name" = "${var.name_prefix}-iam-ecs-role"
}
}
resource "aws_iam_role_policy_attachment" "ecs_task_execution_role_policy" {
role = aws_iam_role.ecs_task_execution_role.name
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}
In the AmazonECSTaskExecutionRolePolicy
, there are essential policies for ECS, including access to CloudWatch.
Also, here is my event bridge rule:
resource "aws_cloudwatch_event_rule" "prod_cron_engage_rule" {
name = "prod-engage-rule"
description = "Run Prod Engage task every 30 minutes."
schedule_expression = "rate(30 minutes)"
}
resource "aws_cloudwatch_event_target" "prod_cron_engage_target" {
target_id = "run-prod-engage-task-every-half-an-hour"
rule = aws_cloudwatch_event_rule.prod_cron_engage_rule.name
arn = aws_ecs_cluster.ecs_cluster.arn
role_arn = aws_iam_role.eventbridge_role.arn
ecs_target {
task_definition_arn = aws_ecs_task_definition.prod_cron_engage_task.arn
task_count = 1
launch_type = "FARGATE"
network_configuration {
subnets = module.vpc.private_subnets
security_groups = [aws_security_group.ecs_sg.id]
assign_public_ip = false
}
tags = {
"Name" = "${var.name_prefix}-ecs-prod-cron-engage"
}
}
}
Here are the EventBridge roles and policies:
resource "aws_iam_role" "eventbridge_role" {
name = "eventbridge-ecs-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Principal = {
Service = "events.amazonaws.com"
}
Effect = "Allow"
Sid = ""
}
]
})
}
resource "aws_iam_role_policy" "eventbridge_policy" {
name = "eventbridge-ecs-policy"
role = aws_iam_role.eventbridge_role.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "ecs:RunTask"
Effect = "Allow"
Resource = [
aws_ecs_task_definition.prod_cron_engage_task.arn
]
},
{
Action = "iam:PassRole"
Effect = "Allow"
Resource = aws_iam_role.eventbridge_role.arn
}
]
})
}
What did I do so far?
At first, I thought maybe, somehow, there is a restriction rule that denies those scheduled tasks from access to CloudWatch (unlikely, but I thought since they are now scheduled with EventBridge, maybe it is possible), So I give both ECS and EventBridge full CloudWatch access. Doesn't change anything.
I tried to create a new log group with those broad permissions and see if the tasks can create that new log group or not. The new log group wasn't there, so the log groups couldn't be created through task definitions.
The EventBridge Policy does not have enough permissions.
Based on this documentation, I changed my Policy to this, and with this, everything works as expected!