Home / server / Questions / 742933
In Process
lsh
Asked: 2015-12-15 08:13:49 +0800 CST

AWS Cloudwatch Agent not picking up changes

  • 772

I'm stumped. I have two log files being watched by the AWS CloudWatch agent. The first one, /var/log/nginx/access.log, works perfectly fine. The second, /var/log/otherserver/access.log, is not having any changes picked up. Not eventually, not ever UNLESS I restart the agent, whereupon it picks up the changes and sends them to CloudWatch as expected.

/var/log/otherserver/access.log is a log file rsync'ed periodically from another server that can't have the agent installed on it. Command looks like:

rsync -av user@host:/var/log/access.log /var/log/otherserver/access.log

The agent definitely can read it, because it reads the changes in after being restarted.

The position of the entry within the config file doesn't appear to matter.

The dates in the log entries for the rsync'ed log file are the same as the server (everything is UTC).

If I move the log file the agent starts complaining with:

2015-12-14 16:02:26,158 - cwlogs.push.stream - WARNING - 3344 - Thread-1 - No file is found with given path '/var/log/otherserver/access.log'.

The configuration of this second log file is almost identical to the first (below).

[website.access.log]
#datetime_format = 09/Dec/2015:14:15:02 +0000
datetime_format = %d/%b/%Y:%H:%M:%S %z
file = /var/log/otherserver/access.log
log_stream_name = master-platform.sh
initial_position = start_of_file
log_group_name = web-access

[app.access.log]
#datetime_format = 09/Dec/2015:14:15:02 +0000
datetime_format = %d/%b/%Y:%H:%M:%S %z
file = /var/log/nginx/lax.access.log
log_stream_name = {hostname}
initial_position = start_of_file
log_group_name = web-access

Does anyone have any clue what is going on here? Or can suggest an alternative agent that is less ... screwy?

amazon-web-services

1 Answers

  1. It's possible your agent state file is corrupted because you kept making changes to the configuration. There are two ways to fix this:

    • Option 1: Use a new name for your configuration block header.
      • That is, change [website.access.log] to [something.else].

    • Option 2: Delete the agent state file after stopping the service.

      sudo service awslogs stop
sudo rm /var/lib/awslogs/agent-state
sudo service awslogs start

    Please note that Option 2 may initially cause duplicate logs to be pushed to CloudWatch as a new state file is created.

    • 1