Home / server / Questions / 1046237
Accepted
Rovanion
Asked: 2020-12-16 02:56:05 +0800 CST

How do I configure the log level of Prometheus' Alertmanager?

  • 772

I've set up Prometheus' Alertmanager to, well, manage alerts from Prometheus. I've got the alerts coming in from Prometheus to Alertmanager, but there the story ends. The alerts never get sent on by Alertmanager to my email endpoint.

In order to figure out where exactly inside Alertmanager that the alerts end their journey I want to turn the log level from info to debug, but have been unable to figure out how. Even finding the log seems like a tough ask right now, its not in /var/log and journalctl -u alertmanager contains so little that it may be that there is another log somewhere.

The manual page for configuring Alertmanager does not mention debug level. I've looked through the source code for mentions of log and found that the setting should be named log.level. Adding the following snippet to the configuration YAML didn't help either:

log:
  level: debug

as Alertmanager failed to start with a failure to parse its config file.

prometheus

2 Answers

  1. Best Answer

    The answer is that its not possible to set Prometheus' Alertmanagers log level to debug through the config file, it is only possible through commandline arguments. Do not ask me why, I'm sure they had their reasons.

    Through Puppet I added the argument to the Systemd unit file for Alertmanager, so that it ended up looking like this:

    [Unit]
Description=Prometheus alertmanager
Wants=basic.target
After=basic.target network.target

[Service]
User=alertmanager
Group=alertmanager
EnvironmentFile=/etc/sysconfig/alertmanager
ExecStart=/usr/local/bin/alertmanager \
--config.file=/etc/alertmanager/alertmanager.yaml \
--storage.path=/var/lib/alertmanager \
--log.level=debug

ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=always

[Install]
WantedBy=multi-user.target

    If you are starting Alertmanager from your shell you can just add the flag --log.level=debug to your invocation.

    The debug messages can then be seen via journalctl -u alertmanager on Linux distributions with the Systemd init system.

    • 3

  2. On Ubuntu systems (if prometheus-alertmanager was installed via apt and from official repo).

    Open

    /etc/default/prometheus-alertmanager

    And add

    ARGS="--log.level=debug"

    to file (replace ARGS="")

    Then restart prometheus-alertmanager via systemd:

    sudo systemctl restart prometheus-alertmanager.service

    Show logs via journalctl:

    journalctl -u prometheus-alertmanager.service --since today
    • 2