I have a Docker Swarm with various services running. I've added a syslog-ng OSE service using https://github.com/linuxserver/docker-syslog-ng. The syslog-ng stack has the following docker-compose.yml:
---
services:
syslog:
image: registry.local.net/syslog-ng:latest
ports:
- 514:514
volumes:
- /path/on/host/syslog-ng/config:/config
- /path/on/host/log:/log
environment:
- PUID=1000
- PGID=1000
- TZ=Etc/UTC
- LOG_TO_STDOUT=1
# healthcheck:
# test: ["CMD-SHELL", "/usr/sbin/syslog-ng-ctl healthcheck -c /config/syslog-ng.ctl || exit 1"]
# interval: 10s
# timeout: 2s
# retries: 3
deploy:
mode: replicated
replicas: 1
restart_policy:
condition: on-failure
update_config:
parallelism: 1
delay: 10s
networks:
syslog:
networks:
syslog:
external: true
and my syslog-ng.conf file is this:
@version: 4.2
@include "scl.conf"
options {
chain_hostnames(no);
flush_lines(0);
use_dns(no);
time_reopen(1);
};
source net {
tcp(ip(0.0.0.0) port(514));
};
template my_log_format {
template("$MSG\n");
};
filter f_access_log { program("httpd.access") };
destination d_logs {
file(
"/log/TEST"
owner("1000")
group("1000")
perm(0664)
template(my_log_format)
);
};
log {
source(net); filter(f_access_log); destination(d_logs);
};
I have configured an Apache service in the Docker Swarm to send its logs to the syslog-ng service using logger -T -t httpd.accept -n syslog -P 514 --rfc3164
.
Syslog-ng creates the configured log file, receives the log lines from Apache, and appends them to the configured log file regularly. Everything seems to be working the way I want. The only problem is that the syslog-ng Docker container keeps exiting and restarting every 70-80 seconds. I've tried adding a healthcheck (commented out above), but it didn't change anything. Inspecting the exited containers show that the healthchecks all pass right up until the container exits with exit code 137, and the docker log of the exited containers doesn't exhibit anything obviously wrong.
The syslog-ng current log file shows the following (minus the timestamps at the beginning of each line):
Accepting connections; addr='AF_INET(0.0.0.0:514)'
syslog-ng starting up; version='4.8.1'
Syslog connection accepted; fd='14', client='AF_INET(10.0.6.244:37126)', local='AF_INET(0.0.0.0:514)'
Seconds before one of the containers exited, top
inside the container showed the following:
top - 08:31:26 up 169 days, 17:44, 0 user, load average: 1.13, 1.05, 1.05
Tasks: 15 total, 1 running, 14 sleeping, 0 stopped, 0 zombie
%Cpu(s): 23.8 us, 1.8 sy, 0.0 ni, 73.9 id, 0.0 wa, 0.2 hi, 0.2 si, 0.0 st
MiB Mem : 7652.9 total, 405.8 free, 2924.0 used, 4968.6 buff/cache
MiB Swap: 8192.0 total, 7883.4 free, 308.6 used. 4728.9 avail Mem
EDIT: I thought I'd found a solution by adding
time-reap(30);
mark-freq(10);
to the syslog-ng global options. The Docker container stayed up for almost 5 minutes after I initially made that change, but now it's back to exiting and restarting every minute or so even with those options.