I have an rsyslog server configured to accept remote messages from around ~400 servers, however it seems once the server gets about ~256 connections it stops accepting new connections and the clients see error messages like the following
cannot connect to syslog.example.org:514: Connection timed out
The rsyslog imtcp module is configured as follows
module(load="imtcp"
MaxSessions="1000"
StreamDriver.Mode="1"
StreamDriver.AuthMode="anon"
StreamDriver.Name="gtls"
)
the systemd unit is also configured with LimitNOFILE=16384
and when i check /proc/$pid/limits
i see that the number of limits looks good
$ /proc/2767537# cat limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 7393 7393 processes
Max open files 16384 16384 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 7393 7393 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
however when i start seeing this error the number of files in /proc/$pid/fd
is close to 256.
/proc/2767537$ ls -l fd | wc -l
253
Further to this the established connections always close to 256
$ ss -ntp4 state established sport eq 514 | wc -l
257
this suggest to me that there is some other limit of 256 i need to tweak but im unable to find what/where.
- I can't see any errors i the logs on the syslog server
- using tcpdump I can see the syn packet come in on port 514 but no sys/ack is ever sent