From the documentation:
maxconn <number>
Sets the maximum per-process number of concurrent connections to <number>. It
is equivalent to the command-line argument "-n". Proxies will stop accepting
connections when this limit is reached. The "ulimit-n" parameter is
automatically adjusted according to this value. See also "ulimit-n". Note:
the "select" poller cannot reliably use more than 1024 file descriptors on
some platforms. If your platform only supports select and reports "select
FAILED" on startup, you need to reduce maxconn until it works (slightly
below 500 in general).
As I understand, when client tries to connect to the proxy and the proxy has maxconn connections, it will just will not send SYN+ACK package to the client so the client will finally timeout.
I want to make haproxy send RST to the client if maxconn is reached. Is it possible?
This is not controlled by haproxy. I think it only sets the listening socket backlog. The "waiting" stuff is done by the operating system. If you would like to send reset instead, you should probably do it from the OS firewall, using the connection tracker or something (in Linux, limit the max number of connections to a specific port using iptables).