Home / server / Questions / 1036195
Accepted
zwol
Asked: 2020-10-03 12:28:47 +0800 CST

Passing 8192 sockets via systemd socket activation - fails with E2BIG

  • 772

I am trying to have systemd start up a daemon and pass it 8,192 listening sockets. I have a .service and .socket file that work reliably with a more "normal" number of listening sockets, like this:

# a-daemon.socket
[Unit]
Description=A Daemon (sockets)
After=network.target

[Socket]
Accept=no
ListenStream=8192

# a-daemon.service
[Unit]
Description=A Daemon
After=network.target
Requires=a-daemon.socket

[Install]
WantedBy=multi-user.target

[Service]
Type=notify
ExecStart=/usr/local/sbin/a-daemon

But if I swap out a-daemon.socket for a version with 8,192 ListenStream lines, one for each TCP port from 8192 through 16383 inclusive, then the daemon won't start any more. The socket unit can be started just fine, but the service unit fails; the only error message I get is

systemd[17563]: a-daemon.service: Failed to execute command: Argument list too long
systemd[17563]: a-daemon.service: Failed at step EXEC spawning /usr/local/sbin/a-daemon: Argument list too long

As I understand it, this can't actually be a problem with the argument list, because systemd doesn't put the socket fd numbers on the daemon's command line or anything like that. I guessed it was instead a problem with a limit on the number of simultaneous open files, so I set DefaultLimitNOFILE=32768 in /etc/systemd/system.conf and an equivalent setting in /etc/security/limits.conf and rebooted. No change. Then I put ExecStartPre=/usr/sbin/prlimit -n in the .service file and tried to restart it, which confirmed that the increased limit had taken effect:

prlimit[18134]: RESOURCE   DESCRIPTION                             SOFT      HARD UNITS
prlimit[18134]: NOFILE     max number of open files               32768     32768 files

But the service still fails, the same way. And now I'm out of ideas. Can you suggest anything I could try doing to make this work?

(I am aware that listening on 8,192 consecutive TCP ports is a weird thing to do. Please take my word for it that I have a good reason which I cannot share.)

socket systemd max-file-descriptors

1 Answers

  1. Best Answer

    Upon staring at the systemd manpages some more, I realized that there is something systemd puts in the argv area whose size is proportional to the number of listening sockets:

    sd_listen_fds_with_names() is like sd_listen_fds(), but optionally also returns an array of strings with identification names for the passed file descriptors, if that is available and the names parameter is non-NULL. This information is read from the $LISTEN_FDNAMES [environment] variable, which may contain a colon-separated list of names. For socket-activated services, these names may be configured with the FileDescriptorName= setting in socket unit files, see systemd.socket(5) for details.

    (boldface - my emphasis) What this means is, if you have 8192 ListenStream entries in a socket unit file, systemd will try to put LISTEN_FDNAMES=[name]:[name]:..., with 8192 repeats, where name is the FileDescriptorName setting, into the environment for the service. FileDescriptorName defaults to the full basename of the socket unit file. This can easily overflow Linux's fairly low, fixed limit on the length of a single environment variable name+value (MAX_ARG_STRLEN, usually 128k), and thus cause execve(2) to fail with E2BIG.

    For a daemon that doesn't care about the names, the cure is to put

    UnsetEnvironment=LISTEN_FDNAMES

    in the service file ([Service] section). This eliminates the problem environment variable and makes the kernel happy.

    I don't know what you do if you actually need the fd names for something and you hit this limit.

    • 1