Dante unable to bind to public ip address

I have a Dante server on a host with multiple public ip addresses but it is unable to bind to the ips of the host. The ips are pingable, I have another service listening on each of them (different port) which works so I know the ips work. On my test server which has 10 public addresses it works with no issues but not on this one which has hundreds. The error:

debug: sockaddr2ifname(): address 138.xxx.93.2.51002 does not belong to interface

Here's my config:

logoutput: /dev/stdout
debug: 1

internal: 138.xxx.93.2 port = 51002
external 138.xxx.93.2

external.rotation: same-same

socksmethod: username none
clientmethod: none

user.privileged: root
user.notprivileged: nobody

client pass {
        from: 45.xxx.56.91/32 to: 138.xxx.93.2/32
        log: error
}

socks pass {
        from: 0/0 to: 0/0
        session.state.key: from
        session.state.max: 100
        session.state.throttle: 10/2
}