Getting real Client IPs over IPv6 with HAProxy inside Docker

I currently have HAProxy running in a Docker container. Docker is just running on one host with IPv4 and IPv6 enabled. HAProxy is the only container with host ports shared and only 80/443 come through the firewall on the external/public interface.

Currently web traffic comes in fine. I have forwardfor set in HAProxy and my backends get the real ClientIP for logging and analytics .. except for IPv6. Although port 80/443 get mapped/translated in such a way by docker so that HAProxy gets the real client IP on IPv4, anything over IPv6 somehow gets translated to IPv4! HAProxy gets a private 172.17.0.1 address as the client IP for all IPv6 connections.

My daemon.json for Docker looks like the following:

{
    "tls": true,
    "tlsverify": true,
    "tlscacert": "/etc/docker/ca.pem",
    "tlscert": "/etc/docker/server.crt",
    "tlskey": "/etc/docker/server-key.pem",
    "ipv6": true,
    "fixed-cidr-v6": "2001:19f0:6001:1c12::/80",
    "hosts": ["127.0.0.1:2376", "10.10.6.10:2376", "fd://"]
}

I've assigned Docker a /80 from my IPv6 subnet. I'm using Vultr as a provider and they assign me 2001:19f0:6001:1c12:: for my IPv6 subnet and 2001:19f0:6001:1c12:5400:01ff:fe49:876e for my host IP address (the machine running docker). I also my DEFAULT_FORWARD policy as ACCEPT in ufw, and have ndppd setup with the following configuration:

proxy ens3 {
  timeout 500
  ttl 30000
  rule 2001:19f0:6001:1c12::/80 {
    static
  }
}

So with IPv6 enabled, I realize my containers all get their own IPv6 addresses. So for my HAProxy instance to get a real Client IP, I need to point the DNS to the container IPv6 address and open a firewall rule for it. But the trouble is, I need that address to be static. Of course this means I can't use the default bridge network and I need to have a user defined network.

I realize I should have done this to being with, and then I remembered what my issue was. I keep getting the following:

.gem/ruby/2.2.0/gems/docker-api-1.33.6/lib/docker/connection.rb:50:in `rescue in request': could not find an available, non-overlapping IPv6 address pool among the defaults to assign to the network (Docker::Error::ServerError)

This is from the Ruby Docker-API library, but the I get the exact same error with the command line tool. I've tried various slices of IPv6 subnets, but none seem to work and people have told me I'll have problems if I use less than a /64.

Remember, the original problem is that I can't get the client-ip in my web server logs for IPv6 connections. If there's a simpler way to solve this, I'm all open to it. If not, what settings do I need to use for my user defined network? I wish I could just delete the default bridge and use its ranges, but that doesn't seem to be an option. Can I keep the default bridge from getting that IPv6 subnet that's defined in the daemon.json and use that range instead for my user defined network (and then statically assign just the HAProxy container so I could use that container's IP in the DNS/AAAA record?) Or is there a simpler way to get IPv6 host ports to connect directly to the container without translation in the same way IPv4 is done?