Home / server / Questions / 834587
In Process
armani
Asked: 2017-02-24 15:38:47 +0800 CST

Transparent Proxy to Docker Network Means TCP is Broken

  • 772

My logging setup is a single Docker host with UDP 514 exposed for syslog. An nginx container has its port published so when you send logs to 10.1.1.100 (in the image below) it first hits nginx, whose config for transparent load balancing to Logstash containers is:

user root;
events {worker_connections 32768;}
stream {
 upstream logstash_servers {
  server logstash-collector-01:514;
  server logstash-collector-02:514;
  server logstash-collector-03:514;
 }
 listen 514 udp;
 proxy_pass logstash_servers;
 proxy_bind $remote_addr transparent;
 }
}

Docker_network

This works fine. However, TCP 514 (or anything TCP, for that matter) doesn't. Even when I add the right listener and configs, I believe the TCP handshake does not complete because with nginx doing transparent load balancing, its proxy_bind passes along, e.g. 10.1.1.5 as a source IP to e.g. 172.18.0.4 (a Logstash instance). That instance then tries to complete the handshake but 10.1.1.5 (and any routers along the way) does not know how to route to the Docker network of 172.18.0.0/16.

Is there a solution here to be able to use TCP for logging?

networking nginx docker logstash

1 Answers

  1. It sounds like nginx might not be the best tool for the job here - either haproxy (if you want simple TCP/UDP load-balancing), or rsyslog (if you want something slightly more application-aware) would seem better fits, and would fulfill the same function as nginx currently is.

    Looking at your setup, you could even have a front-end logstash listener on 514, that could forward to another logstash instance for actual processing. As logstash doesn't AFAIK do load-balancing, you could instead 'route' events based on some tags/types you put into place.

    Checking https://www.nginx.com/resources/admin-guide/tcp-load-balancing/, in order to have both TCP and UDP, you would need something like:

    stream {
    upstream logstash_servers {
        server logstash-collector-01:514;
        server logstash-collector-02:514;
        server logstash-collector-03:514;
    }
    server {
        listen 514;
        ...
    }
    server {
        listen 514 udp;
        ...
    }
}

    I am not completely clear on whether using the logstash_servers in the UDP server block will cause nginx to speak UDP or TCP to them, so you might need to make both available.

    I would be a little doubtful that the issue is down to the nginx node using the 'wrong' IP as source when connecting to the logstash backends - typically, your nginx front end will use whatever the 'right' IP is to do that, which in your case, would mean an address within 172.18/16. This is similar to how nginx is used when reverse proxying to localhost: in that situation, nginx will communicate with the localhost application using a source IP of 127.0.0.1 (or ::1), not the IP that nginx saw the request coming from.

    And there is also the fact that you seem to be saying this all works for UDP - which would tend to say routing is not an issue as such, but that something in your TCP setup might be.

    • 1