Using the official latest HAProxy Docker container and the following config file:
frontend logging_frontend
bind *:1514
mode tcp
timeout client 1m
default_backend logging_backend
backend logging_backend
mode tcp
balance roundrobin
timeout connect 10s
timeout server 1m
server logstash-collector-01 logstash-collector-01:1514 check
server logstash-collector-02 logstash-collector-02:1514 check
server logstash-collector-03 logstash-collector-03:1514 check
I'm getting logs (sent via nxlog to 1514 over TCP) which don't store the actual client IP but rather the Docker gateway IP. For example:
2017-03-02T15:57:41.585Z 172.18.0.1 {"EventTime":"2017-03-01 15:25:02","Hostname":"server.mycompany.net","Keywords":58096435992,"EventType":"ERROR","SeverityValue":4,"Severity":"ERROR","EventID":551,"SourceName":"Microsoft-Windows-SMBServer","ProviderGuid":"{D48CE617-33A2-4BC3-A5C7-11AA4F29619E}","Version":1,"Task":551,"OpcodeValue":0,"RecordNumber":5267740,"ProcessID":4,"ThreadID":5732,"Channel":"Microsoft-Windows-SMBServer/Security","Domain":"NT AUTHORITY","AccountName":"SYSTEM","UserID":"S-1-5-184","AccountType":"User","Message":"SMB Session Authentication Failure\r\n\r\nClient Name: \\10.1.1.43\r\nClient Address: 10.1.1.43:54118\r\nUser Name: \r\nSession ID: 0xAF9FC8000015\r\nStatus: The attempted logon is invalid. This is either due to a bad username or authentication information. (0xC000006D)\r\n\r\nGuidance:\r\n\r\nYou should expect this error when attempting to connect to shares using incorrect credentials.\r\n\r\nThis error does not always indicate a problem with authorization, but mainly authentication. It is more common with non-Windows clients.\r\n\r\nThis error can occur when using incorrect usernames and passwords with NTLM, mismatched LmCompatibility settings between client and server, duplicate Kerberos service principal names, incorrect Kerberos ticket-granting service tickets, or Guest accounts without Guest access enabled","Opcode":"Info","EventReceivedTime":1488470262,"SourceModuleName":"eventlog","SourceModuleType":"im_msvistalog"}
Note that the reporting server is 10.2.3.95, the Docker host is 10.1.38.223, and the Docker bridge interface (and local container subnet) is 172.18.0.0/16 (with 172.18.0.1, logged in the log, being the gateway).
From a tcpdump (modified to show interfaces):
- The external interface, eno16777728, gets a PUSH packet (containing log data) from 10.2.3.95 to the host IP 10.1.38.223
- This gets DNAT'd by Docker to go from 10.2.3.95 to HAProxy container (172.18.0.6)
- HAProxy acknowledges the PUSH (final packet of screenshot shows the SNAT'd reply back on the external interface using 10.x.x.x IPs)
- (Out of order in image - check timestamps) HAProxy decides on backend container 172.18.0.3 for load balancing destination
- (Out of order in image - check timestamps) The Docker interface gateway SNATs the packet on its virtual ethernet interface (WHY?)
- ACK from backend server to Docker gateway
- ACK from backend server to HAProxy (?)
- ACK from backend server to HAProxy (?)
When I try to make HAProxy pass on the client IP transparently using source 0.0.0.0 usesrc clientip
in the backend
stanza, nothing gets logged at all.
To me, this question has two avenues to success; either:
- How can the behavior of logging the Docker gateway IP be explained?
or more importantly:
- How can I get the original client IP to appear in these logs, preferably as the client IP field (at beginning of log following timestamp)?
0 Answers