dcmbrown's questions

While trying to figure out why our Varnish 4.1 install (on CentOS7 via the varnish-cache.org repo) was not following vcl rules set out to log the client IP address in an X-Forwarded-For header (see: Varnish 4 logging proxy/load balancer instead of client IP addresses) I happened to notice something odd while looking through varnishlog files:

- Begin        req 9353447 rxreq
- Timestamp    Start: 1488771709.337974 0.000000 0.000000
- Timestamp    Req: 1488771709.337974 0.000000 0.000000
- ReqStart     172.25.20.65 19903
- ReqMethod    GET
- ReqURL       /about-us/
- ReqProtocol  HTTP/1.1
- ReqHeader    host: www.<notreallythishost>.com
- ReqHeader    Accept: */*
- ReqHeader    Accept-Encoding: gzip, deflate
- ReqHeader    Cache-Control: no-cache
- ReqHeader    From: bingbot(at)microsoft.com
- ReqHeader    Pragma: no-cache
- ReqHeader    User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
- ReqHeader    X-Forwarded-For: 40.77.167.41
- ReqHeader    X-Forwarded-Port: 80
- ReqHeader    X-Forwarded-Proto: http
- ReqHeader    Connection: keep-alive
- ReqUnset     X-Forwarded-For: 40.77.167.41
- ReqHeader    X-Forwarded-For: 40.77.167.41, 172.25.20.65
- VCL_call     RECV
- ReqUnset     X-Forwarded-For: 40.77.167.41, 172.25.20.65
- ReqHeader    X-Forwarded-For: 172.25.20.65
- ReqUnset     Accept-Encoding: gzip, deflate
- ReqHeader    Accept-Encoding: gzip
- ReqUnset     User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
- VCL_return   hash
- VCL_call     HASH
- VCL_return   lookup
- VCL_call     MISS
- VCL_return   fetch
- Link         bereq 9353449 fetch
- Timestamp    Fetch: 1488771709.338395 0.000421 0.000421
- RespProtocol HTTP/1.1

This fully explains why we were never able to get anything other than the load balancer IP address logged through varnishncsa at any point regardless of the logging technique we used.

It looks as though in the request processing it builds up the X-Forwarded-For header by adding the AWS Load Balancer IP address to the header, but as vc_call is called summarily decontructs it again and removes the original client IP address. So how do I go about keeping the X-Forwarded-For intact and why is Varnish shifting IPs off the left side rather than than just adding them onto the X-Forwarded-For header like it should? Bug?

WARNING: Long. Lots of info here.

3 years ago someone asked Why is iptables not blocking an IP address? and it turned out the reason was because the servers were behind CloudFlare which made it impossible to block IP addresses directly they way they wanted to unless you use it differently. Any reverse proxy or load balancer would cause the same thing.

Similarly we have setup fail2ban with a rule to ban any bots which attempt to brute-force their way into the administrative login or spam xmlrpc. The site is sitting behind a load balancer so obviously we can't directly ban the IP address but iptables is supposed to be accepting the connection and pattern matching the packet data to ban specific traffic.

This is fail2ban jail.conf config:

[wp-auth] 
enabled = true
filter = wp-auth
action = iptables-proxy[name = lb, port = http, protocol = tcp]
         sendmail-whois[name=LoginDetect, [email protected], [email protected], sendername="Fail2Ban"]
logpath = /obfuscated/path/to/site/transfer_log
bantime = 604800
maxretry = 4
findtime = 120

This is the simply pattern match for wp-login requests:

[Definition]
failregex = ^<HOST> .* "POST /wp-login.php
ignoreip = # our ip address

This is our fail2ban iptables action which is supposed to be able to block these bots but for the most part doesn't seem to. It is from the CentOS site Tips section for fail2ban behind a proxy. For the sake of brevity I've left only the section header comments in place.

# Fail2Ban configuration file
#
# Author: Centos.Tips
#

[INCLUDES]

before = iptables-blocktype.conf

[Definition]

# Option:  actionstart
actionstart = iptables -N fail2ban-<name>
              iptables -A fail2ban-<name> -j RETURN
              iptables -I <chain> -p <protocol> --dport <port> -j fail2ban-<name>

# Option:  actionstop
actionstop = iptables -D <chain> -p <protocol> --dport <port> -j fail2ban-<name>
             iptables -F fail2ban-<name>
             iptables -X fail2ban-<name>

# Option:  actioncheck
actioncheck = iptables -n -L <chain> | grep -q 'fail2ban-<name>[ \t]'

# Option:  actionban
actionban = iptables -I fail2ban-<name> 1 -p tcp --dport 80 -m string --algo bm --string 'X-Forwarded-For: <ip>' -j DROP

# Option:  actionunban
actionunban = iptables -D fail2ban-<name> -p tcp --dport 80 -m string --algo bm --string 'X-Forwarded-For: <ip>' -j DROP

[Init]
# Default name of the chain
name = default

# Option:  port
port = http

# Option:  protocol
protocol = tcp

# Option:  chain
chain = INPUT    

So as I mentioned the site is on a pair of servers behind an elastic load balancer and seems to work in test. We can add any of our own IP addresses and we cannot reach the site. Despite this bots seem to be able to get through.

[root:~/] iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N fail2ban-SSH
-N fail2ban-lb
-A INPUT -p tcp -m tcp --dport 80 -j fail2ban-lb
-A INPUT -p tcp -m tcp --dport 22 -j fail2ban-SSH
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5666 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 3306 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 24007:24020 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A fail2ban-SSH -j RETURN
-A fail2ban-lb -p tcp -m tcp --dport 80 -m string --string "X-Forwarded-For: 91.200.12.33" --algo bm --to 65535 -j DROP
-A fail2ban-lb -p tcp -m tcp --dport 80 -m string --string "X-Forwarded-For: 91.134.50.10" --algo bm --to 65535 -j DROP
-A fail2ban-lb -p tcp -m tcp --dport 80 -m string --string "X-Forwarded-For: 160.202.163.125" --algo bm --to 65535 -j DROP
-A fail2ban-lb -p tcp -m tcp --dport 80 -m string --string "X-Forwarded-For: 162.243.68.232" --algo bm --to 65535 -j DROP
-A fail2ban-lb -j RETURN

Port 80 is the only port open to all. All others are ACL'd via AWS Security Groups. IPtables appears to be processing in the correct order and should therefore be blocking these IPs based on their X-Forwarded-For header. There is a Firefox plugin which allows you to send these headers with initial requests and we get blocked as a result with any of these bot IPs as well.

The source IP address does not appear to be forging the X-Forwarded-For header as we've been playing with as the ELB rewrites them anyway. tcpdump does not show any extra information on the packet at the server level.

22:07:14.309998 IP ip-10-198-178-233.ec2.internal.11054 > ec2-10.4.8.71.http: Flags [P.], seq 2545:3054, ack 19506, win 166, options [nop,nop,TS val     592575835 ecr 2772410449], length 509
E..1..@[email protected]
...
f.p+..P.Nz.
20............
#Q.[.?.QPOST /wp-login.php HTTP/1.1
host: www.thiswebsite.com
Accept: */*
Accept-Language: zh-cn
Cache-Control: no-cache
Content-Type: application/x-www-form-urlencoded
Referer: http://www.thiswebsite.com/wp-login.php
User-Agent: Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; 125LA; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022)
X-Forwarded-For: 91.200.12.33
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Content-Length: 21
Connection: keep-alive

These requests are all being logged in the transfer_log. When we do the same thing and forge the X-Forwarded-For we get caught by iptables before ever reaching Apache. tcpdump also shows our extra IPs.

20:10:25.378873 IP ip-10-198-178-233.ec2.internal.11054 > ec2-10.4.8.71.http: Flags [P.], seq 3157:3860, ack 124583, win 267, options [nop,nop,TS     val 526293643 ecr 2507283790], length 703
E...Tf@.@.[.
...
f.p,O.P...GU........m.....
.^...r.QPOST /wp-login.php HTTP/1.1
host: www.thiswebsite.com
Accept: /
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.5
Cache-Control: no-cache
Cookie: __utma=190528439.16251225.1476378792.1478280188.1478289736.3; __utmz=190528439.1476378792.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);     _icl_current_language=en; __utmc=190528439; __utmb=190528439.2.10.1478289736; __utmt=1
Pragma: no-cache
Referer: http://www.thiswebsite.com/
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0
X-Forwarded-For: 91.200.12.33, <our ip address>
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Connection: keep-alive

I also have the ELB access log here which I expect to see an entry for, just not the Apache transfer logs.

2016-11-07T22:07:14.309917Z mLB 91.200.12.33:60407 10.4.8.71:80 0.000079 1.99244 0.000091 200 200 21 3245 "POST http://www.thiswebsite.com:80/wp-login.php HTTP/1.1" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; 125LA; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022)" - -

So the IP address (at least according to the ELB) does not appear to be forced at the X-Forwarded-For level. Why is traffic from it not being blocked? The IP address also shows up constantly in the fail2ban log with the usual:

fail2ban.actions[11535]: INFO [wp-auth] 91.200.12.33 already banned

We have a number of groups of servers on AWS that use trios of servers to serve out sites. Each server, or set of servers is within a different availability region, one of which is a redirection server while the other two are the primary servers accessed behind an elastic load balancer. We finally have one client on board for using Route53 and was looking up some details of pointing zone apex domains at elastic load balances. Most DNS providers which support this seem to refer to this as DNS aliases of A records. Some sites however seem to refer to a similar record called an ANAME record. This is not the typical CNAME, or the atypical DNAME record. Most of these sites also refer back to DNSMadeEasy simpleDNS which uses it.

So what is the difference between Route53 (or elsewhere's) A record ALIASes and ANAME records? Is there one? Does either of these have, or are included in an IEFT RFC implementation document or are aliased records all implemented within the provider they are use on?

[edit]: incorrect DNS provider listed for ANAME usage