kvz's questions

Using HAProxy 1.6 and a clever hack, I now have an HAProxy tcp mode frontend, that detects if the browser is capable of SNI, and based on that, routes to a strongly ciphered SSL termination backend, or a weaker one. This ensures A+ grading on SSL labs, while still allowing all browsers except IE6 to use SSL.

Here is my config. It has some template variables in it that should be self-explanatory, but aren't in areas relevant to my question:

frontend https_incoming
 bind 0.0.0.0:443
 mode tcp
 option tcplog
 tcp-request inspect-delay 5s
 tcp-request content accept if { req.ssl_hello_type 1 }
 use_backend https_strong if { req.ssl_sni -m end .transloadit.com }
 default_backend https_weak

backend https_strong
 mode tcp
 option tcplog
 server https_strong 127.0.0.1:1665

frontend https_strong
 bind 127.0.0.1:1665 ssl crt ${DM_ROOT_DIR}/envs/ssl/haproxy-dh2048.pem no-sslv3 no-tls-tickets ciphers ${strongCiphers}
 mode http
 option httplog
 option httpclose
 option forwardfor if-none except 127.0.0.1
 http-response add-header Strict-Transport-Security max-age=31536000
 reqadd X-Forwarded-Proto:\ https
 reqadd FRONT_END_HTTPS:\ on
 use_backend http_incoming

backend https_weak
 mode tcp
 option tcplog
 server https_weak 127.0.0.1:1667

frontend https_weak
 bind 127.0.0.1:1667 ssl crt ${DM_ROOT_DIR}/envs/ssl/haproxy.pem no-sslv3 ciphers ${weakCiphers}
 mode http
 option httplog
 option httpclose
 option forwardfor if-none except 127.0.0.1
 http-response add-header Strict-Transport-Security max-age=31536000
 reqadd X-Forwarded-Proto:\ https
 reqadd FRONT_END_HTTPS:\ on
 use_backend http_incoming

Problem: the https_incoming frontend knows the Client IP, but since it is in mode tcp, it cannot save this information in a mode http X-Forwarded-For header. option forwardfor is not valid in TCP mode.

From another question on serverfault I already found that I could use:

• LVS
• PROXY protocol

So that the X-Forwarded-For header isn't even needed anymore as from what I understand, in the case of LVS: packets are spoofed so the source becomes the Client IP, and in the case of PROXY: packets are encapsulated to carry the Client IP.

These both seem like they could work. LVS however seems quite a heart-surgery for us that could have side-effects, and PROXY has the downside that proxies/application upstream/downstream, might not be fully compatible yet.

I was really hoping for something more lightweight, and that's when I found the new "Capture" feature of HAProxy 1.6 as it mentions:

you can declare capture slots, store data in it and use it at any time during a session.

it goes on to show the following example:

defaults 
 mode http

frontend f_myapp
 bind :9001
 declare capture request len 32 # id=0 to store Host header
 declare capture request len 64 # id=1 to store User-Agent header
 http-request capture req.hdr(Host) id 0
 http-request capture req.hdr(User-Agent) id 1
 default_backend b_myapp

backend b_myapp
 http-response set-header Your-Host %[capture.req.hdr(0)]
 http-response set-header Your-User-Agent %[capture.req.hdr(1)]
 server s1 10.0.0.3:4444 check

It appears to me, information is stored in a frontend, and then later used in a backend, so perhaps I can take the Client IP in TCP mode, save it, and use that later down the line, maybe like so:

http-response set-header X-Forwarded-For %[capture.req.hdr(0)]

I've looked at the capture docs and there it seems capture is more oriented at http mode headers, but then I have also seen a mailing list conversation successfully demonstrating the use of a tcp-request capture.

I've tried several things, among which:

tcp-request capture req.hdr(RemoteAddr) id 0
# or
tcp-request content capture req.hdr(RemoteHost) id 0

But as you can see, I haven't got a clue what the syntax should be and under which key this information would be available, nor can I find it in the (I think) relevant documentation.

Questions: Would it be possible to capture the Client IP in TCP mode, and later down the line, write this information into the X-Forwarded-For header in HTTP mode? If so, what would be the syntax for this?

Let's say I have items A, B, and C. I have two backend servers: server01 and server02.

Item A can be handled by server01, item B and C can be handled by server02. New items and servers get added and removed, and we programatically update a hash table between item id and backend server in a file, Redis, Memcache, or similar (whatever the proxy would support).

Can I make a frontend such as /items/${id}, and have HAProxy route to the correct backend server based on ${id}? So by looking up what id is associated with what server?

If not, would Nginx be able to do this?

On cloud platforms you often hear that because of high load on neighboring VMs, disks over oversubscribed ethernet, backups, or live migration to other hardware, the virtual machine can 'freeze' for a moment.

I have the suspicion that this is happening to one of our Ubuntu virtual machines on a Cloud provider I'm not looking to publicly shame.

Every night it's unavailable to external monitoring services. The machine itself looks healthy in terms of load, traffic, etc. The provider suggests the network is fine though.

I would like to be able to (dis)prove VM freezes are causing these pagers.

One idea I had was to write the date to a log every second, and after an short moment of unavailability see if we skipped a 'beat'.
However that seems flawed because what if the VM maintains its own clock and allows a drift from the Host's hardware.
If our internal clock freezes along with the VM, we'd still have a nice sequence of seconds in that log file, and a clock that's now behind on the real time.

Is there a better way / tool I can use to determine that there are machine freezes?

I would guess real time and our time would be a tell, then again, there are other causes for drifting clocks.

My tar command

tar --remove-files -cvif 2011-08-02_14-05-09.tar 2011-08-02_14-05-09

Writes 80MB/s. That's beyond the capacity we can spare because MySQL starts queuing up writes and eventually we get monitoring alerts cause systems won't wait forever on MySQL and time out.

Q: Can I throttle tar to only do 20MB/s?

Didn't find anything in the manpage and we already tried with the lowest ionice class: /usr/bin/ionice -c3 tar <...> but that doesn't impact the MBs written/s and MySQL still freezes up.

Maybe ionice is ignored because we're on a /dev/md0 software RAID device?