Home / server / Questions / 1026728
In Process
Michael Altfield
Asked: 2020-07-26 10:32:09 +0800 CST

Nginx post-ssl-termination debug mirror

  • 772

How can I tell nginx to output all incoming traffic for a given server{} to a specific/additional/redundant location after decrypting it (like port mirroring on routers) so that it can be analyzed for troubleshooting purposes?

I need to debug the traffic coming-in to an https-only website. The server is headless, and I'd rather

  1. Not install a GUI just for wireshark
  2. Not move my private keys off the server for later analysis on my local GUI with wireshark

So wireshark (the most commonly-cited solution for decrypting an https stream) is not a solution. I want something that's CLI-only and requires minimal tools done live on the server. I checked tcpdump's https support; it's absent but you can combine a ton of other tools and get it functional, but it's not exactly ideal.

My web server is nginx. I'm wondering if there's some way to setup nginx such that--in addition to the way it normally handles its traffic for a given server{} block--send the decrypted traffic to some mirror (just for debugging) that I can then analyze directly with tcpdump, ideally in real-time.

Is it possible to add a proxy_pass mirror to an nginx server{} block such that nginx handles the traffic as it usually does plus sending the decrypted traffic to some "debug" location (which could just be an nc -l 8080 or something)?

ssl tcpdump nginx https

3 Answers

  1. I found out recently that nginx does indeed have this functionality beginning with version 1.13.4. Using the mirror directive you can forward a copy of each request to a different HTTP server. The response, if any, is ignored.

    In the server or location for which you want requests mirrored, add a mirror directive with a URL path that will never occur on your web site. Then, add a new location that serves exactly that path and passes the requests to some other server.

    A simplistic example:

    server {
    location / {
        mirror /Jahg3bioshes;
        # ... the rest of your directives
    }

    location = /Jahg3bioshes {
        internal;
        proxy_pass http://sink.server.example$request_uri;
    }
}

    Note that while nginx ignores the response, it does not ignore the time taken to serve the response, and if the mirror server is slow to respond, it will also slow down the original request. It should probably just be some trivial server, e.g. a freshly installed web server with the distribution default configuration.

    • 2

  2. nginx support lua as a scripting language. So you can write your own functions to manipulate and save data by calling ngx.var.request_body and ngx.var.echo_request_body, or use the one which already there, for example https://github.com/thinkinside-LQ/httpdump

    • 1

  3. I can think of two ways of doing what you want. Either by using SSLKEYLOG and tshark, or by installing a reverse TLS proxy.

    OPTION 1: SSLKEYLOG + TSHARK

    The first option would be to log the SSL master secrets to an SSLKEYLOGFILE, capture the encrypted traffic with tcpdump and analyze it with tshark (the command line version of Wireshark).

    OPTION 2: REVERSE TLS PROXY

    The second option would be to install a TLS inspection proxy that supports "Reverse Proxy" mode, which decrypts incoming traffic and then re-encrypts it before forwarding it to the web server. The proxy must also support sending a copy of the traffic in DECRYPTED mode to an external interface somehow.

    You might be able to do "OPTION 2" with SSLsplit, but I'm not sure. I do know that you can do it with PolarProxy though (disclaimer: PolarProxy is developed by us at Netresec). We have a blog post called Reverse Proxy and TLS Termination that explains how to set up the reverse proxy. You can then use tcpreplay to push the decrypted traffic to a network interface.

    • 1