Ping a Specific Port

Question

Alexey

Asked: 2020-04-18 06:12:56 +0800 CST2020-04-18 06:12:56 +0800 CST 2020-04-18 06:12:56 +0800 CST

Removing one cookie in proxy_pass of nginx

772

I'm trying to proxy_pass the whole request coming to my nginx to certain base URL to another upstream server removing server cookie. All other cookies, HTTP headers, the URL itself should be kept as is, sent to the upstream and response passed back to my client.

So far what I have tried is following this guide on nginx forum. Here is how my location directives look like:

set $new_cookie $http_cookie;

if ($http_cookie ~ "(.*)(?:^|;)\s*server=[^;]+(.*)") {
  set $new_cookie $1$2;
}

location  ~ ^/d/application(.*)$ {
  # here we serve this from another container running locally, no proxy_pass
  resolver 127.0.0.11;
  proxy_pass         http://application:8080/d/application$1;
  proxy_redirect     off;
  proxy_set_header   Host $host;
}

location  ~ ^/d/(.*)$ {
  # any other URL NOT starting with /d/application we would like to proxy_pass to another backend and remove server cookie
  resolver 127.0.0.11;
  proxy_pass         https://anotherapp.mydomain.com/d/$1;
  proxy_redirect     off;
  proxy_set_header Cookie $new_cookie;
  proxy_set_header   Host $host;
}

The problem. Now for just one request to /d/somethingelse the logs look like this:

proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 101 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 400 106 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 2020/04/17 09:56:45 [error] 9#0: *10 upstream sent too big header while reading response header from upstream, client: 10.18.6.15, server: , request: "GET /d/somethingelse HTTP/1.1", upstream: "https://10.18.16.1:443/d/somethingelse", host: "anotherapp.mydomain.com"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 502 568 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 502 568 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 502 568 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 502 568 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
proxy_1        | 10.18.6.15 - - [17/Apr/2020:09:56:45 -0400] "GET /d/somethingelse HTTP/1.1" 502 568 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"

UPDATE: adding

      proxy_buffer_size          128k;
      proxy_buffers              4 256k;
      proxy_busy_buffers_size    256k;

to the location with proxy_pass removes the too big header error, but still there are many requests sent to the upstream at the same moment for just one request from the browser

Now I'm struggling with just the forwarding to the upstream host. If I comment out proxy_set_header Cookie $new_cookie; and cookie regexp and setting $new_cookie, I'm still seeing many requests to the upstream.

Added proxy_ssl_verify off; to the second location - still seeing the same issue.

1 Answers

Voted

Alexey · Answer 1 · 2020-04-21T00:21:50+08:00

I was able to make it work, but I am still not clear on why I need two exactly the same if's rewriting the value of the cookie.

So the root cause of the issue: I had too many requests, because the cookie was not removed (I forgot I couldn't debug it without actually removing the server cookie as it's the environment that the nginx of https://anotherapp.mydomain.com/d/ will forward this request back to me in case server cookie is present - which had been happening.

Now I have the following question still open: why doesn't it work with just one if like

set $new_cookie $http_cookie;

if ($new_cookie ~ "(.*)(?:^|;)\s*server=[^;]+(.*)") {
  set $new_cookie $1$2;
}

and then

proxy_set_header   Cookie $new_cookie;

in the location block. But it does work with two exactly the same if's like:

set $new_cookie $http_cookie;

if ($new_cookie ~ "(.*)(?:^|;)\s*server=[^;]+(.*)") {
  set $new_cookie $1$2;
}

if ($new_cookie ~ "(.*)(?:^|;)\s*server=[^;]+(.*)") {
  set $new_cookie $1$2;
}

and

proxy_set_header   Cookie $new_cookie;

in the location block?

Here is how the logs look like in case of one if:

proxy_1        | 2020/04/20 04:11:24 [notice] 10#0: *76 "(.*)(?:^|;)\s*server=[^;]+(.*)" matches "cookie1=111; canary=0; cookie2=222; cookie3=1; server=something; server_php=something; server_asp=null; cookie4=something; cookie5=something; XDEBUG_SESSION=PHPSTORM; cookie6=something; cookie7=something; cookie8=something

and below is the case with two if's (when it works) :

proxy_1        | 2020/04/20 04:14:27 [notice] 10#0: *1 "(.*)(?:^|;)\s*server=[^;]+(.*)" matches "cookie1=111; canary=0; cookie2=222; cookie3=1; server=something; server_php=something; server_asp=null; cookie4=something; cookie5=something; XDEBUG_SESSION=PHPSTORM; cookie6=something; cookie7=something; cookie8=something
proxy_1        | 2020/04/20 04:14:27 [notice] 10#0: *1 "(.*)(?:^|;)\s*server=[^;]+(.*)" matches "cookie1=111; canary=0; cookie2=222; cookie3=1; server=something; server_php=something; server_asp=null; cookie4=something; cookie5=something; XDEBUG_SESSION=PHPSTORM; cookie6=something; cookie7=something; cookie8=something

Removing one cookie in proxy_pass of nginx

Can you pass user/pass for HTTP Basic Authentication in URL parameters?

Ping a Specific Port

Check if port is open or closed on a Linux server?

How to automate SSH login with password?

How do I tell Git for Windows where to find my private RSA key?

What's the default superuser username/password for postgres after a new install?

What port does SFTP use?

Command line to list users in a Windows Active Directory group?

What is a Pem file and how does it differ from other OpenSSL Generated Key File Formats?

How to determine if a bash variable is empty?