Quintin Par's questions

I have a website with all of the pages served from nginx’s http cache and rarely invalidated or expired.

The average total page download size is around 2 MB But despite being a static site with no funny logic my server response is around a second

I recorded nginx’s $request_time and it comes to around 400 milliseconds from the server

and each file at 20-30 KB average

400 millisecond seems to be absurd.

I am behind Cloudflare and

sendfile        on;
tcp_nopush     off;
tcp_nodelay on;
keepalive_timeout  300s;
keepalive_requests 10000;

What should I be doing to bring down the response time to the 150-millisecond range?

Edit: First part of my tunning.

Realized I didn’t have SSL OSCP on. Tweaked code to

# https://github.com/autopilotpattern/wordpress/issues/19
    ssl_session_cache   shared:SSL:50m;
    ssl_session_timeout 1d;

    ssl_certificate /etc/letsencrypt/live/site.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/site.com/privkey.pem;
    ssl_trusted_certificate /etc/letsencrypt/live/site.com/chain.pem;
    ssl on;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
    ssl_prefer_server_ciphers on;
    ssl_stapling on;
    ssl_stapling_verify on;

I’ll report back on improvement.

Edit 2:

Here’s the webpage test result for 3G connection hit from India to a US west coast server

My proxy cache path is set to a very high size

proxy_cache_path  /var/lib/nginx/cache  levels=1:2   keys_zone=staticfilecache:180m  max_size=700m;

and the size used is only

sudo du -sh *
14M cache
4.0K    proxy

Proxy cache valid is set to

proxy_cache_valid 200 120d;

I track HIT and MISS via

add_header X-Cache-Status $upstream_cache_status;

Despite these settings I am seeing a lot of MISSes. And this is for pages I intentionally ran a cache warmer an hour ago.

How do I debug why these MISSes are happening? How do I find out if the miss was due to eviction, expiration, some rogue header etc? Does Nginx provide commands for this?

Edit: Full config

    # at http level
    proxy_cache_path  /var/lib/nginx/cache  levels=1:2 inactive=400d keys_zone=staticfilecache:180m  max_size=700m;
    proxy_temp_path /var/lib/nginx/proxy;
    proxy_connect_timeout 30;
    proxy_read_timeout 120;
    proxy_send_timeout 120;
    #prevent header too large errors
    proxy_buffers 256 16k;
    proxy_buffer_size 32k;
    #httpoxy exploit protection
    proxy_set_header Proxy "";

    # at server level 
    add_header Cache-BYPASS-Reason $skip_reason;

    # define nginx variables
    set $do_not_cache 0;
    set $skip_reason "";
    set $bypass 0;

    # security for bypass so localhost can empty cache
    if ($remote_addr ~ "^(127.0.0.1|Web.Server.IP)$") {
        set $bypass $http_8X0;
    }

    # skip caching WordPress cookies
    if ($http_cookie ~* "comment_author_|wordpress_(?!test_cookie)|wp-postpass_" ) {
        set $do_not_cache 1;
        set $skip_reason Cookie;
    }

    # Don't cache URIs containing the following segments
    if ($request_uri ~* "/wp-admin/|/xmlrpc.php|wp-.*.php") {
        set $skip_cache 1;
        set $skip_reason URI;
    }

    # https://guides.wp-bullet.com/how-to-configure-nginx-reverse-proxy-wordpress-cache-apache/
    location / {
            proxy_pass http://127.0.0.1:8000;

            proxy_set_header X-Real-IP  $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto https;
            proxy_set_header X-Forwarded-Port 443;
            proxy_set_header Host $host;
            proxy_set_header Accept-Encoding "";

            # may need to comment out proxy_redirect if get login redirect loop
            proxy_redirect off;

            proxy_cache_key "$scheme://$host$uri";
            add_header X-Nginx-Cache-Head "$scheme://$host$uri";
            proxy_cache staticfilecache;
            proxy_cache_valid       200 301 302 100d;
            proxy_cache_valid 404 1m;


            add_header Cache-Control public;

            proxy_ignore_headers Expires;
            proxy_ignore_headers  "Cache-Control";
            proxy_ignore_headers X-Accel-Expires;

            proxy_hide_header "Cache-Control";
            proxy_hide_header Pragma;
            proxy_hide_header Server;
            proxy_hide_header Request-Context;
            proxy_hide_header X-Powered-By;
            proxy_cache_revalidate on;

            proxy_hide_header X-AspNet-Version;
            proxy_hide_header X-AspNetMvc-Version;
            #proxy_pass_header X-Accel-Expires;


            add_header X-Nginx-Cache-Status $upstream_cache_status;

            proxy_cache_use_stale  error timeout invalid_header updating http_500 http_502 http_503 http_504;
            proxy_cache_bypass $arg_nocache $do_not_cache $http_8X0;
            proxy_no_cache $do_not_cache;

    }

    location ~* \.(jpg|png|gif|jpeg|css|js|mp3|wav|swf|mov|doc|pdf|xls|ppt|docx|pptx|xlsx)$ {
            proxy_cache_valid 200 120d;
            expires 364d;
            add_header Cache-Control public;
            proxy_pass http://127.0.0.1:8000;
            proxy_cache staticfilecache;
            add_header X-Nginx-Cache-Status $upstream_cache_status;
            proxy_cache_use_stale  error timeout invalid_header updating http_500 http_502 http_503 http_504;
    }

I run a fairly heavy trafficked website and due to some unfortunate incidents the machines that are in my cloud at Linode went down. And I have only a single Load Balancer machine exposed to the outside world (one IP).

Also my site is a candidate for over 6,000 static pages that can be mirrored. Now my DNS is CloudFlare.

What can I do to maintain a static mirror of my site and route to it incase my site goes down.
Since I am running from Linode I don’t have anything like Route53 to detect downtime at an IP address and point to another one.

What are the strategies that people use statically mirror site and work against downtime?

My Nginx reverse proxy works on the same machine as the webserver(apache) as follows

server {  server_name site.net;
    location / {
        proxy_pass http://localhost:82;
        proxy_set_header   Host             $host;
        proxy_set_header   X-Real-IP        $remote_addr;

    }
}

Now instead of using TCP connections to the backend apache, how can I tune it to use unix sockets?

Edit:
Can someone help with the full flow, instructing apache to listen on unix sockets too

Is there a way I can make Nginx to notify me if hits from a referrer goes beyond a threshold?

e.g If my website is featured at Slashdot and all of sudden I have 2K hits coming in an hour I want to be notified when goes beyond 1K hits an hour.

Will it be possible to do this in Nginx? Possibly without lua? (since my prod is not lua compiled)

I am creating a user as follow

user { $username:
    comment => "$name",
    shell   => "$shell",
    managehome => false,
    password  => "$password",
    groups => $groups
}

Now as u can see I am doing a managehome is false Now later down the lane I need to push a file to the user’s home directory.

$key = "${homedir}/${name}/file"

    file { $key:
    ensure => present,
    owner  => $username,
    group  => $username,
    mode   => 600,
    content => "$keyvalue",
    subscribe => User[$username],
}

How can I get the user’s home directory for this?

I tried to run logwatch at follows

[root@machine cron.daily]# ./0logwatch 
ERROR: Date::Manip unable to determine TimeZone.

Execute the following command in a shell prompt:
        perldoc Date::Manip
The section titled TIMEZONES describes valid TimeZones
and where they can be defined.

My date is as follows

root@machine cron.daily]# date
Thu Aug 23 06:25:21 GMT 2012

Now based on details in various forums I tried to fix this by setting

/etc/timezone to “+0800”

but it didn’t work

My /etc/localtime points to /usr/share/zoneinfo/GMT and is managed by puppet

How do I go about fixing this? I still want all my machines to be in GMT timezone.

EDIT:
Sadly,
Both the changes are not working:

[root@machine cron.daily]# cat /etc/TIMEZONE 
UTC

Quanta’s

[root@machine cron.daily]# cat ~/.bash_profile 
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin
export TZ=GMT

export PATH
[root@machine cron.daily]# source ~/.bash_profile
[root@machine cron.daily]# ./0logwatch 
ERROR: Date::Manip unable to determine TimeZone.

Execute the following command in a shell prompt:
        perldoc Date::Manip
The section titled TIMEZONES describes valid TimeZones
and where they can be defined.

I have two changes to ip route & sysctl that disable tcp slow start. Here’s how I do it

ip route show

Make a note of the line starting with default.

Pick up the IP from the default line and run

sudo ip route change default via $ip_address dev eth0 initcwnd 12 
sudo sysctl -w net.ipv4.tcp_slow_start_after_idle=0

How can I create a puppet script out of this? One that can be deployed to many machines of the same type – CentOS 6

Edit: Added bounty to get a working example for

sudo ip route change default via $ip_address dev eth0 initcwnd 12 

I changed the tcp initial window in my machine to 10 as shown below

[user@site etc]$ sudo ip route change default via 17.255.209.1 dev eth0  proto static initcwnd 10 

And changed tcp_slow_start_after_idle as shown below

[user@site etc]$ sudo sysctl -a | grep tcp_slow_start_after_idle
net.ipv4.tcp_slow_start_after_idle = 0

a ip route show confirmation is given below

[user@site etc]$ ip route show
default via 17.255.209.1 dev eth0  proto static  initcwnd 10
169.254.0.0/16 dev eth0  scope link  metric 1002
17.255.209.0/24 dev eth0  proto kernel  scope link  src 17.255.209.19

Now when I do a tcpdump on the website I don’t seem to see a change in the initial window with the WIN/MSS remaining 4 as default. 5840/1460=4

[user@site etc]$ sudo tcpdump -n -i any 'tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn and port 80'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
11:17:45.048174 IP 21.101.151.198.45873 > 17.255.209.19.http: Flags [S], seq 2008673341, win 5840, options [mss 1460,sackOK,TS val 1724223146 ecr 0,nop,wscale 6], length 0

The curl hit I did to the webpage requested around 30 KB of data.

[user@machine ~]$ curl http://www.site.com/js/main.js > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 88212  100 88212    0     0   179k      0 --:--:-- --:--:-- --:--:--  272k

What could be wrong in my approach?

Kernel

[user~]$ uname -r
3.0.4x86_64-linode21

As an update, here' are the results when I try google.com

[user@site ~]$ sudo tcpdump -n -i any 'tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn and host www.google.com'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
17:20:28.033236 IP 17.255.209.19.42799 > 74.125.127.106.http: Flags [S], seq 3148947324, win 14600, options [mss 1460,sackOK,TS val 193695310 ecr 0,nop,wscale 4], length 0

As you can see WIN/MSS is 14600/1460=10 in this case

I tried hitting my site from the server machine itself through curl and here's the result:

[user@site ~]$ sudo tcpdump -n -i any 'tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn and host www.site.com'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
17:25:14.584338 IP 17.255.209.19.35008 > 17.255.209.19.http: Flags [S], seq 3894567470, win 32792, options [mss 16396,sackOK,TS val 193981861 ecr 0,nop,wscale 4], length 0

WIN/MSS is 32792/16396=2 in this case

I was checking the IW for my site and realized that its 4.

[user~]$ sudo tcpdump -n -i any 'tcp[tcpflags] & (tcp-syn|tcp-ack) == tcp-syn and port 80'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
10:13:20.971391 IP 175.x.x.x > 17.x.x.x.http: Flags [S], seq 3929903562, win 5840, options [mss 1460,sackOK,TS val 1701657127 ecr 0,nop,wscale 6], length 0

I run

[user~]$ uname -r
3.0.4x86_64-linode21

How can I increase the Initial Window?

Do you recommend recompiling to new kernel? (Don’t want to do) or is there another way?

An nginx HTTP cached hit takes

[site ~]$ time curl www.site.com/ >/dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6440    0  6440    0     0  26002      0 --:--:-- --:--:-- --:--:-- 39753

real    0m0.268s
user    0m0.004s
sys     0m0.012s

at least .3 sec.

I run a nmap scan of my hosts daily to check for open ports.

sudo nmap -f -sS -sV --log-errors -append-output -p1-9999 host.com 

But along with the output I get a long list of fingerprint submissions for unrecognized ports like this

==============NEXT SERVICE FINGERPRINT (SUBMIT INDIVIDUALLY)==============
SF-Port4000-TCP:V=5.21%I=7%D=2/9%Time=4F30CAC%P=x86_64-redhat-linux-gnu%r
SF::\r\nERR\x20UNKNOWN_COMMAND\x20Unknown\+server\+commandCSeq:\r\nERR\x20
-------------------------------------------

How do I remove these from my nmap reports?

I have a blog hosted at http://site.com/blog.

How do I instruct nginx to rewrite requests from site.com to site.com/blog?

This should not be permanent.

I have a website primarily hosted on EC2 with the backends (MySql) replicated async to a Rackspace store. I am also planning to have a suite of web server, cache & db on standby at Rackspace in the eventuality that Amazon has an outage.

With this is perspective, is there a solution that will allow me to switch between EC2 and Rackspace depending on availability?

I have my static assets configured as

 location @upstream {
   proxy_pass http://localhost:82;

    proxy_set_header   Host             $host;
    proxy_set_header   X-Real-IP        $remote_addr;
    proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
 }

 location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
     try_files $uri @upstream;
     access_log off;
     expires max;
     add_header Cache-Control public;
 }

The expires max is not respected.
I think its respecting the incoming server header.

How can modify this to set expires header?

I have a micro instance mounted with an EBS volume. I have to pay per million read/writes to the EBS volume.

But by the very nature of EBS I/O is very slow and consequently services like MySql is slow.

Can someone suggest ways to speed up my instance (barring the obvious comment that micro instances are not for such use)?

I have nginx running multiple domains under a single server directive as

server {
        listen       80;
        server_name  www.domain.com;
        server_name  x.domain.com;
        server_name  y.domain.com;

----
----
----
}

Now, I need to use location directive to match a subdomain and apply basic auth to it. The equivalent of

location x.domain.com {
        auth_basic "Admin Login";
        auth_basic_user_file /etc/nginx/.htpasswd;
}

How do I do this?

Can someone teach me how to configure a centos instance to send mails.

My intention is to send mails from a bash script based on some signals and alerts to a build user. I don’t intend to use it as a mass mailing machine.

What are the differences between HAProxy and Nginx when it comes to their abilities as a reverse proxy?

Other than for historical reasons, is there is reason to have “www” in a URL?

Should I create a permanent redirect from www.xyz.com to xyz.com, or from xyz.com to www.xyz.com? Which one would you suggest and why?

How does a site like rambler serve dynamic content so fast? Even faster than Yahoo (which has a server in my country- SE Asia; rambler does not).

Is this purely Nginx’s capability? Where should I be looking into to learn about such capabilities?

Pretty much a newbie here, I believe that serverfault.com if served from Nginx will be much faster the IIS 7 (assuming db access time to be same in both the case). Is this a fair assumption?

Edit:

Post from Karl using Nginx in front of IIS7