Sparsh Gupta's questions

I was wondering if anyone has any comments / benchmarks / suggestions on Varnish Vs Memcache when used to serve content (usually 2-5kb JS file depending on the parameters in request; thousands of diff files are possible)

Currently we are using nginx + Varnish + LAMP to do this.

I was reading about memcached module of nginx and was wondering if anyone has any benchmarks for caching content in Memcache and serving them directly from nginx (or pulling from LAMP incase of a miss), rather then passing the request to Varnish and letting it do the same job.

Overall, is there any benchmark on setting and getting (as well as occassional purging) of content (usually 2-5kb) with Memcache Vs Varnish

Thanks

We rely heavily on memcache and are serving a few billion requests per month. We have 5 memcache servers. Last night, we saw an 25% increase in our traffic. The graphs show that requests and data transfered by each memcache increased and made them crash. It started a chain reaction and each memcache server crashed one after another (Load per server increased).

We found no logs in syslog, messages, memcache log file (Verbose settings was off).

I have two questions:

1. How can I find out why exactly this happened. If load is an issue for memcache, is there any documentation on how much a normal memcache (running on decent config) can handle. How can I increase this value.

2. How can I ensure they never go down again. It eventually impacted our mysql servers and replication and impacted a lot of other related services. Do I need more memcache servers?

I started my memcache using this init.d script: http://pastebin.com/wfMnB4ta where ENABLE_MEMCACHE is YES in /etc/default/memcached

/usr/share/memcached/scripts/start-memcached: http://pastebin.com/LaUugXye

Thanks

Last night one of the developer ran a wrong mySQL statement (Query inserting a row with duplicate primary key) on master machine. Suddenly the replication to its slave stopped. Slave started writing it to mysql_relay_logs and before we could monitor and fix it, the disk ran out of space.

Master is configured to save bin_logs for last 3 days so we still have not completely lost the replication. Can someone help me fix this mess now.

My Master Status:

mysql> SHOW MASTER STATUS\G
*************************** 1. row ***************************
File: mysql-bin.000017
Position: 30844254
Binlog_Do_DB: 
Binlog_Ignore_DB: 
1 row in set (0.00 sec)

Slave Status: mysql

> SHOW SLAVE STATUS\G
*************************** 1. row ***************************
             Slave_IO_State: 
                Master_Host: 192.168.140.110
                Master_User: replication_user
                Master_Port: 3306
              Connect_Retry: 60
            Master_Log_File: mysql-bin.000014
        Read_Master_Log_Pos: 61132382
             Relay_Log_File: mysqld-relay-bin.000037
              Relay_Log_Pos: 1405976
      Relay_Master_Log_File: mysql-bin.000014
           Slave_IO_Running: No
          Slave_SQL_Running: No
            Replicate_Do_DB: 
        Replicate_Ignore_DB: 
         Replicate_Do_Table: 
     Replicate_Ignore_Table: 
    Replicate_Wild_Do_Table: 
Replicate_Wild_Ignore_Table: 
                 Last_Errno: 1062
                 Last_Error: Error 'Duplicate entry '13676-1-2' for key 1' on query. Default database: 'XXX'. Query: 'YYY'
               Skip_Counter: 0
        Exec_Master_Log_Pos: 1405839
            Relay_Log_Space: 8974037551
            Until_Condition: None
             Until_Log_File: 
              Until_Log_Pos: 0
         Master_SSL_Allowed: No
         Master_SSL_CA_File: 
         Master_SSL_CA_Path: 
            Master_SSL_Cert: 
          Master_SSL_Cipher: 
             Master_SSL_Key: 
      Seconds_Behind_Master: NULL
1 row in set (0.00 sec)

Can someone give me exact commands to make everything back to normal? as I said I am sure we have lost data on slave as it ran out of disk but we must be having data on master.

My next question is: How to ensure this doesnt happen again.

Thanks a ton Sparsh Gupta

I have a very common LAMP stack with about 6-7 machines with PHP and Apache and a few machines with MySQL server (master+slave, sharding)

I am exploring options of moving to MySQL Cluster. I couldnt find a lot of information about it and reviews and benefits online specially when used in a webapp with PHP.

Can someone share their thoughts or difficulties / experiences with MySQL Cluster.

Actual Question: Is it a good idea to move to mysql cluster or stick to slaves + sharding for a live webapp.

Thanks

I have a very old server slow (ubuntu 9.04) which has a hell lot of other things running on it along with SubVersion.

I now plan to move subversion to a better faster server but ofcourse dont want to lose any data or history. Loads of people have access to the SVN and I dont want any changes at their end (I will also move the domain to new server).

What is the best way to do this. While moving, I also want to update the old svn version 1.5.4 to the latest stable version 1.7.0

Is there any possible issues I should be aware of before doing this? Any guide which can help me here

Thanks

I am using nginx as reverse proxy with varnish in the background. I have LAMP fueling varnish. The problem I am seeing on Varnish server is that the connection from nginx is getting established everytime a request is served.

We are serving more than 300 req/second and we see something like:

netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
      2 ESTABLISHED
      1 FIN_WAIT1
      7 LISTEN
      1 SYN_RECV
  43187 TIME_WAIT

This is ofcourse undesired. What would be best is that we setup a tunnel or a connection between nginx and varnish ones and all communication happen over that static connection. I am not quite sure how exactly this can be done or if there are better ways to do it.

What else can I do here?

Thanks

I'm configuring a failover server tasked to accept any incoming request, and reply with blank 200 response. The idea is to minimize the reply time and to ensure we dont send any 40x or 50x.

I tried using return 200; for the desired locations within Nginx, but my monitoring systems (Pingdom) didn't like the response, and consider the server not responding.

Is there a better way to do this, of course with minimal overhead on the server?

I am using DNS round robin load balancing and have divided my total traffic onto multiple servers. Each server does around 300-400req/second but I am interested in having an aggregate graph telling me the TOTAL of all requests per second served by our architecture.

Is there any way I can do this. Right now each graph in Munin comes as a separate graph as they depict things on one server.

I am using configuration as follow which doesn't work

doesnt work for me, does this configuration got errors?

[TRAFFIC.AGGREGATED]
  update no
  requests.graph_title nGinx requests
  requests.graph_vlabel nGinx requests per second
  requests.draw LINE2
  requests.graph_args --base 1000
  requests.graph_category nginx
  requests.label req/sec
  requests.type DERIVE
  requests.min 0
  requests.graph_order output
  requests.output.sum \
    lb1.visualwebsiteoptimizer.com:nginx_request_lb1.visualwebsiteoptimizer.com_request.request \
    lb3.visualwebsiteoptimizer.com:nginx_request_lb2.visualwebsiteoptimizer.com_request.request \
    lb3.visualwebsiteoptimizer.com:nginx_request_lb3.visualwebsiteoptimizer.com_request.request

I dont want to build from source and was looking to find some repository where I can simply do an apt-get install php5 and get the latest version. Does anyone know of any reliable source I can use to do the same or is the only option I have to build it myself?

What is the best way to sync two folders completely in real time over two servers (Ubuntu) in same network? Do you think syncing folders is a better idea than having a shared folder? If a shared folder is better, how can I set this up?

I am using munin on a system which is adding a log entry in syslog everytime the munin-node cron job executes. It is not an issue but it sometimes makes other errors spotting difficult. There are entries like

Feb 28 07:05:01 li235-57 CRON[2634]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi)

every 5 minutes and I was wondering how can I stop the messages going into syslog. For munin specific errors I anyways have to keep an eye on /var/log/munin/*

Thanks Sparsh

I have a nginx reverse proxy. The server is close to serving 600-700 requests per second. I have a Munin HTTP load time plugin which is outputting this:

http://monitor.wingify.com/munin/visualwebsiteoptimizer.com/lb1.visualwebsiteoptimizer.com-http_loadtime.html

Now, the problem is I am seeing some spikes in the graph. Expected response times should always be under 200ms. I am keeping an eye on syslog and messages but I am unable to figure out the actual cause of this. I was wondering if there is any good HTTP response time profiling system which I can install / embed with this nginx server and get a detailed reports / logs on the breakup of time taken by different things and what exactly is the cause of the spikes.

The profiling system would also help me understand bottlenecks and how can I further optimize the latency.

Most important right now is to investigate the cause of the spikes in the HTTP load time graphs (similar pattern is reported by external monitors - Pingdom) and to fix it to get consistent response times

Thanks

I recently installed CSF firewall and have made live a new server which is accepting around 600req/second.

Its basically a reverse proxy and I found in pingdom and munin that for a particular time, the response times of the server increased by 3 folds. I looked into logs and exactly at that point, I found loads of entries like this:

Feb 27 15:22:09 li235-57 dhclient: DHCPREQUEST of <My IP address> on eth0 to 207.192.68.72 port 67
Feb 27 15:22:09 li235-57 dhclient: send_packet: Operation not permitted

Can you help me understand what went wrong and what exactly happened? Is there any settings I should change in CSF or any pointer to debug this further?

I just installed CSF firewall on my server and I have started seeing the following entries in my log file. Can some one help me understanding as to what is happening

Thanks

Feb 25 10:45:19 li235-57 kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=fe:fd:ad:ff:e1:39:88:43:e1:7c:75:3f:08:00 SRC=81.4.153.90 DST=173.255.225.57 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=58295 DF PROTO=TCP SPT=43639 DPT=4899 WINDOW=65535 RES=0x00 SYN URGP=0 
Feb 25 10:45:22 li235-57 kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=fe:fd:ad:ff:e1:39:88:43:e1:7c:75:3f:08:00 SRC=81.4.153.90 DST=173.255.225.57 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=58483 DF PROTO=TCP SPT=43639 DPT=4899 WINDOW=65535 RES=0x00 SYN URGP=0 
Feb 25 10:55:51 li235-57 kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=fe:fd:ad:ff:e1:39:88:43:e1:7c:75:3f:08:00 SRC=82.178.69.36 DST=173.255.225.57 LEN=50 TOS=0x00 PREC=0x00 TTL=108 ID=3839 DF PROTO=UDP SPT=52904 DPT=80 LEN=30 
Feb 25 10:58:04 li235-57 kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=fe:fd:ad:ff:e1:39:88:43:e1:7c:75:3f:08:00 SRC=82.178.182.44 DST=173.255.225.57 LEN=55 TOS=0x00 PREC=0x00 TTL=47 ID=54421 DF PROTO=UDP SPT=53915 DPT=80 LEN=35 

I recently had a server downtime. I looked everywhere and the only thing I found in my log files is:

Feb 17 18:58:04 localhost kernel: possible SYN flooding on port 80. Sending cookies.
Feb 17 18:59:33 localhost kernel: possible SYN flooding on port 80. Sending cookies.

Can someone give me more information about it. WHat is it, How can I debug the cause and how can I fix the same. I also posted ipconntrack suddenly became toooo large which has another data point I found unusual, wondering if the two things is connected as they occured exactly at the same time but at different servers. One at reverse proxy and other at actual backend Varnish server)

Thanks

I was wondering if some one can point to some good documentation which can give me detailed information about some kernel variable. I am right now mostly interested in the following variables / parameters and how it effects the system and how to decide their optimum values

net.ipv4.conf.all.send_redirects = 0
net.ipv4.ip_conntrack_max = 231072
net.ipv4.netfilter.ip_conntrack_tcp_max_retrans = 3
net.ipv4.netfilter.ip_conntrack_tcp_be_liberal = 0
net.ipv4.netfilter.ip_conntrack_tcp_loose = 3
net.ipv4.netfilter.ip_conntrack_tcp_timeout_max_retrans = 300
net.ipv4.netfilter.ip_conntrack_log_invalid = 0
net.ipv4.netfilter.ip_conntrack_generic_timeout = 600
net.ipv4.netfilter.ip_conntrack_icmp_timeout = 30
net.ipv4.netfilter.ip_conntrack_udp_timeout_stream = 180
net.ipv4.netfilter.ip_conntrack_udp_timeout = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close = 10
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 120
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 432000
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 60
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 120
net.ipv4.netfilter.ip_conntrack_checksum = 1
net.ipv4.netfilter.ip_conntrack_buckets = 8192
fs.file-max = 98304

Thanks a lot for the help

I have asked a few questions here and got some good replies. I am still facing some issues with ipconntrac and connections though firewall.

http://monitor.wingify.com/munin/visualwebsiteoptimizer.com/lb1.visualwebsiteoptimizer.com.html#Network

I am looking at some guidance and steps I should follow on ubuntu 10.10 64bit machine that will ensure the issues with ipconntrack and time_wait reduces and the server starts performing better and more reliabily.

I have a nginx server with load balancing and reverse proxy. Right now its behing another nginx but very soon I plan to make it front, where it will receive TCP connections from clients directly at a rate of 500req/second

I am having some big troubles with the server. I have pasted my configurations here and I am kinda sure that the problem is with ipconntrac and similar things which are alient to me

http://paste.org/pastebin/view/28543

root@load_balancer:/proc/sys/net/ipv4# netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c
     67 CLOSING
    727 ESTABLISHED
    173 FIN_WAIT1
    183 FIN_WAIT2
     19 LAST_ACK
      5 LISTEN
    447 SYN_RECV
      1 SYN_SENT
  27970 TIME_WAIT

Its a ubuntu machine with mainly nginx (load balancer and reverse proxy) installed.

It surely isnt great. Can you help me understand whats going on and how can I fix it. This is my live server and I am sure its in a bad shape right now. Any document or commands to fix this, or settings I should make to make this better and reduce time wait and fin_wait1/2 better would be awesome.

I have two Varnish servers and I plan to add more varnish servers. I am using a nginx load balancer to divide traffic to these varnish servers. To utilize maximum RAM of each varnish server, I need that same request reaches same varnish server. Same request can be identified by one GET parameter in the request URL say 'a'

In a normal code, I would do something like- (if I need to divide all traffic between 2 Varnish servers)

if($arg_a % 2 == 0) {
    proxy_pass varnish1;
}

if($arg_a % 2 == 1) {
    proxy_pass varnish2;
}

This is basically doing a even / odd check on GET parameter a and then deciding which upstream pool to send the request. My question are-

1. What is the nginx equivalent of such a code. I dont know if nginx accepts modulas
2. Is there a better/ efficient hashing function built in with nginx (0.8.54) which I can possibly use. In future I want to add more upstream pools so I need not to change %2 to %3 %4 and so on
3. Any other alternate way to solve this problem

I have a nginx server with 5 backend servers. We serve around 400-500 requests/second. I have started getting a large number of Upstream Timed out errors (110: Connection timed out)

Error string in error.log looks like

2011/01/10 21:59:46 [error] 1153#0: *1699246778 upstream timed out (110: Connection timed out) while reading response header from upstream, client: {IP}, server: {domain}, request: "GET {URL} HTTP/1.1", upstream: "http://{backend_server}:80/{url}", host: "{domain}", referrer: "{referrer}"

Any suggestions how to debug such errors. I am unable to find a munin plugin to keep a check on number of upstream errors. Sometime the number of errors per day is way too high and somedays its a more decent 3 digit number. A munin graph would probably help us finding out any pattern or correlation with anything else

How can we make the number of such error as ZERO