alfish's questions

Sometimes on a busy Pressflow site I notice a timeout for unauthenticated users while the server's port is far from being saturated, there is plenty of unused memory and the server load is very low.

In addition to that, the bandwidth usage of the site has dropped by about %30 since I moved to the new Varnish-powered server with new configurations. Hence I doubt the drop might be because of Varnish's mal-configurations.

Here are some relevant parameters:

DAEMON_OPTS="-a :80 \
              -T localhost:6082 \
              -b localhost:8080 \
              -u varnish -g varnish \
              -S /etc/varnish/secret \
               -p thread_pools=14 \
               -p thread_pool_min=200  \
               -p thread_pool_max=5000 \
               -p listen_depth=128 \
               -p thread_pool_add_delay=2\
               -p lru_interval=20 \
               -h classic,72227  \
               -p session_linger=120 \
               -p sess_workspace=32768 \
               -p connect_timeout=600 \
               -s malloc,15G"

# And here is the latest output of varnishstat:

1+21:07:54 
Hitrate ratio:       10       13       13
Hitrate avg:     0.8405   0.8408   0.8408

    21003013       158.92       129.32 Client connections accepted
    35390015       283.85       217.91 Client requests received
     4696051        26.99        28.92 Cache hits
          79         0.00         0.00 Cache hits for pass
      939223         4.00         5.78 Cache misses
    10041628        89.95        61.83 Backend conn. success
         108         0.00         0.00 Backend conn. failures
    20651750       162.92       127.16 Backend conn. reuses
     3202624        22.99        19.72 Backend conn. was closed
    23854404       187.90       146.88 Backend conn. recycles
       17397         0.00         0.11 Fetch head
    20499527       189.90       126.22 Fetch with Length
     6536900        37.98        40.25 Fetch chunked
     2410249        15.99        14.84 Fetch wanted close
           4         0.00         0.00 Fetch failed
        2261          .            .   N struct sess_mem
        1728          .            .   N struct sess
      263703          .            .   N struct object
      264848          .            .   N struct objectcore
      138282          .            .   N struct objecthead
         116          .            .   N struct vbe_conn
        2800          .            .   N worker threads
        2801         0.00         0.02 N worker threads created
        2889         0.00         0.02 N worker threads limited
         832         0.00         0.01 N overflowed work requests
           1          .            .   N backends
      658946          .            .   N expired objects
        6948          .            .   N LRU nuked objects
     3425448          .            .   N LRU moved objects
    31922006       257.87       196.56 Objects sent with write
    21002526       160.92       129.32 Total Sessions
    35390015       283.85       217.91 Total Requests
         750         0.00         0.00 Total pipe
    29752808       252.87       183.20 Total pass

So here are my question: How should I know whether Varnish threads have been saturated? How should I manipulate the parameters to increase the throughput? Thanks

While I could use memcached on Debian to use the default 11211 port, but I've had great difficulty setting up Unix socket.

From what I've read, I know that I need to create a memcache.socket and add:

-s /path/to/memcache.socket
-a 0766

To /etc/memcached.conf and comment out the default connection port and IP, i.e.

-p 11211 
-l 127.0.0.1

However, when I restart memcached I get internal server errors on Drupal site.

I'm trying to implement unix sockets to avoid TCP/IP overhead and boost overall memcached performance, however not sure how much performance gain one can expect of this tweak.

I appreciate your hints or possibly configs to to resolve this.

Here is the problem: I have two domains which have pointed to two available IPs on a server. Say:

siteone.com pointed to 1.2.3.4 sitetwo.com pointed to 1.2.3.5

What I did to configure nginx vhosts was to put

server {
        listen    site1.com:80; # 
        server_name   site1.com www.site1.com;

on vhost1 file and

server {
        listen    site2.com:80; # 
        server_name   site2.com www.site2.com;

on a (separate) vhost2 file. The two vhost files are exact replicates expect the site1 and site2 paths, of course. And vhost1 is working well on its own.

However after restarting nginx I get this error:

Restarting nginx: the configuration file /etc/nginx/nginx.conf syntax is ok [emerg]: bind() to 1.2.3.5:80 failed (99: Cannot assign requested address) configuration file /etc/nginx/nginx.conf test failed

When I put

server {
        listen   *:80; # 
        server_name   site2.com www.site2.com;

On site2 vhost, the nginx restarts without error but does not render pages on site2 path.

I'm confused and appreciate your help.

In Debian, I have installed memcached (using this guide) to lower the otherwise unmanageable load on mysql database. The database is on a separate server, and memcached and Varnish are on the front server. Is it a potential security hole to leave memcached unprotected by a firewall? If so, how should I secure it? The situation is especially worrisome,as I've received (unproved) reports of cookie thefts on the server. Thanks

I have received an old server which has a xeon E5405 (2GHz) processor. I'd like to buy and add a more powerful cpu (something like Xeon X5492) which has a higher clock speed but the same LGA 771 socket. The aim is to improve the computational power with best bangs for bucks.

So I am am wondering whether it is technically possible to mix the cpus, and, if so, is it a good idea? That is, what are the possible downside of using the mix? Thanks

I am wondering what is the command/utility to have a real-time view of incoming IPs to my server, ideally along with the port and connected.

I am looking for a well tested bash script (or alternative solution) to do so, in order to avoid max_connection to be exhausted. I know that it is fighting the symptoms, but really need such script as a short term solution.

I get this message from MysqlTunner.pl:

join_buffer_size >= 4 M This is not advised

On the other hand, I read in Debian's my.cnf guide about jont_buffer_size that:

This buffer is used for the optimization of full JOINs (JOINs without indexes). Such JOINs are very bad for performance in most cases anyway, but setting this variable to a large value reduces the performance impact. See the "Select_full_join" status variable for a count of full JOINs. Allocated per thread if full join is found

So I'm wondering which one should I believe? Currently I've set join_buffer_size = 64M as part of efforts to cope with scalability problem of a high-traffic site whose queries are not particularly optimized. I appreciate your hints on this.

Thats pretty much the question: I would like to know who owns the files belonging to a deleted user

I see a lot of 'File name too long' errors in apache error log on Debian.

How can I fix this?

I see this a lot in my server's apache logs (coming from different IPs and directed at different pathes). I am wondering if it can be an attack attempt, and in any case, how to deal with it?

In a new Xeon 55XX server with 4xSSD at raid 10 with Debian 6, I have experienced 2 random shut downs within two weeks after the server being built. Looking at bandwidth logs before shut down does not indicate anything unusual. The server load is usually very low (about 1) and it is collocated far away.There seem to be no power outage while the server went down.

I know that I look at /var/log but not sure which logs should I investigate and what should I look for. So Appreciate your hints.

I frequently get this error on random pages when the are like ~+10K connections (netstat -an |grep 80 |wc -l). This happens while there is still +10GB of free memory and server load is less than 3.

My relevant configs:

/etc/php5/fpm/php.ini

memory_limit = 1024M
default_socket_timeout = 120

/etc/php5/fpm/pool.d/www.conf

pm = dynamic 
pm.max_children = 30 
pm.start_servers = 5
pm.min_spare_servers = 2 
pm.max_spare_servers = 5 
;pm.max_requests =20000 #uncommented with any value didn't fixed
request_terminate_timeout = 120s

I also doubled the above vlues (i.e 60 children, 10 start, 10 max servers) with no avail.

/etc/nginx/nginx.conf

worker_connections 4024;
keepalive_timeout 10;

Nginx itself behind Varnish and memcached is enable. page rendering is pretty fast in general.

This seems to be a quite common problem you I could not find any helpful solution or even analysis of the possible causes. So your hints/solutions are greatly appreciated.

I have been using a pair of Intel 320 in RAID 1 on my busy Ubuntu server for about 4 month. While the disk usage never reached 50% of the capacity, but I FEEL that the server has become less responsive, so I'm wondering whether I need to run a trim, an if yes, how to do so? (The Linux kernel is 3.0 after upgrading to Ubuntu 10.10)

Thanks

I am using nginx along with php-fpm to server a busy mysql-based site. Currently I am using keepalive_timeout = 5, however I have no clear idea about how to optimally adjust it. So appreciate your hints.

It used to be my favorite backup transport agent but now I frequently get this result from s3cmd on the very same Ubuntu server/network:

root@server:/home/backups# s3cmd put bkup.tgz s3://mybucket/
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      36864 of 2711541519     0% in    1s    20.95 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      36864 of 2711541519     0% in    1s    23.96 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      28672 of 2711541519     0% in    1s    18.71 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.05)
WARNING: Waiting 9 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      28672 of 2711541519     0% in    1s    18.86 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.25)
WARNING: Waiting 12 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      28672 of 2711541519     0% in    1s    15.79 kB/s  failed
WARNING: Upload failed: /bkup.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=1.25)
WARNING: Waiting 15 sec...
bkup.tgz -> s3://mybucket/bkup.tgz  [1 of 1]
      12288 of 2711541519     0% in    2s     4.78 kB/s  failed
ERROR: Upload of 'bkup.tgz' failed too many times. Skipping that file.

This happens even for files as small as 100MB, so I suppose it's not a size issue. It also happens when I use put with --acl-private flag (s3cmd version 1.0.1)

I appreciate if you suggest some solution or a lightweight alternative to s3cmd.

I am going to sell a Linux server to a random person. For obvious reasons I want to clean up the hard disks so that the current data on the disk can never be retrieved. What is the safest way to do so? OS re-install? rm -rf *? or something else?

The server is colocated and I don't have physical access.

I used this guide to set it up and it was working to server my django app for a while, but after adding a new django module, without touching any of the nginx or uwsgi configs, now the server leads to 502 502 Bad Gateway error. The nginx error.log says:

*1 upstream prematurely closed connection while reading response header from upstream, client: x.x.x.x, server: blabla.com, request: "GET / HTTP/1.1", upstream: "uwsgi://127.0.0.1:4000", host: "x.x.x.x"

I appreciate your hints to resolve this.

I have a couple of Dell Poweredge R200 coloed in the US and thinking to move some of them to Europe. I am wondering if this is possible without modification (regarding 110 vs 230 voltage difference)? I am also not sure what is the amper usage of Dell R200.

Here is the problem: I have a Linux server in Europe serving a busy Drupal site using nginx+php-fpm, I have another Linux server in the US (where a big portion of my visitors are coming from). The second server is heavily under-used. I am wondering how to make use of the second server to deliver my site's static content?