UpTheCreek's questions

I have a (node.js) application which is occasionally causing 100% CPU usage. I've attached to the process with strace when it's been in this state - but I don't know what to make if the strace output. The results repeat between these two patterns (shortened) :

mmap(0x30c3ac700000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x30c3ac700000
mmap(0x3364514ba000, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x3364514ba000
munmap(0x3364514ba000, 286720)          = 0
munmap(0x336451600000, 761856)          = 0
mmap(0x336451500000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x336451500000
mmap(0x2b9c33880000, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x2b9c33880000
munmap(0x2b9c33880000, 524288)          = 0
munmap(0x2b9c33a00000, 524288)          = 0
...

And...

munmap(0x2b9c33900000, 1048576)         = 0
munmap(0x336451500000, 1048576)         = 0
munmap(0x30c3ac700000, 1048576)         = 0
munmap(0x247e37500000, 1048576)         = 0
munmap(0x20d76c800000, 1048576)         = 0
munmap(0x1cae0d600000, 1048576)         = 0
munmap(0x163545100000, 1048576)         = 0
munmap(0x32dcfe700000, 1048576)         = 0
munmap(0x1a1feff00000, 1048576)         = 0
munmap(0x3fb72f00000, 1048576)          = 0
munmap(0x366536900000, 1048576)         = 0
...

Can anyone shed any light as to what might be going on here? Thank you!

I discovered recently that you can match a backend dynamically, based on the request hostheader, like this:

 use_backend %[req.hdr(host),lower]

However, does anyone know any way I could use the subdomain of the request hostheader to match the backend?

E.g. something along these lines:

backend one
backend two

use_backend %[<SUBDOMAIN OF HOSTHEADER>,lower]

which would match like this:

 one.example.com -> backend one
 two.example.com -> backend two

I have a deep directory structure, with a large number of files (about 1M). I'd like to execute a command against each. (In my case the files are pngs which I'd like to run optipng against.)

I've tried this:

find . -name *.png | xargs sudo optipng -o2

but am getting the error argument list too long.

I'm assuming that xargs should be able to handle this, and that there must be a problem with my syntax.

We have a virtual server with an NFS share that is written to by other servers. Today the NFS share became inaccessible. The syslog was full off messages like this:

RPC: fragment too large: 311176

I've seached on google but can't find much information about this. Could someone explain what this means?

On one of our busy (virtual) Debian 6 servers we are seeing a lot of messages in the syslog like this:

kernel: xen_netfront: xennet: skb rides the rocket: 19 slots

We are seeing some connectivity problems, and I believe it's possibly related to this. I've tried researching the issue, but can find very little information (but have confirmed that it can cause packet loss). I'm hoping that someone on here knows more about this issue.

What might be causing this issue?

Could this be a problem with the virtual host? (and why is Xen mentioned here)

What can I do to stop this happening?

Thank you.

We have a number of Xen virtualised servers, all running Debian 6 64bit. We are having an intermittent problem where occasionally a server will stop responding over the network. When this happens we can't ping the server, and our app logs indicate that it is unable to connect to other servers on the network.

This has happened to a few different unrelated servers now, and the only common factors are the VPS host and associated infrastructure, the OS, and our OS settings. I'm following this up with the host, but really need to get to the bottom of it.

I don't really have much to go on at the moment. The only os log entries that I can find that co-incide with the event is one line in the syslog:

Nov 21 19:36:10 xxxxxx ntpd[2460]: xxxx:4f8:xxx:xxx:1:2:3:4 interface xxxx:7e00::xxxx:91ff:xxxx:1bd4 -> (null)

However I think that is a result of the network connection dying, rather than a clue to its cause.

MTR reports from a working server show nothing useful.

So, how should I go about trying to understand what's happening here? Are there any network specific logs that I don't know about which should be checking?

Thank you!

I recently checked one of our redis processes to what ulimits where applied using:

cat /proc/<redis-pid>/limits

And was suprised to learn that is was at the low default value:

Limit                     Soft Limit           Hard Limit           
Max open files            4016                 4016 

I was suprised, because we have the following configured:

# /etc/sysctl.conf 
fs.file-max = 100000

.

# /etc/security/limits.conf
* soft nofile 100000
* hard nofile 100000

.

# /etc/ssh/sshd_config
UsePAM yes

.

# /etc/pam.d/sshd
session required pam_limits.so

Can anyone tell me why the increased ulimit is not being applied to the running redis process?

The redis process is running as the user 'redis', the server has been rebooted since the limits were increased. We are on Debian Squeeze.

We recently had a problem with one of our servers (Debian Squeeze) becoming unresponsive during heavy-ish load. Looking at the kernel logs, I think this is the cause:

kernel: nf_conntrack: table full, dropping packet

As I understand it, this is the conntrack module, which does some stateful tracking of connection, reporting that the table used to store the connection details is full.

From the research I have done, there seem to be two ways to mitigate this:

1. Increase size of the table.

2. Remove the module from the system altogether.

However, neither /proc/sys/net/ipv4/ip_conntrack_maxnor /proc/sys/net/ipv4/netfilter/ip_conntrack_max exist on this machine (there is no ipv4 catalogue under net).

If I do lsmod I get no results.

So, I'm a bit confused - perhaps someone could clarify the situation for me?

• Is conntrack installed? If so, where are the settings? And why doesn't it show up in lsmod?
• If conntrack is not installed, what is issuing the table full messages?

Thank you

We recently had a spike in traffic, which while only moderate in size, caused haproxy to max out one of the CPU cores (and the server became unresponsive). I'm guessing that I'm doing something inefficiently the config, and so would like to ask all the haproxy experts out there if they would be so kind as to critique my config file below (mainly from a performance perspective).

The config is intended to distribute between a group of http-application servers, a group of servers that handle websockets connections (with a number of seperate processes on different ports), and a static file webserver. It's working well appart from the performance issue. (Some details have been redacted.)

Any guidance you could offer would be much appreciated!

HAProxy v1.4.8

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
        global
                daemon
                maxconn         100000
                log             127.0.0.1 local0 notice
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
        defaults
                log                     global
                mode                    http
                option                  httplog
                option                  httpclose               #http://serverfault.com/a/104782/52811
                timeout connect         5000ms
                timeout client          50000ms
                timeout server          5h                      #long timeouts to stop WS drops - when v1.5 is stable, use 'timeout tunnel';
#---------------------------------------------------------------------
# FRONTEND
#---------------------------------------------------------------------
        frontend public
                bind *:80
                maxconn         100000
                reqidel ^X-Forwarded-For:.*                     #Remove any x-forwarded-for headers
                option forwardfor                               #Set the forwarded for header (needs option httpclose)

                default_backend app
                redirect prefix http://xxxxxxxxxxxxxxxxx code 301 if { hdr(host) -i www.xxxxxxxxxxxxxxxxxxx }
                timeout client  5h                              #long timeouts to stop WS drops - when v1.5 is stable, use 'timeout tunnel';

        # ACLs
        ##########
        acl static_request hdr_beg(host) -i i.
        acl static_request hdr_beg(host) -i static.
        acl static_request path_beg /favicon.ico /robots.txt
        acl test_request hdr_beg(host) -i test.
        acl ws_request hdr_beg(host) -i ws
        # ws11
        acl ws11x1_request hdr_beg(host) -i ws11x1
        acl ws11x2_request hdr_beg(host) -i ws11x2
        acl ws11x3_request hdr_beg(host) -i ws11x3
        acl ws11x4_request hdr_beg(host) -i ws11x4
        acl ws11x5_request hdr_beg(host) -i ws11x5
        acl ws11x6_request hdr_beg(host) -i ws11x6
        # ws12
        acl ws12x1_request hdr_beg(host) -i ws12x1
        acl ws12x2_request hdr_beg(host) -i ws12x2
        acl ws12x3_request hdr_beg(host) -i ws12x3
        acl ws12x4_request hdr_beg(host) -i ws12x4
        acl ws12x5_request hdr_beg(host) -i ws12x5
        acl ws12x6_request hdr_beg(host) -i ws12x6

        # Which backend....
        ###################
        use_backend static   if static_request
        #ws11
        use_backend ws11x1 if ws11x1_request
        use_backend ws11x2 if ws11x2_request
        use_backend ws11x3 if ws11x3_request
        use_backend ws11x4 if ws11x4_request
        use_backend ws11x5 if ws11x5_request
        use_backend ws11x6 if ws11x6_request
        #ws12
        use_backend ws12x1 if ws12x1_request
        use_backend ws12x2 if ws12x2_request
        use_backend ws12x3 if ws12x3_request
        use_backend ws12x4 if ws12x4_request
        use_backend ws12x5 if ws12x5_request
        use_backend ws12x6 if ws12x6_request
#---------------------------------------------------------------------
# BACKEND - APP
#---------------------------------------------------------------------
        backend app
                timeout server          50000ms #To counter the WS default
                mode http
                balance roundrobin
                option httpchk HEAD /upchk.txt
                server app1 app1:8000             maxconn 100000 check
                server app2 app2:8000             maxconn 100000 check
                server app3 app3:8000             maxconn 100000 check
                server app4 app4:8000             maxconn 100000 check
#---------------------------------------------------------------------
# BACKENDs - WS
#---------------------------------------------------------------------
#Server ws11
        backend ws11x1
                server ws11 ws11:8001 maxconn 100000
        backend ws11x2
                server ws11 ws11:8002 maxconn 100000
        backend ws11x3
                server ws11 ws11:8003 maxconn 100000
        backend ws11x4
                server ws11 ws11:8004 maxconn 100000
        backend ws11x5
                server ws11 ws11:8005 maxconn 100000
        backend ws11x6
                server ws11 ws11:8006 maxconn 100000
#Server ws12
        backend ws12x1
                server ws12 ws12:8001 maxconn 100000
        backend ws12x2
                server ws12 ws12:8002 maxconn 100000
        backend ws12x3
                server ws12 ws12:8003 maxconn 100000
        backend ws12x4
                server ws12 ws12:8004 maxconn 100000
        backend ws12x5
                server ws12 ws12:8005 maxconn 100000
        backend ws12x6
                server ws12 ws12:8006 maxconn 100000
#---------------------------------------------------------------------
# BACKEND - STATIC
#---------------------------------------------------------------------
        backend static
                server static1 static1:80   maxconn 40000

I've been running mongodb as a replica set, and have recently attempted to move to a sharded cluster.

One thing I'm a little unsure about is how I should be starting the mongos process. Previously I had used the 10gen rpms when installing mongodb, which installed an init.d script for the mongod service. I'd like to have mongos run in a similar way, but am unsure about how I should be doing this really. Is it a case of writing a custom init script?

I'm on Debian 6, and Mongodb 2.4

[I'd previously posted this quesion in the mongodb google group, but am copying here as I didn't get any response - I'll add any responses here too]

I need to set up some VPSs for serving static content (many small files). I plan on using Nginx for this, and would like to set it up so that we are able to scale out relatively easily. The requirements are:

• Many files (at least hundreds of thousands).
• Small file sizes (less than 10KB).
• Files are being added constantly by an application on neighbouring servers.
• New files must be immediately available to all Nginx servers.

My current plan is:

• Have a 'master' server with an NFS share containing all of the files.
• Application producing new files interacts only with master.
• Have multiple Nginx servers mounting this NFS share.
• Load balance across Nginx instances.

One obvious problem with this is that the 'master' server is a single point of failure (any remedy for this?). Are there other potential issues which I have overlooked? Are there elements here which will not scale well in this way? Would anyone suggest an alternative approach?

Regarding memory requirements, I'm assuming I should give each Nginx server as much as possible so that hot files can be cached (by OS? Nginx?) an not have to be reqested form the NFS share constantly.

Lastly, am I crazy not to use a CDN?

Is there an easy way to dissable All backends in haproxy, and instead serve an appropriate maintenance page (for http requests)?

I've read a little about the dissabled option, which I understand is per-server, but am wondering if there is a way to simply stop traffic to all backends?

Can anyone with experience of deploying CPU intensive applications into 'cloud' environments (like e.g. Linode), comment on whether these environments have been suitable?

It seems to me that with VPS/Cloud hosting, specifications like RAM, Disk space, network, are very well defined, whereas CPU power is rather more of a grey area. And when you need to move to a more powerful machine, you often seem to have to up everything (memory, disk space etc.) - which quickly gets expensive...

So, are these environments appropriate, our would dedicated servers be the more cost effective route?

EDIT:

To clarify, I'm talking about constantly high CPU usage patterns.

I'm interested in modelling various server/network configurations for a web application. I'd like to know in advance which parts of the system are going to be bottlenecks, and whether the bottlenecks are bound by CPU/Memory/Network etc.

One think I've been thinking about is taking a single test server, and setting up each 'real' server as a virtual machine on this, configured as they would be in the wild. I'm going to try this, but wanted to ask the serverfault comunity if anyone has tried this approach before. Is it viable?

I'm not expecting benchmarks, or anything like that of course, but am thinking it might be useful for modeling relative performance, highlighing bottlenecks, and providing a sanity check on the architecture.

What does Virtual Bytes mean in Worker Processes pane in IIS7.5 Manager?

If I open IIS Manager (On WindowsServer 2008r2), and go to the Worker Processes page, I can see that one of my AppPool processes reports

Private Bytes (KB) 106,435.00
Virtual Bytes (KB) 748,788.00

However, if I look at task manager, the associated w3wp.exe process reports 69,276K.

Can anyone explain to me how these figures relate to each other?

Thanks

Does anyone know of a way (specific to 7zip or general) to limit the CPU usage of 7zip while it is archiving a long list of files? OS is Windows Server 2008 R2, and 7zip is run via command line from a batch file.

By default MySQL InnoDB stores all tables of all DBs in one global file. You can change this by setting innodb_file_per_table in the config, which then creates one data file for each table.

I am wondering why innodb_file_per_table is not enabled by default. Are there downsides to using it?

What's the best way to do a 301 redirect from non-www 'somedomain.com' to 'www.somedomain.com' in IIS7.5?

Thank you

What actions should be taken to 'lock-down'/harden/secure a windows server 2008 R2 machine, when used as a web server (with remote desktop access)?

Are there any elements that are enabled as standard that can be dissabled/removed?

Thank you.

If I have a pfsense based firewall in front of my servers, can I use this also for DNS?

Thanks