I have a strange behaviour in my server :-/. Is a OpenVZ VPS (I think is OpenVZ, because /proc/user_beancounters exists and df -h returns /dev/simfs drive. Also ifconfig returns venet0). When I do cat /proc/stat, I can see how each second about 50-100 processes are created and happens about 800k-1200k context switches! All that info is with the server completely idle, no traffic nor programs running.
Top shows 0 load average and 100% idle CPU.
I've closed all non-needed services (httpd, mysqld, sendmail, nagios, named...) and the problem still happens. I do ps -ALf each second too and I don't see any changes, only a new ps process is created each time and the PID is just the same as before + 1, so new processes are not created, so I thought that process growing in cat /proc/stat must be threads (Yes, seems that processes in /proc/stat counts threads creation too as this states: http://webcache.googleusercontent.com/search?q=cache:8NLgzKEzHQQJ:www.linuxhowtos.org/System/procstat.htm&hl=es&tbo=d&gl=es&strip=1).
I've changed to /proc dir and done cat [PID]\status with all PIDs listed with ls (Including kernel ones) and in any process voluntary_ctxt_switches nor nonvoluntary_ctxt_switches are growing at the same speed as cat /proc/stat does (just a few tens/second), Threads keeps the same also.
I've done strace -p PID to all process too so I can see if any process is crating threads or something but the only process that has a bit of movement is ssh and that movement is read/write operations because of the data is sending to my terminal.
After that, I've done vmstat -s and saw that forks is growing at the same speed processes in /proc/stat does. As http://linux.die.net/man/2/fork says, each fork() creates a new PID but my server PID is not growing!
The last thing I can think of is that all process data that proc/stat and vmstat -s show is shared with all the other VPS stored in the same machine, but I don't know if that is correct... If someone can throw some light on this I would be really grateful.
In your case I would assume that your kernel is not virtualizing some or all values in /proc/stat, so what you're seeing should be caused by activity in other containers or in the hardware node itself.
You haven't mentioned your specific kernel version so we can't check for sure, but a similar issue was discussed in http://forum.openvz.org/index.php?t=msg&goto=44507& .
PS: check out /proc/vz/vestat instead.
I have seen at least one similar question here about someone with 2 identical ubuntu servers behind a load balancer...and I will tell you what I told him:
syspro ("system profiler")
or try stuffing some limits in /etc/security/limits.conf Apply the limits to both machines and compare which executables show up. It will at least flag a process or thread and narrow the list of suspects.
man limits.conf: hard & soft for file descriptors, mem locked, stack size,cpu time, nice levels, etc