Debian Linux server hangs after a week or so

Do the servers really hang or are they just unreachable by ping?

Install a monitoring tool such as Munin (or similar) which will show you graphs of not just the CPU load but also memory stats, disk usage, and various other bits and pieces - you can configure it to monitor lots of aspects. Nex time the server hangs, check the graphs for any unusual signs. You will learn to see what a normal graph looks like so anything out of the ordinary is suspicious (although not necessarily wrong).

Are you sure you are checking all server logs? ie do you have web/mail/ftp/dns/other servers? check all such logs! Don't forget to enable debug logging while troubleshooting.

If the server crashes every week or so it could be something that happens regularly, ie a cron job, or log rotation, stuff like that.

Make sure you get all system emails (root alias). You can even install OSSEC which is a great tool for keeping an eye on the logs and getting emails when things go wrong. But this OSSEC tool is just an automated way of looking at the logs, so nothing magical.

Networking issues? dhcp lease expired?

Please show relevant content of /var/log/messages and/or /var/log/kern.log it's possible the kernel logged some crash reports or something else that could shed some light. When I experienced such unexplained hangs it was due to a bad driver, because logging isn't very verbose I wasn't able to find out the exact driver.

In my case there were soft lockups (kernel: [XXXX] BUG: soft lockup - CPU#X). After some research I found http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=556030 and the last comment provided some insight and a way to make logging more verbose. It's an easy kernel modification but if you don't feel comfortable compiling your own kernel it may not be the best thing to do.

Just updating the kernel or installing a newer version and rebooting may fix the problem.

Quoting:

We extensively researched the problem.

The TLB flush softlockup is only a CONSEQUENCE of a deadlock.

Background: The TLB flush is issued by a CPU to a number of other CPUs using inter-processor interupts to progagate paging changes. Then the issuing CPU loops until all processor acknowledge the change. If such processor is in deadlock on a spinlock, this never hapens, then the softlockup triggers. The deadlock arise on a spinlock, this lock may be held by user code sometimes (through /proc or /sys interfaces of modules).

The only way to identify the root cause (i.e. which driver is causing problems) is to dump ALL CPU stacks in the soft lockup code.

One way to do that is to modifiy the kernel and add

            arch_trigger_all_cpu_backtrace() 

in the

            kernel/softlockup.c:softlockup_tick() 

function.

This is based on NMI IPI which ensure all stacks are dump, even in the case of deadlock (well don't expect the impossible to happen either).

You should easily find the faulty driver and post the relevant bug.

Apparently the problem was related to some python scripts that caused the server to hang. I don't understand why they hanged the server but at least they don't hang it any more.