Home / server / Questions / 245530
In Process
Low Kian Seong
Asked: 2011-03-10 20:34:54 +0800 CST

How can I see logs in a server after a kernel panic hang?

  • 772

I am running a production gentoo Linux machine, and recently there was a situation where the server hung in my co-located premises and when I got there I noticed that the server was hung on what appeared to be a kernel panic hang. I rebooted the machine with a hard reboot and was disappointed to find out that I could not find a shred of evidence anywhere on why the machine hung.

Is it true that when I do a hard reboot the messages itself will get lost or is there a setting I can do somewhere say in syslog-ng or maybe in sysctl to at least preserve the error log so that I can prevent such mishaps from happening in the future ? I am running a 2.6.x kernel by the way.

Thanks in advance.

linux linux-kernel kernel-panic

4 Answers

  1. To log a kernel panic, using netconsole is way easier than using a serial link ;)

    There is a useful entry about it on ubuntu wiki that can be used on other linux flavors... Helped me for instance with archlinux.

    Note that netconsole can be also be setup at boot.

    • 5

  2. Generally in these situations I've set up a remote serial console and then made sure to log everything that appears on the console. You can do this by connecting the serial port of the machine to another machine via an rs-232 cable. This might be getting a little tricker these days as serial ports are gradually disappearing, but I think most servers still have them.

    Then on that other machine, run conserver and configure console logging. This provides a good way to both log everything on the console and provide an alternate login path to the system.

    If you don't want to set up and adminster another linux box, an alternative is to install a serial console server. Various companies such as Avocent, Cyclades, and Raritan make these boxes which provide a network interface to 2-48 serial ports. However this sort of thing will be more expensive than just setting up another linux box of course.

    • 4

  3. You can try replicating syslog messages to another machine. Maybe not all of them but only alerts and critical and disable caching for them.

    • 0

  4. I think there are three possible solutions for that:

    1. Configure kdump that creates a dump when the kernel panics (use "magic sysrq c" to force a panic). Probably the best solution if you really have a kernel panic.
    2. Configure a serial console (nowadays maybe netconsole). Good to see the famous "last words".
    3. Configure an external syslog server that receives the kernel messages. However take care that messages are send immediately (without delay). If the network stack crashes, the important messages might get lost, however.
    • 0