Home / server / Questions / 408099
Accepted
Bojan Hrnkas
Asked: 2012-07-17 03:32:20 +0800 CST

Server not responding, after reset, all data rolled few months back

  • 772

A Debian Server stopped responding earlier today. After hardware reset, the data on server was rolled back several months ago to the state it was in february. How is this possible?Everything, logs, webdata, databases, are as if the computer has been turned off on 03. February and turnd on today.

Does anyone know what could cause this? Hacked? How can I prevent this from happening again?

Thanks a lot!

EDIT:

Here is the update what really happened.

On the Server there is a Mirror RAID configuration. One of the HDDs stopped writing data on February 3rd. When second HDD broke earlier today, the system could only boot from the first one. That is why all the data was that old.

Apparently, this is one in the million situation.

EDIT:

So you don't get into same situation as me:
http://www.tcpdump.com/kb/os/linux/raid-alerting.html

debian hacking data-loss

3 Answers

  1. Best Answer

    Here is the update what really happened.

    On the Server there is a Mirror RAID configuration. One of the HDDs stopped writing data on February 3rd. When second HDD broke earlier today, the system could only boot from the first one. That is why all the data was that old.

    So you don't get into same situation as me: http://www.tcpdump.com/kb/os/linux/raid-alerting.html

    • 2

  2. if you're using unionfs to overlay changes and that overlay died or was reset, you'd end up "winding the clock back", but that would be quite unusual.

    • 0

  3. The mdadm package comes with its own scripts to monitor the raid status daily and send an email upon failure:

    /etc/cron.daily/mdadm

    # cron.daily/mdadm -- daily check that MD devices are functional
(..)
# As recommended by the manpage, run
#      mdadm --monitor --scan --oneshot
# every day to ensure that any degraded MD devices don't go unnoticed.
# Email will go to the address specified in /etc/mdadm/mdadm.conf

    Normally the user that gets the emails is configured as root in /etc/mdadm/mdadm.conf so in order to send the emails to a different address, instead of the local root account, add an entry like this to /etc/aliases:

    root: [email protected]

    Suffice it to say your MTA such as exim should be configure to be able to send out email.

    • 0