Home / server / Questions / 95328
Accepted
Keith Palmer Jr.
Asked: 2009-12-18 06:06:07 +0800 CST

FreeBSD RAID rebuild by data center, now data in /var/logs and /var/db/mysql is *gone*, OH NO!

  • 772

OK, our data center did a RAID rebuild (and possibly some other stuff) on our server overnight. Something failed at some point, I checked on things this morning, and it appears as if the /var partition has been entirely wiped and brought back to a default install state.

i.e., everything we had in /var/logs is gone and everything we had in /var/db/mysql is gone, it's as if that stuff has been wiped out and replaced by a default install of FreeBSD (i.e. minimal logs, no stored .gzip logs, no more databases, etc.) BAD, REALLY BAD!

Actually, looking at it more, before if I did a "df -h" I'd see a /var partition. Now, I don't see it anymore. But I can get to /var... it just doesn't show up in "df -h". I do see an entry in /etc/fstab for /var.

I have pretty much no idea what they did. I'm waiting to hear from them. So I realize that this is a somewhat unanswerable question. I'm just looking for ideas on what I can check/what might have happened, etc. until I get more details from the data center about what they actually did.

In the meantime, can anyone propose a theory for where the data in /var might have gone... and how I can check if it still exists somewhere?

EDIT We're on the right track, thanks guys!

[kp@afekan /var/log]$ sudo tail -n 10 /var/log/dmesg.today 

SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
Trying to mount root from ufs:/dev/mfid0s1a
WARNING: /tmp was not properly dismounted
/tmp: mount pending error: blocks 24 files 0
WARNING: /usr was not properly dismounted
WARNING: R/W mount of /var denied.  Filesystem is not clean - run fsck
WARNING: R/W mount of /var denied.  Filesystem is not clean - run fsck
bce0: link state changed to UP
bce1: link state changed to UP

They're working on it!

FIXED! Problem solved, /var never got mounted, needed to be fsck'd. Thanks guys!

raid

2 Answers

  1. Best Answer

    Your /var partition isn't mounted. Probably there was an old, default install in /var; later on they might have added another volume and mounted it over.

    Do ask them to check if every volume has been properly mounted; check /etc/fstab for the exact name/slice.

    Also, it is also very likely that your current var partition has been created at system startup by the various daemons which, since they could not find their own directories, decided to create /var/log, /var/tmp, etc.

    Say hi to the folks in the data centre :P

    • 8

  2. Please show the output of the following three commands:

    # cat /etc/fstab
# mount 
# df -kh

    It's possible that they have not mounted /var from /etc/fstab for whatever reason and it's living on your root directory.

    • 4