One of our servers recently experienced some file system corruption and our root file system was automatically remounted as read only. The steps I took to recover were:
- attempted to
remount > mount -n -o remount /this failed - rebooted the server
- was prompted to perform a manual
fsck, there were 5 orphaned inodes which required fixing.
After performing these steps I was able to gain access and the file system was writable again. Unfortunately I don't have any informative logs as none were written or I would have included these.
One cause that was been suggested was that our database was too busy to write the data to disk properly and this caused the issue, the high level of cache memory was given as an indication that this might be the case. However I'm not sure about this as although cache is high we aren't using the swap at all (output of free below).
$ free -m
total used free shared buffers cached
Mem: 2041 1879 162 0 62 1599
-/+ buffers/cache: 216 1825
Swap: 471 0 471
Is there any way I can diagnose the fault after it has happened? Does MySQL look like a likely candidate?
If not are there any steps I should take in the future if this happens again?
Orphaned inodes are benign and perfectly normal whenever you have an unclean dismount. They are simply files that had been deleted, but were still open when the fs was remounted read only. They are not the cause, but merely a symptom. You need to check your kernel logs to see what the actual problem was that caused the read only remount. You also might want to run some SMART diagnostics to make sure the drive isn't failing.
First sanity check your server:
A high level of cache is desirable and shouldn't corrupt your file-system in any way.