Home / server / Questions / 12356
Accepted
Bob
Asked: 2009-05-27 12:13:51 +0800 CST

Verifying Integrity of Data

  • 772

I just set up a FreeNAS server with a terabyte drive. I want to only have one hard drive in each machine so I have been taking as much data as possible and sending it over the LAN to FreeNAS. I have noticed at least one file didn't copy properly and is now corrupt. (I am also noticing some strange permission issues, but that is another question.) Now that most of the data is over on the FreeNAS server, is there an automated way of verifying nothing else is corrupt?

I am not exactly sure how to describe how the file was corrupt. Basically it appeared to be a 178 megabyte video file, but when accessing it to play or even move, the windows machine accessing it gave a generic could not access error message. I used FreeNAS's web copy interface to move the file, once it was moved, the file was 76 megs, and could not be played.

linux corruption truenas

2 Answers

  1. Best Answer

    Always run:

    cd /filesystem; \
find . -type f -exec md5sum {} \; >& /filesystem-md5.log

    and then

    cd /filesystem-new; \
md5sum -c /filesystem-md5.log

    before and after copying a large amount of data.

    You'll be surprised how much random data corruption you experience in the real world.

    When you find a corrupt file, cmp -l badfile goodfile to attempt to understand the nature of the corruption.

    This is why I beg for end-to-end integrity checking in all cases. Unfortunately filesystem and OS vendors do not take this seriously.

    • 10

  2. You can check Aide. I guess there's other integrity tools out there.

    It creates a database from the regular expression rules that it finds from the config file. Once this database is initialized it can be used to verify the integrity of the files. It has several message digest algorithms (md5,sha1,rmd160,tiger,haval,etc.) that are used to check the integrity of the file. More algorithms can be added with relative ease. All of the usual file attributes can also be checked for inconsistencies. It can read databases from older or newer versions. See the manual pages within the distribution for further info. There is also a beginning of a manual.

    • 1