Home / ubuntu / Questions / 439496
Accepted
user2089518
Asked: 2014-03-27 06:09:32 +0800 CST

Recursive Checksumming

  • 772

I've been using the following command line to do recursive checksumming of directories. It seems to get the job done, but still being a newbie I've been wondering, are there any potential problems with doing it this way? Is it possible for this command to miss files or otherwise mess up?

find ./dir/ -type f -exec sha1sum {} \; > files.sha1
command-line

2 Answers

  1. Best Answer

    There's nothing wrong I can see with your approach. You're excluding directories and find will include hidden files by default. Yeah, it's fine.

    But I'll offer you an alternative because that's what I do:

    shopt -s globstar dotglob
sha1sum **

    globstar enables a recursive match for ** and dotglob enables matching hidden files. Between them they expand to all the filenames and sha1sum can parse them all.

    The main problem with this approach is it will pass all the filenames off to sha1sum in one fat pile. While this can be somewhat faster at small loads, it will explode if you have too many filenames. I don't know what the cut-off is.

    • 9

  2. Python script with hashlib and os.walk

    Aside from using find and globstar, python has modules for hashsum calculation and recursive walk through directory tree. Thus, one can write a simple script just as presented below. In fact, this script is pretty much the same as what I've used for this answer with one minor difference.

    This script assumes that you want to recursively walk through the current working directory, so make sure you cd to the desired top directory first.

    I would also recommend that you save it in ~/bin directory and run source ~/.bashrc prior to usage, since that way, you can just type the name of the script on command-line.

    The script gathers all files, including the hidden-ones ( with the leading dot in the filename ).

    Script Source

    #!/usr/bin/env python3
import os
import sys
from hashlib import sha1

def get_sha1sum(file_path):
    sha1sum = sha1()
    with open(file_path, 'rb') as fd:
        data_chunk = fd.read(1024)
        while data_chunk:
              sha1sum.update(data_chunk)
              data_chunk = fd.read(1024)
    return str(sha1sum.hexdigest())

def find_files(treeroot):
    for dir,subdirs,files in os.walk(treeroot):
         for f in files: 
             full_path = os.path.join(dir,f)
             path_sha1sum = get_sha1sum( full_path  )
             print(path_sha1sum,full_path)

def main():
    find_files('.')

if __name__ == '__main__': main()

    Demo Run

    bash-4.3$ cd Wallpapers/
bash-4.3$ recursive_checksum.py 
c66af072272d2c516e29832d0a86afa0e8e61d8c ./moon_moon.jpg
37829801c48ea0420414fc78de45adb13e4b117f ./wat.png
6cc3dd2541d00aa5fb8fd6ec703d3c7653ce4708 ./hard_drive_wallpapers/hard-drive.jpg
52fbff84cba6bbbfadc5777c1189ec39aef9176a ./hard_drive_wallpapers/hard-drive5.jpg
5bfe52eb8b31f50dc7bd1b1991dcc1d7260ec65e ./hard_drive_wallpapers/hard-drive4.jpg
f2f85eaa24c8c5b82bbedd55f887ea5fc520ac21 ./hard_drive_wallpapers/hard-drive3.jpg
bcdcf278c176fa93557627a33bedebe4e508e27a ./hard_drive_wallpapers/hard-drive2.jpg
    • -1