Home / server / Questions / 309458
Accepted
fields
Asked: 2011-09-09 09:15:03 +0800 CST

What's the best program to use for copying 12TB of data onto 4 3TB hard drives on linux?

  • 772

I have a 12TB enclosure that's filled with data (ext3). I'd like to make a copy of it, spanned across individual 3TB disks, each of which is readable on its own (no RAID or bonding). Ideally, I'd like this process to spit out some index of which files are on which disk, but that's not strictly necessary. The end result must be readable files on the disk in the original directory structure, not an archive file. Splitting directories (duplicating the directory structure) across volumes is fine.

The 3TB disks are not all mounted at the same time (and not able to be), so I need something that will let me pause the copying to switch disks. I strongly prefer a solution that doesn't require me to figure out which files go on which disk or keep track of what's been copied.

Is there an existing package that will do this?

hard-drive linux copying

3 Answers

  1. Best Answer

    I don't know of an existing package that will do this but here is a completely untested no-error-checking script that may work:

    #!/bin/sh

SRC=/foo
DST=/bar
IDX=/tmp/CONTENTS

DISK_ID=`uuidgen`

echo "$DISK_ID" > $DST/disk_id

cd $SRC
find . -type f | while read FILE; do

    FILE_SIZE=`stat -c '%a' "$FILE"`
    FREE_DST=`stat -c '%a' "$DST"`

    if [ $FREE_DST -lt $FILE_SIZE ]; then

            while [ $FREE_DST -lt $FILE_SIZE ]; do
                    read -p "Please mount a new disk at $DST and hit enter:"
                    FREE_DST=`stat -c '%a' "$DST"`
            done

            DISK_ID=`uuidgen`
            echo "$DISK_ID" > $DST/disk_id
    fi


    DST_DIR=`dirname "$FILE"`
    DST_DIR="$DST/$DST_DIR"
    [ -d "$DST_DIR" ] || mkdir -p "$DST_DIR"

    cp -p "$FILE" "$DST_DIR"

    echo "$DISK_ID $FILE" >> $IDX
done
    • 1

  2. It's probably out of the scope you are looking for, but the free version of Unraid would do this. You could just assign the 3 drives as data with no parity drive. Set high water to "fill up" and set split levels to whatever meets your needs.

    • 0

  3. rsync anyone? make a few folders, one for each drive, move data that will occupy one drive, then just do a check with rsync, not actually move the data.

    check man page

    • 0