Home / server / Questions / 329273
Accepted
cedivad
Asked: 2011-11-10 04:25:15 +0800 CST

Store and backup 200 million small files

  • 772

My disk are 10x1TB SAS 7200 RPM in a RAID 10 with a MegaRaid 9260 hardware controller with cache/BBU. This results in a 4.6TB RAID 10 volume. hdparm -t (when device is empty) results in 500MB/s.

RAID chunk size is 64KB, filesystem block size is 2KB (I'm going to change it to the minimum chunk size and 4KB block size).

The directory pattern is /data/x/yz/zyxabc.gz

I'm using EXT4 with plans to move to XFS. The OS is RHEL 6.

As of now, it works great. The workload is 99% reads and it can read up to 300 files/second under normal conditions. The problem is backups. It takes 6 days to backup with scp. rsync is even slower. DD goes at about 2MB/s. LVM snapshots could be an option if I take the snapshot, back it up, and then delete it. Data consistency is very important to me.

Files are about 0.5-4KB each. Would I see increased backup performance if I stored all of the files in a database instead? What other alternatives are there for me to tackle the problem of backing up this many small files in a reasonable window?

backup linux storage xfs ext4

4 Answers

  1. Have you considered solutions like AMANDA or Bacula?

    • 3
  2. Best Answer

    i plan to move to XFS

    You'd better pre-order tons of Prozac in that case. :-) XFS sucks a lot on that pattern (lots of tiny files), alas.

    If you're considering FS change Reiser3 is the only option worth of trying in that case, IMO. With notail you get less CPU overhead, w/o notail — less disk space overhead.

    RAID chunk of 64 K is also beyond of sanity — why overflow disk I/O queues with such tiny patterns? Increase it instead of decreasing! With lots of simultaneous I/O it won't hurt.

    Now when it comes to backing up, it's possible to mention COW FSes. Such as Btrfs, or Nilfs. LVM-2 snapshots possibly are ok as well, so you can try combine it with migration to Reiser3. But I guess COW FSes have more chances to give you what's needed.

    • 2

  3. Either use a backup solution that supports incremental backups, such as those already mentioned, or perhaps can you use a script that traverses the tree and only copies files with a certain modify time?

    I'm not sure what you mean by "I need consistency" though. Do you mean all files need to be backed up at the same point in time (i.e. snapshot)? In that case I'm not sure any sort of tar, copy, rsync or similar will work - you'll HAVE to use something that can create file-system snapshots, or pause whatever process is creating these files in the first place.

    • 1

  4. "DD goes at about 2MB/s"

    I'm confused, doesn't dd do a sequential (or attempt to) read of the device? Is it competing with the online use of these files? If that's the case I think more disks/faster disks are in order. 1TB SAS is still 7,200 RPM if I'm not mistaken, you can pick up 600GB 15K SAS which will cut your seeks drastically.

    Are you dumping it to a RAMDisk? So that your destination location can't be the bottleneck of the DD test (and you're not dumping it right back to the local disk, again causing high seeks).

    If 2MB/s the best you're going to get out of the fastest possible read pattern, you need faster disks.

    However, dd wont get you a consistent snapshot without combining it with something else.

    • 0