Home / server / Questions / 6711
In Process
bene
Asked: 2009-05-10 14:38:38 +0800 CST

filesystem for millions of small files

  • 772

Which Linux filesystem would you choose for best speed in the following scenario:

  • a hundred million files
  • ~2k file size on average
  • >95% read access
  • pretty random access
  • high concurrency (>100 processes)

Note: The files are stored in a deep hierarchical tree to avoid large directories. Each leaf directory contains around one thousand files.

How would you benchmark it?

linux filesystems benchmark

7 Answers

  1. In terms of random seeks Reiser wins, followed by EXT4, followed by JFS. I'm not sure if this will correlate exactly to directory lookups, but it seems like it would be an indicator. You'll have to do your own tests for that specifically. EXT2 beats the pants off everything for file creation times, likely due to its lack of a journal, still EXT4 beats everything except Reiser which you may not want to use due to hans reiser's current status.

    You might want to look into drives that support NCQ, and make sure your install is setup to use it. Under heavy seeking it should provide a speed boost.

    Lastly, make sure your machine has a ton of ram. Since the files aren't often updated, linux will end up caching most of them to ram if it's got free space. If your usage patterns are right, this will give you a massive speed boost.

    • 23

  2. I agree with most of what Andrew said, except that I would recommend Reiser4 or the older (but better supported) ReiserFS. As those tests (and the documentation for ReiserFS) indicate, it is designed for precicesly the situation you are asking about (large numbers of small files or directories). I have used ReiserFS in the past with Gentoo and Ubuntu without any problems.

    As to the status of Hans Reiser, I do not see it as being a problem with the code or stability of the File System itself. Reiser4 is even sponsored by both DARPA and Linspire so while I agree that the further development of the Reiser File System is undetermined, I do not thing that should be a deciding factor as to whether anyone should use it or not.

    • 8

  3. I know this is not a direct answer to your question, but in these cases I think a database might be more suitable to host this. Small files can be stored in binary format in a database table and retrieved at wil. The software that is using these files should be able to support this though...

    • 4

  4. Somebody over on the Unix StackExchange created a benchmark (with source) to test just this scenario:

    Q: What is the most high-performance Linux filesystem for storing a lot of small files (HDD, not SSD)?

    The best read performance seems to come from ReiserFS.

    • 3

  5. In my experience, ext2 blows ext4 out of the water for small files. If you don't care about write integrity, it's great. For example, subversion creates lots and lots and lots of small files, which ext4 and other filesystems (XFS) choke on (run a cron job that rsyncs the data to ext4 from ext2 every half hour or so virtually solves the problem.)

    Running these commands makes ext2 even faster (even though most of these options make the filesystem unstable after a crash unless you run sync before it crashes). These commands have almost no effect on ext4 with small files.

    echo 15 > /proc/sys/vm/swappiness
echo 10 > /proc/sys/vm/vfs_cache_pressure
echo 99 > /proc/sys/vm/dirty_ratio
echo 50 > /proc/sys/vm/dirty_background_ratio
echo 360000 > /proc/sys/vm/dirty_expire_centisecs
echo 360000 > /proc/sys/vm/dirty_writeback_centisecs
echo "2000" > /proc/sys/vm/vfs_cache_pressure
    • 3

  6. I guess ext3 (or ext4), maybe JFS would be nice solution. I'd be wary with ext4 and btrfs (filesystems are tricky - be prepared with backups if you want to use latest, newest stuff).

    There are also various parameteres you can tweak during mkfs time to tune the filesystem to your liking.

    I'd certainly recommend against XFS. Not because it's a bad filesystem, but creation/deletion is a costly operation on it.

    To avoid problems with directory searches, use an intelligent naming scheme, for example:

    <first letter of id>_<last letter of id>/<id>

    or similar, more complicated schemes. This will speed up your directory searches and thus overall access speeds. (It's an old unix trick, back from V7 I think)

    • 1

  7. Most FS will choke with more than 65K files in a dir, I think that is still true of ext4. The Reiser file systems do not have that limit (the folks at mp3.com paid to make sure of that). Not sure about anything else, but that is one of the usage scenarios that ReiserFS was made for.

    • 1