Home / server / Questions / 438400
Accepted
Alex Flo
Asked: 2012-10-16 01:25:03 +0800 CST

Optimal directory structure for keeping millions of files on an ext4 system

  • 772

I need to keep millions of files on an ext4 system.
I understand that having a structure with multiple subdirectories is the general accepted solution. I wonder what would be the optimal approach in terms of number of dirs/subdirs.
For example I tried a structure like 16/16/16/16 (that is, (sub)directories from 1 to 16) and I found that I am able to move 100K files to this structure in 2m50s.
When trying to move 100K files to a 8/8/8/8/8/8 structure it took 11 minutes. So the 16/16/16/16 approach seems to be better but I was wondering if anyone has some empirical experience with an even better dir/subdir distribution.

linux

2 Answers

  1. 1) XFS now is better choice than EXT4.

    2) Even 1024 per directory entries should be quite okay, so you can appropriately reduce level of sub-dirs depth.

    • 2
  2. Best Answer

    After lots of testing I figured out a solution that works for me just find and I hope it may help others as well:
    I created a 255/255 dir structure and then I used the ramdisk (/dev/shm) to quickly create the directories and then save files inside them. Then once every minute I move the files to HDD.
    Using this approach reduced the time to save 100K files in 255/255 directories from about 5mins to some 5 seconds.

    • 0