Background: I'm running lighttpd 1.4.28-1 on CentOS 5.3 to serve the static assets.
Signal: recently, it gets slow sometimes. I'm getting the below errors in the kernel log and /var/log/messages
:
proftpd[5145]: (::ffff:xx[::ffff:xx]) - FTP session opened.
proftpd[5145]: (::ffff:xx[::ffff:xx]) - Preparing to chroot to directory 'xx'
EXT3-fs warning (device dm-3): ext3_dx_add_entry: Directory index full!
Sep 16 15:30:34 xx last message repeated 489 times
The inode information:
df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/ddf1_p3
77037568 9996012 67041556 13% /
/dev/mapper/ddf1_p1
26104 35 26069 1% /boot
tmpfs 505834 1 505833 1% /dev/shm
The contents of the filesystem:
tune2fs 1.39 (29-May-2006)
Filesystem volume name: /
Last mounted on: <not available>
Filesystem UUID: 3e9a2bc7-b6d3-4c62-b31d-a9b8ea5ea106
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 77037568
Block count: 77017618
Reserved block count: 3850880
Free blocks: 52289663
Free inodes: 67588146
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 1005
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 32768
Inode blocks per group: 1024
Filesystem created: Fri Oct 30 16:00:47 2009
Last mount time: Fri Sep 9 01:43:27 2011
Last write time: Fri Sep 9 01:43:27 2011
Mount count: 7
Maximum mount count: -1
Last checked: Fri Oct 30 16:00:47 2009
Check interval: 0 (<none>)
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Journal inode: 8
First orphan inode: 75267694
Default directory hash: tea
Directory Hash Seed: 08fb2c9a-2241-4344-973f-045753e3d9ed
Journal backup: inode blocks
The lighttpd's document root is /data/www/lighttpd
, and the quick count talk me the biggest folder only contains 2k files:
find . -type d | while read d; do f=`ls -1 "$d" | wc -l`; echo $f $d; done | sort -n
Is journal corruption? Is there anything else that could cause this?
I had a similar problem and solved it with fsck after a quick research that led me to this blog post: http://rackerhacker.com/2008/10/13/ext3_dx_add_entry-directory-index-full/
NB: I strongly recommend backing up all your data before doing anything related to the filesystem.