I've got a very simple java 7 app that reads from a proprietary file format (Hadoop sequencefile) sitting on disk xvdb and creates millions of 2-20kb files on disk xvdf.
ByteBuffer byteBuffer = ByteBuffer.wrap(imageBinary, 0, ((BytesWritable)value).getLength());
File imageFile = new File(filePath);
FileOutputStream fos = new FileOutputStream( imageFile );
fos.getChannel().write(byteBuffer);
fos.close();
Running iostat -d 30 shows that we're doing more than double reading on the disk than writing. There is no other activity on this volume than the application above which is only writing to this disk.
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
xvdap1 0.40 0.00 3.07 0 92
xvdb 19.90 828.67 0.00 24860 0
xvdap3 0.00 0.00 0.00 0 0
xvdf 988.93 3538.93 1385.47 106168 41564
mount options:
/dev/xvdf on /mnt/ebs1 type ext4 (rw,noatime,nodiratime)
Creating a file requires first determining whether or not that file already exists. Since these files are so small, the reading of metadata to determine how and where to create the file exceeds the tiny write done once the file has been created.
If you're familiar with data structures, thing about adding a tiny leaf node to a binary tree, B-tree, or similar structure. You're going to do a lot of reading to figure out where the leaf node goes, whether it's already in the tree, and so on. That will be much greater than the tiny amount of data in the leaf node.