Home / server / Questions / 1036342
In Process
Peeyush Kushwaha
Asked: 2020-10-05 07:04:46 +0800 CST

Tmpfs vs NVME SSDs from performance POV [duplicate]

  • 772

I have a script that creates large temporary files.

I'm inclined to use tmpfs for this, however I did a quick search for tmpfs performance and found this which reports speed as about 2GiB/s.

Modern NVME SSDs have comparable speeds. Putting aside differences other than performance (e.g. longer SSD life by not touching disk, SSDs having a larger while RAM is limited), are there any performance advantages of using tmpfs over SSDs?

performance ssd tmpfs

3 Answers

  1. tmpfs, being an extension of the pagecache, really operates as a "transparent" ramdisk. This means it provides very fast sequential read/write speed, but especially fast random IOPs (compared to a storage device).

    Some examples, collected on an aging Ryzen 1700 with run-of-the-mill memory:

    • dd if=/dev/zero of=test.img bs=1M count=4096 shows 2.8 GB/s

    • overwriting the just allocated files with dd if=/dev/zero of=test.img bs=1M count=4096 conv=notrunc,nocreat shows 3.5 GB/s

    • fio --rw=randread (random read IOPS) shows 492K iops for queue depth 1 (single-thread) workload, with 2.2M iops for queue depth 8 (8-threads) workloads. This vastly exceeds any NVMe flash-based disk (eg: Intel P4610) and even XPoint-based disks (eg: Intel Optane P4801X)

    For comparable performance, you would need an array of NVMe disks or, even better, memory-attached storage as NVDIMM.

    In short: if you can live with tmpfs volatile storage (ie: if you lose power, you will lose any written data) it is difficult to beat it (or ramdisks in general).

    However, you asked about writing large files to tmpfs, and this can be a challenge on its own: after all, writing GB-sized files will readily eat your available memory size (and budget).

    • 11

  2. One number you found is not a substitute for testing. Run your workload on your hardware and check if it is satisfactory. Are you doing small random access I/O, or the entire file?

    Yes, modern busses and interconnects mean DRAM and NVMe can both drive GB/s class of sequential I/O. It also can be true that certain reads from DRAM are 10x to 100x faster than persistent solid state storage. Whether this matters to you depends on your workload. See the various visualizations of Latency Numbers Every Programmer Should Know to get an idea of the orders of magnitude.

    After establishing performance, the operational considerations should not be dismissed. tmpfs will go away on reboot. If your use can tolerate no durability, great, but usually that is annoying. Yes, not writing to SSDs reduces their wear. Whether this matters depends on - you guessed it - your workload.

    • 8

  3. In terms of how it's implemented, tmpfs uses the same filesystem caching as other Linux filesystems, but the operations to flush data to disk are not implemented, so it's effectively cache only. In practice, this may mean that if your workload never flushes to disk, and fits entirely in memory (the latter would be a requirement to use tmpfs anyway), then even for filesystems that are backed by disk, requests would typically be served out of cache anyway, so you would see minimal performance difference between tmpfs and another filesystem.

    Of course, if your workload does flush to disk, then disk write times will matter, so the suggestion given by other answers - to test performance with as realistic a workload as possible - is still the right answer.

    • 1