Is it possible to temporarily disable the ZFS ARC cache?
I am trying to benchmark a ZFS SSD array using fio and want to avoid ZFS caches (via the ARC) from skewing the results. The alternative of benchmarking using a file size much larger than the system memory (i.e. 64GB fio file on a system with 1x 32GB system RAM) will increase the benchmark duration dramatically.
Using Ubuntu 19.10, fio 3.1.2 and ZFS on Linux 0.8.1
In ZFS that is not just filesystem, benchmarking is really hard and should always be simulation of future workload. So do writes that you are expecting, otherwise you made your system for something else than usage.
Since ARC and ZIL in RAM are things that ZFS rely heavily, those should never be taken out of load simulation. For analogy, we tested a new Ferrari but installed lawnmower engine to reduce fuel cost in long testing.
If you want to test with L2ARC and without, just remove L2ARC disk for testing time. Same for Intent Log device. That should be done just to see if you need one.
Article to gather more information on; Benchmarking ZFS