Home / server / Questions / 734630
Accepted
Mark Booth
Asked: 2015-11-08 03:17:51 +0800 CST

How do I prevent ntfs-3g use starving my ZFS ARC of memory?

  • 772

I have an HP Microserver running OpenMediaVault 2.1.18 (Stone burner), based on Debian GNU/Linux 7 (wheezy).

A few days ago, I needed to copy a large number of files from a USB drive, formatted with NTFS. I connected it, mounted it and started the copy and left it to complete.

A day later, I find that it has copied hardly anything, and my NAS performance had tanked. Investigating, I find that page and buffer cache are now taking up over 3/4 of system memory (together these normally take less than 10%) and the ZFS ARC is now squeezed into the remaining memory (less than 1/4 it's normal size).

When I realised what was wrong, I stopped the copy, unmounted and unplugged the drive and very rapidly the caches went back to their normal size, then the ARC started to fill itself back to it's normal level.

So, if I need to do do more direct copies in the future, how do I prevent mounting ntfs drives causing large amounts of memory to be consumed, starving my ZFS ARC of the memory it needs?

Note: I tried running echo 3 > /proc/sys/vm/drop_caches (to "free pagecache, dentries and inodes" according to man 5 proc), as shodanshok suggested, but this also flushed my ARC (except the memory needed to cache the L2ARC header). My system thus had lots of free memory, but since things are only cached in either ARC or L2ARC, my system then spent ages thrashing the drives to repopulate the ARC, not a desirable state of affairs.

In this image, you can see the ZFS starvation at the start of the 5th, when I started my ntfs copy, things returning to normal when unmounting the ntfs drive early on the 6th, and then finally the ARC being cleared (and rebuilt) after using drop_caches, on the afternoon of the 7th.

ARC starvation

zfs

1 Answers

  1. Best Answer

    The most simple solution is to periodically flush buffer and pagecache memory. You can do it easily running a similar bash line in the background (or in another tty):

    while true; do echo 1 > /proc/sys/vm/drop_caches; sleep 5; done

    Alternatively, you can try a different way to avoid polluting the buffer/pagecache. Basically, you need to copy each file using DIRECT_IO, bypassing the pagecache entirely. You can do that with a command similar to:

    dd if=srcfile of=dstfile bs=1M iflag=direct

    What the above command do is to open the srcfile with O_DIRECT option, bypassing the pagecache during reading but not during write. If you need to bypass even write caching, you can append a further oflag=direct option.

    The real problem with the above solution is that you must specify a file one by one, executing multiple (possibly thousands of). This means that you had to script around dd, so that you can copy an entire directory three.

    Other options include:

    • use ddrescue instead of dd
    • use a simple cp while first loading libdirectio (see here for more details)

    Finally, you can try running your NTFS application/copy via systemd-run and setting specific memory constrains, which should work for limiting pagecache also.

    • 1