Home / server / Questions / 581669
In Process
gertvdijk
Asked: 2014-03-13 15:00:15 +0800 CST

Why isn't the arc_max setting honoured on ZFS on Linux?

  • 772

I'm running ZoL 0.6.2 from their PPA on Ubuntu 12.04. It's on a host with 16GB of memory intended to run some VMs using KVM/Libvirt. After some time ZoL is using an insane amount of memory, reaching 98% of RAM usage with some VMs running. This results new processes refusing to start "unable to allocate memory". I can't even start all my VMs anymore which before using ZFS were using about 40-50% of RAM.

As far as I understand, without tweaking, ZoL should release memory as soon as the system is short on memory. Well, it doesn't. So I decided to set the arc_max setting to 1GB.

# echo 1073741824 >> /sys/module/zfs/parameters/zfs_arc_max

Still, it does not release any memory.

As you can see from the below ARC statistics, it's using more memory than it's configured to (compare c=7572030912 with c_max=1073741824).

What am I doing wrong here?

# cat /proc/spl/kstat/zfs/arcstats
4 1 0x01 84 4032 43757119584 392054268420115
name                            type data
hits                            4    28057644
misses                          4    13975282
demand_data_hits                4    19632274
demand_data_misses              4    571809
demand_metadata_hits            4    6333604
demand_metadata_misses          4    289110
prefetch_data_hits              4    1903379
prefetch_data_misses            4    12884520
prefetch_metadata_hits          4    188387
prefetch_metadata_misses        4    229843
mru_hits                        4    15390332
mru_ghost_hits                  4    1088944
mfu_hits                        4    10586761
mfu_ghost_hits                  4    169152
deleted                         4    35432344
recycle_miss                    4    701686
mutex_miss                      4    35304
evict_skip                      4    60416647
evict_l2_cached                 4    0
evict_l2_eligible               4    3022396862976
evict_l2_ineligible             4    1602907651584
hash_elements                   4    212777
hash_elements_max               4    256438
hash_collisions                 4    17163377
hash_chains                     4    51485
hash_chain_max                  4    10
p                               4    1527347963
c                               4    7572030912
c_min                           4    1038188800
c_max                           4    1073741824
size                            4    7572198224
hdr_size                        4    66873056
data_size                       4    7496095744
other_size                      4    9229424
anon_size                       4    169150464
anon_evict_data                 4    0
anon_evict_metadata             4    0
mru_size                        4    1358216192
mru_evict_data                  4    1352400896
mru_evict_metadata              4    508928
mru_ghost_size                  4    6305992192
mru_ghost_evict_data            4    4919159808
mru_ghost_evict_metadata        4    1386832384
mfu_size                        4    5968729088
mfu_evict_data                  4    5627991552
mfu_evict_metadata              4    336846336
mfu_ghost_size                  4    1330455552
mfu_ghost_evict_data            4    1287782400
mfu_ghost_evict_metadata        4    42673152
l2_hits                         4    0
l2_misses                       4    0
l2_feeds                        4    0
l2_rw_clash                     4    0
l2_read_bytes                   4    0
l2_write_bytes                  4    0
l2_writes_sent                  4    0
l2_writes_done                  4    0
l2_writes_error                 4    0
l2_writes_hdr_miss              4    0
l2_evict_lock_retry             4    0
l2_evict_reading                4    0
l2_free_on_write                4    0
l2_abort_lowmem                 4    0
l2_cksum_bad                    4    0
l2_io_error                     4    0
l2_size                         4    0
l2_asize                        4    0
l2_hdr_size                     4    0
l2_compress_successes           4    0
l2_compress_zeros               4    0
l2_compress_failures            4    0
memory_throttle_count           4    0
duplicate_buffers               4    0
duplicate_buffers_size          4    0
duplicate_reads                 4    0
memory_direct_count             4    66583
memory_indirect_count           4    7657293
arc_no_grow                     4    0
arc_tempreserve                 4    0
arc_loaned_bytes                4    0
arc_prune                       4    0
arc_meta_used                   4    427048272
arc_meta_limit                  4    2076377600
arc_meta_max                    4    498721632

# free -m
             total       used       free     shared    buffers     cached
Mem:         15841      15385        456          0         75         74
-/+ buffers/cache:      15235        606
Swap:            0          0          0
zfsonlinux

7 Answers

  1. IMHO the parameters in /sys/module/zfs/parameters can only be set to 0 / 1 - disabled / enabled." Correction: depends on the parameter

    I am in the same boat wanting to limit zfs' memory usage and it seems one has to create a /etc/modprobe.d/zfs.conf file and enter the parameter and the desired value in there. This change will take effect upon reboot.

    echo "options zfs zfs_arc_max=34359738368" >> /etc/modprobe.d/zfs.conf

    To effect the running module one can change the zfs_arc_max parameter.

    echo "34359738368" > /sys/module/zfs/parameters/zfs_arc_max

    Please note the use of > to replace the content of the file in contrast to adding to the file with >>.

    source: https://stackoverflow.com/a/18808311

    • 30

  2. Once you modify your arc size, you need to drop your caches. 

    echo 3 > /proc/sys/vm/drop_caches

    and wait (your prompt won't return immediately, but other processes will continue to run). It will slowly unload the caches (2.5 min for my 24GB cache on 2 pair of raid 1'd 2TB WD blacks on a 2Ghz 4 year old cpu on a box w/64GB) - beware, you will suddenly have no caches, and any processes reading data will be pulling off raw disk so you'll probably see IO wait jump up for a while til cache is repopulated.

    • 11

  3. http://fibrevillage.com/storage/169-zfs-arc-on-linux-how-to-set-and-monitor-on-linux

    this article is very good

    starting version ZoL 0.6.2 you can set the c_max at runtime, however the ARC size will not be released automatically. to force the RAM to be released it is needed to export the zpool.

    • 4

  4. One problem you could be falling into is ZFS caching the virtual machine files (virtual disks). To avoid that I always set the primarycache property to "metadata" on filesystems containing virtual disks.

    The logic being that the guest OS have better hint on what areas of its disks to cache.

    • 2

  5. You have one extra ">" too many.

    The command should be

    echo 1073741824 > /sys/module/zfs/parameters/zfs_arc_max

    not ">>"

    >> means "add to" (existing list).
    > means "overwrite" (value).

    This is why the command code in your question will not work.

    • 2

  6. AFAIK one of the following conditions must be met to adapt the parameter.

    1. On a running system: export all zpools, remove zfs modules, re-enable zfs module (per definition this can't be done if / is on zfs).
    2. Regenerate the initramfs image when changing the parameter so after a reboot it will work. This is needed as the zfs.conf file location is not yet mounted at that moment in the boot process.
    • 0

  7. Settings are not set when arc_max is lower than arc_min, you need to set arc_min too.

    /etc/modprobe.d/zfs.conf
options zfs zfs_arc_min=0
options zfs zfs_arc_max=2147483648

    you need rebuild initramfs with update-initramfs -u and reboot system

    you can check

    # arc_summary
ARC size (current):                                    10.6 %  216.9 MiB
    Target size (adaptive):                        97.9 %    2.0 GiB
    Min size (hard limit):                         97.9 %    2.0 GiB
    Max size (high water):                            1:1    2.0 GiB
    • 0