Home / server / Questions / 704443
Accepted
starbeamrainbowlabs
Asked: 2015-07-09 12:03:10 +0800 CST

tar uses too much memory for its buffer - workaround?

  • 772

I am taring and then compressing a bunch of files&directories on my Ubuntu Server VPS for a backup. It only has 1GB of RAM and 128MB of Swap (I can't add more - OVH use OpenVZ as their virtualisation software), and every time tar runs it uses a ton of memory for it's buffer, causing everything else to get swapped out - even when using nice -n 10.

Is there any way to force tar to use a small buffer and reduce it's memory usage? I am worried that once the backup gets to be a certain size, my server will go down because tar won't have enough memory for it's buffer.

I am using bzip2 to compress, and I have already limited it's memory usage with the -4 option.

Edit: Here is what htop looks like when I have had tar running for a while:

enter image description here

Here is a link to the full gif

Edit 2: Here is the tar command I am using:

nice -n 20 tar --exclude "*node_modules*" --exclude "*.git/*" --exclude "/srv/www-mail/rainloop/v*"  -cf archive.tar /home /var/log /var/mail /srv /etc
backup

1 Answers

  1. Best Answer

    Your image shows quite the contrary, actually.

    As you can see under the RES column, tar memory consumption is quite low. You RAM usage appear to increase simply because Linux is actively caching the data read by the tar command. This, in turn, causes memory pressure and dirty page writeback (basically, the system flush its write cache to accommodate for the greater read-caching required) and, possibly, useful data are evicted from the I/O cache.

    Unfortunately, it seems that tar itself can not be instructed to use O_DIRECT or POSIX_FADVISE (both of which can be used to "bypass" the cache). So, using tar there is not a real solution out here...

    • 2