Home / ubuntu / Questions / 1100937
Accepted
Charles Green
Asked: 2018-12-15 09:44:44 +0800 CST

How can I limit page cache/buffer size

  • 772

My problem is this: My laptop has a relatively slow disk subsystem (amd I'm not going to buy a better one) and I backup my system using rsync which works well for me. However, during the backup process, the files read are read into the buffer/cache of the system, which eventually triggers the swap system.

For example, running cat Win10.qcow2 > /dev/null a 60 GB file will result in 

free -h
              total        used        free      shared  buff/cache   available
Mem:           15Gi       2.2Gi       210Mi       170Mi        13Gi        12Gi
Swap:          30Gi        14Mi        30Gi

and if I write to a real device, like my USB backup drive, the swap starts being used, up to a couple of GB. I do have vm.swappiness = 0 in /etc/sysctl

By itself, this is not bad, but because of my slow disk system, the computer becomes less than sprightly in response to inputs. Painfully slow, in fact.

What I would like to have is a method for limiting the amount of page buffer that the process can consume, leaving enough room to run smaller commands, such as opening a terminal.

What I have tried, is using lxc, which did not limit the system use of buffers, docker which I could not fully figure out yet, and I'm attempting to get lxd running, but I'll need some time to figure that one out.

There is a program nocache which I think works, but rsync then does not output progress indicators.

swap rsync ram 18.10

1 Answers

  1. Best Answer

    By default when rsync updates your backup it creates a copy of the file and then moves it into place. To avoid this step you can have rsync write directly to your backup with the --inplace argument.

    As per https://linux.die.net/man/1/rsync:

    --inplace

    This option changes how rsync transfers a file when the file's data needs to be updated: instead of the default method of creating a new copy of the file and moving it into place when it is complete, rsync instead writes the updated data directly to the destination file.

    This has several effects:

    (1) in-use binaries cannot be updated (either the OS will prevent this from happening, or binaries that attempt to swap-in their data will misbehave or crash),

    (2) the file's data will be in an inconsistent state during the transfer,

    (3) a file's data may be left in an inconsistent state after the transfer if the transfer is interrupted or if an update fails,

    (4) a file that does not have write permissions can not be updated, and

    (5) the efficiency of rsync's delta-transfer algorithm may be reduced if some data in the destination file is overwritten before it can be copied to a position later in the file (one exception to this is if you combine this option with --backup, since rsync is smart enough to use the backup file as the basis file for the transfer).

    WARNING: you should not use this option to update files that are being accessed by others, so be careful when choosing to use this for a copy.

    This option is useful for transfer of large files with block-based changes or appended data, and also on systems that are disk bound, not network bound.

    The option implies --partial (since an interrupted transfer does not delete the file), but conflicts with --partial-dir and --delay-updates. Prior to rsync 2.6.4 --inplace was also incompatible with --compare-dest and --link-dest.

    • 2