Home / server / Questions / 743412
In Process
jithujose
Asked: 2015-12-17 05:07:23 +0800 CST

Rsync hangs: expand file_list pointer array to N bytes, did move

  • 772

Rsync is going into "interruptible sleep" mode after transferring some files from a local folder to a NFS folder. The folder I am trying to backup contains more than 180gb of data.

This is what rsync outputs before it hangs:

[sender] expand file_list pointer array to 524288 bytes, did move

I am running Ubuntu Server 14.04 LTS with rsync version 3.1.0 protocol version 31 and I am running rsync with these options:

/usr/bin/rsync -rHAXxvvut --numeric-ids --progress {SRC_FOLDER} {NFS_FOLDER}

Thanks for any hints

linux

4 Answers

  1. Considering that the rsync you're using is an open-source software, it's quite easy to get accesso to related source code.

    After downloading the main .tar.gz and applied the Ubuntu patch (rsync_3.1.0-2ubuntu0.4.diff.gz), you end up with exactly the code underlying the rsync you're using. Something like this:

    $ mkdir rsync
$ cd rsync/
$ wget http://archive.ubuntu.com/ubuntu/pool/main/r/rsync/rsync_3.1.0.orig.tar.gz
$ wget http://archive.ubuntu.com/ubuntu/pool/main/r/rsync/rsync_3.1.0-2ubuntu0.4.diff.gz
$ gzip -d rsync_3.1.0-2ubuntu0.4.diff.gz
$ tar zxvf rsync_3.1.0.orig.tar.gz 
$ cd rsync-3.1.0/
$ patch -p1 < ../rsync_3.1.0-2ubuntu0.4.diff

    Now a simple grep can quickly tell us the context of your error message:

    $ grep -r 'expand file_list pointer array to' 
flist.c:        rprintf(FCLIENT, "[%s] expand file_list pointer array to %s bytes, did%s move\n",

    So you're lucky, as your error message is used in a single fragment of a single file. Nameli: flist.c.

    Let's give a look:

    flist.c context

    It's relatively easy to guess that the routine containing the error message (lines 325, 326, 327, 328) is named flist_expand and sounds like something needed to ensure that the whole file list (to rsync) can be held in a properly sized in-memory structure (aka: the more files you need to rsync, the more memory is required to handle rsync-computations, and as such a list is not known "in advance", it need to be dinamically computed, by allocating proper chunks of memory to a "list" [more or less]).

    So, I would bet that your problem rely NOT on the size of data you're rsync-ing, but on the number of files. I'd try splitting your rsync in multiple sub-rsync, by focusing on internal subfolders.

    Actually, it would be nice to better investigate the:

    1. line 328: (new_ptr == flist->files) ? " not" : "");
    2. line 334: out_of_memory("flist_expand");

    but this goes much beyond my initial goal :-)

    Anyway, I would bet that checking your logs you would find some "out of memory" message.... :-)

    HTH!

    • 4

  2. Two remarks that may help

    • have always been satisfied with the -a option (from the man "archive mode; equals -rlptgoD (no -H,-A,-X)")

    • rsync may be waiting for NFS to give access to a file. It seems NFS is able to actually get rsync locked (probably when overwriting a certain file), and it'd be interesting to see which file was being accessed by rsync just before entering its "sleep". There is a command to see what files are currently opened by rsync lsof -ad3-999 -c rsync (from askubuntu)

    • 0

  3. Using rsync to transfer content to an NFS folder is likely to be very inefficient. Think about what's involved when rsync wants to get a checksum of the remote file, or to modify a remote file in place. You're far better to have rsync talk to an rsync process running on the file server. If at all possible I'd change that first and then look to see if your current problem still exists. I.e. use rsync over ssh or run an rsync daemon process, and leave NFS out of the picture entirely.

    To find out what rsync is doing, strace might be useful:

    strace -p <PID>

    Or have strace start rsync like so:

    strace rsync [rsync options] <src> <target>

    To attach to a running process by PID may require root permissions, though that can be changed (by root).

    • 0

  4. I had the same problem and managed to get around the problem by doing the transfer remotely. For example, instead of doing rsync -avvxAX source/ destination/, I did rsync -avvxAX source/ root@localhost:destination/

    rsync probably have specific optimizations for local transfers. Until we can figure out the exact cause of the problem, this workaround fixes the problem for me.

    My system: Gentoo Linux, kernel 4.3.0, systemd, rsync 3.1.2 protocol version 31

    • 0