Home / ubuntu / Questions / 1274866
Accepted
Fabman
Asked: 2020-09-14 09:30:09 +0800 CST

How can I analize why rsync copies very slow after copying very fast a few files on ntfs drives?

  • 772

I'm backing around 2Tb of data from a ntfs disk to another ntfs disk with rsync ( I tried also with midnight commander ) and the copy starts with a "good" 25Mb/s speed, but after copying a couple of gigabytes the speed drops down to around 5Mb/s sometimes even less.

If I stop the copy and start again rsync to continue the copy the speed starts again around 25Mb/s and then goes back to 5Mb/s.

This is the start of rsync, from here everything copies that slowly.

>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/11-Advanced Algorithms (COMPSCI 224), Lecture 11.mp4
    502,527,183 100%   25.29MB/s    0:00:18 (xfr#2, ir-chk=1021/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/12-Advanced Algorithms (COMPSCI 224), Lecture 12.mp4
    494,046,164 100%   25.45MB/s    0:00:18 (xfr#3, ir-chk=1020/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/13-Advanced Algorithms (COMPSCI 224), Lecture 13.mp4
    389,502,911 100%   25.77MB/s    0:00:14 (xfr#4, ir-chk=1019/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/14-Advanced Algorithms (COMPSCI 224), Lecture 15.mp4
    401,384,534 100%   14.92MB/s    0:00:25 (xfr#5, ir-chk=1018/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/15-Advanced Algorithms (COMPSCI 224), Lecture 16.mp4
    498,564,894 100%    4.94MB/s    0:01:36 (xfr#6, ir-chk=1017/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/16-Advanced Algorithms (COMPSCI 224), Lecture 17.mp4
    417,205,204 100%    2.30MB/s    0:02:52 (xfr#7, ir-chk=1016/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/17-Advanced Algorithms (COMPSCI 224), Lecture 18.mp4
    495,885,960 100%    6.16MB/s    0:01:16 (xfr#8, ir-chk=1015/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/18-Advanced Algorithms (COMPSCI 224), Lecture 19.mp4
    475,335,986 100%    2.75MB/s    0:02:45 (xfr#9, ir-chk=1014/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/19-Advanced Algorithms (COMPSCI 224), Lecture 20.mp4
    485,359,371 100%    1.40MB/s    0:05:29 (xfr#10, ir-chk=1013/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/20-Advanced Algorithms (COMPSCI 224), Lecture 21.mp4
    505,021,448 100%    6.46MB/s    0:01:14 (xfr#11, ir-chk=1012/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/21-Advanced Algorithms (COMPSCI 224), Lecture 22.mp4

The drives are not noticeable fragmented.

I tried mounting the drives with the big_writes and the async option but I'd seen no difference.

I tested the speed with hdparm with and without cache and the speed is faster than those 25Mb/s.

I monitored with iostat and the %rwqm is very high, but I have no idea if that's bad or ok.

enter image description here

I'm trying to find the bottleneck or the problem but I haven't been able to locate it, any help on how could I monitor and analyze the problem would be appreciated.

Update: I tried disabling the disk write cache to check if the speed drop was happening after the drive filled the cache:

sudo hdparm -W0 /dev/sde

But the problem persist and the drive behaves the same.

Update:

$ free -h                                                                                                                          
total        used        free      shared  buff/cache   available                                                                  
Mem:          7.5Gi       1.1Gi       146Mi        13Mi       6.2Gi       6.1Gi                                                                  
Swap:            0B          0B          0B    
                                                                                              
$ sudo dmidecode -s bios-version                                                                                     
4.6.4                                                                                                                                            

$ sysctl vm.swappiness                                                                                               
  vm.swappiness = 60

$ top

top - 21:16:09 up 6 days, 23:40,  5 users,  load average: 3.45, 2.89, 2.58
Tasks: 179 total,   1 running, 178 sleeping,   0 stopped,   0 zombie
%Cpu(s): 12.5 us, 20.1 sy,  0.0 ni, 17.4 id, 48.4 wa,  0.0 hi,  1.6 si,  0.0 st
MiB Mem :   7689.6 total,    254.0 free,   1151.4 used,   6284.2 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   6225.8 avail Mem 
                                                                    
PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND

7320 usuario   20   0  284924 172348   1288 S  13.5   2.2   1:53.63 rsync                                                                        
7305 usuario   20   0  174100  92956   2748 S  12.9   1.2   2:13.81 rsync                                                                       
17809 usuario   20   0 1754596 476276  18104 S  11.9   6.0   2080:14 kodi-x11                                                                    
27195 root      20   0   18972   9736   1840 S   7.9   0.1  58:03.15 mount.ntfs                                                                  
14377 root      20   0   19948  10536   1764 S   6.3   0.1 126:50.65 mount.ntfs                                                                  
13856 root      20   0       0      0      0 S   3.3   0.0  52:25.26 usb-storage                                                                   
132 root      20   0       0      0      0 S   3.0   0.0 221:29.02 kswapd0                                                                     
25821 usuario   20   0   23816  14980   2884 S   2.3   0.2   9:46.40 tmux: server                                                                 
4909 root      20   0       0      0      0 D

Update after swapfile created:

free -h pre slow down:

$ free -h
              total        used        free      shared  buff/cache   available
Mem:          7.5Gi       427Mi       135Mi        11Mi       7.0Gi       6.8Gi
Swap:         4.0Gi       3.0Mi       4.0Gi

free -h during slow down:

$ free -h
              total        used        free      shared  buff/cache   available
Mem:          7.5Gi       421Mi       127Mi        11Mi       7.0Gi       6.8Gi
Swap:         4.0Gi       8.0Mi       4.0Gi

But still low speed after a few files copied:

sending incremental file list
.d..t...... _University Courses/MIT/MIT - Computers System Security 2014/
>f+++++++++ _University Courses/MIT/MIT - Computers System Security 2014/19-20. Mobile Phone Security.mp4
    866,880,572 100%   28.03MB/s    0:00:29 (xfr#1, ir-chk=1003/53245)
>f+++++++++ _University Courses/MIT/MIT - Computers System Security 2014/20-21. Data Tracking.mp4
    666,687,288 100%   27.70MB/s    0:00:22 (xfr#2, ir-chk=1002/53245)
>f+++++++++ _University Courses/MIT/MIT - Computers System Security 2014/21-22. Guest Lecture by MIT IS&T.mp4
    716,033,570 100%   25.97MB/s    0:00:26 (xfr#3, ir-chk=1001/53245)
>f+++++++++ _University Courses/MIT/MIT - Computers System Security 2014/22-23. Security Economics.mp4
    693,628,642 100%   25.63MB/s    0:00:25 (xfr#4, ir-chk=1000/53245)
>f+++++++++ _University Courses/MIT/MIT - Design and Analysis of Algorithms 2015/01-1. Course Overview, Interval Scheduling.mp4
  1,069,906,667 100%   24.40MB/s    0:00:41 (xfr#5, ir-chk=1122/53368)
>f+++++++++ _University Courses/MIT/MIT - Design and Analysis of Algorithms 2015/02-2. Divide & Conquer - Convex Hull, Median Finding.mp4
    919,036,317 100%    3.29MB/s    0:04:26 (xfr#6, ir-chk=1121/53368)
>f+++++++++ _University Courses/MIT/MIT - Design and Analysis of Algorithms 2015/03-R1. Matrix Multiplication and the Master Theorem.mp4
     67,469,312   6%    4.71MB/s    0:03:23
lubuntu monitoring rsync ntfs ntfs-3g

1 Answers

  1. Best Answer

    You've got no SWAP!

    Let's create a 4G /swapfile...

    Note: Incorrect use of the dd command can cause data loss. Suggest copy/paste.

    sudo swapoff -a           # turn off swap
sudo rm -i /swapfile      # remove old /swapfile

sudo dd if=/dev/zero of=/swapfile bs=1M count=4096

sudo chmod 600 /swapfile  # set proper file protections
sudo mkswap /swapfile     # init /swapfile
sudo swapon /swapfile     # turn on swap
free -h                   # confirm 8G RAM and 4G swap

    Confirm this line in /etc/fstab... and confirm no other “swap” lines...

    /swapfile    none    swap    sw      0   0

    reboot                    # reboot and verify operation

    Update #1:

    Creating a /swapfile did not fix the copy speed problem.

    However, we laid down a fresh GPT partition table on the external drive, and created a fresh ext4 partition (native to Linux), instead of NTFS (not native to Linux), and copies are occurring at the expected speed.

    Further testing to see if going back to a NTFS partition causes the problem again, proves that ntfs-3g mounted partitions are the problem. Suggest trying ntfs mounted partitions.

    Update #2:

    Solved!. It was the ntfs driver, ntfs and ntfs-3g are very very slow. I'd been looking around and I found that Paragon developed a free version of their commercial ntfs kernel space driver. I installed it and after ten hours of copying is still copying at 40Mb/s without hiccup. See https://www.paragon-software.com/home/ntfs-linux-professional/.

    • 1