I'm backing around 2Tb of data from a ntfs disk to another ntfs disk with rsync ( I tried also with midnight commander ) and the copy starts with a "good" 25Mb/s speed, but after copying a couple of gigabytes the speed drops down to around 5Mb/s sometimes even less.
If I stop the copy and start again rsync to continue the copy the speed starts again around 25Mb/s and then goes back to 5Mb/s.
This is the start of rsync, from here everything copies that slowly.
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/11-Advanced Algorithms (COMPSCI 224), Lecture 11.mp4
502,527,183 100% 25.29MB/s 0:00:18 (xfr#2, ir-chk=1021/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/12-Advanced Algorithms (COMPSCI 224), Lecture 12.mp4
494,046,164 100% 25.45MB/s 0:00:18 (xfr#3, ir-chk=1020/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/13-Advanced Algorithms (COMPSCI 224), Lecture 13.mp4
389,502,911 100% 25.77MB/s 0:00:14 (xfr#4, ir-chk=1019/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/14-Advanced Algorithms (COMPSCI 224), Lecture 15.mp4
401,384,534 100% 14.92MB/s 0:00:25 (xfr#5, ir-chk=1018/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/15-Advanced Algorithms (COMPSCI 224), Lecture 16.mp4
498,564,894 100% 4.94MB/s 0:01:36 (xfr#6, ir-chk=1017/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/16-Advanced Algorithms (COMPSCI 224), Lecture 17.mp4
417,205,204 100% 2.30MB/s 0:02:52 (xfr#7, ir-chk=1016/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/17-Advanced Algorithms (COMPSCI 224), Lecture 18.mp4
495,885,960 100% 6.16MB/s 0:01:16 (xfr#8, ir-chk=1015/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/18-Advanced Algorithms (COMPSCI 224), Lecture 19.mp4
475,335,986 100% 2.75MB/s 0:02:45 (xfr#9, ir-chk=1014/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/19-Advanced Algorithms (COMPSCI 224), Lecture 20.mp4
485,359,371 100% 1.40MB/s 0:05:29 (xfr#10, ir-chk=1013/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/20-Advanced Algorithms (COMPSCI 224), Lecture 21.mp4
505,021,448 100% 6.46MB/s 0:01:14 (xfr#11, ir-chk=1012/53136)
>f+++++++++ _University Courses/Harvard/Harvard - Advanced Algorithms 2016/21-Advanced Algorithms (COMPSCI 224), Lecture 22.mp4
The drives are not noticeable fragmented.
I tried mounting the drives with the big_writes and the async option but I'd seen no difference.
I tested the speed with hdparm with and without cache and the speed is faster than those 25Mb/s.
I monitored with iostat and the %rwqm is very high, but I have no idea if that's bad or ok.
I'm trying to find the bottleneck or the problem but I haven't been able to locate it, any help on how could I monitor and analyze the problem would be appreciated.
Update: I tried disabling the disk write cache to check if the speed drop was happening after the drive filled the cache:
sudo hdparm -W0 /dev/sde
But the problem persist and the drive behaves the same.
Update:
$ free -h
total used free shared buff/cache available
Mem: 7.5Gi 1.1Gi 146Mi 13Mi 6.2Gi 6.1Gi
Swap: 0B 0B 0B
$ sudo dmidecode -s bios-version
4.6.4
$ sysctl vm.swappiness
vm.swappiness = 60
$ top
top - 21:16:09 up 6 days, 23:40, 5 users, load average: 3.45, 2.89, 2.58
Tasks: 179 total, 1 running, 178 sleeping, 0 stopped, 0 zombie
%Cpu(s): 12.5 us, 20.1 sy, 0.0 ni, 17.4 id, 48.4 wa, 0.0 hi, 1.6 si, 0.0 st
MiB Mem : 7689.6 total, 254.0 free, 1151.4 used, 6284.2 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 6225.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7320 usuario 20 0 284924 172348 1288 S 13.5 2.2 1:53.63 rsync
7305 usuario 20 0 174100 92956 2748 S 12.9 1.2 2:13.81 rsync
17809 usuario 20 0 1754596 476276 18104 S 11.9 6.0 2080:14 kodi-x11
27195 root 20 0 18972 9736 1840 S 7.9 0.1 58:03.15 mount.ntfs
14377 root 20 0 19948 10536 1764 S 6.3 0.1 126:50.65 mount.ntfs
13856 root 20 0 0 0 0 S 3.3 0.0 52:25.26 usb-storage
132 root 20 0 0 0 0 S 3.0 0.0 221:29.02 kswapd0
25821 usuario 20 0 23816 14980 2884 S 2.3 0.2 9:46.40 tmux: server
4909 root 20 0 0 0 0 D
Update after swapfile created:
free -h pre slow down:
$ free -h
total used free shared buff/cache available
Mem: 7.5Gi 427Mi 135Mi 11Mi 7.0Gi 6.8Gi
Swap: 4.0Gi 3.0Mi 4.0Gi
free -h during slow down:
$ free -h
total used free shared buff/cache available
Mem: 7.5Gi 421Mi 127Mi 11Mi 7.0Gi 6.8Gi
Swap: 4.0Gi 8.0Mi 4.0Gi
But still low speed after a few files copied:
sending incremental file list
.d..t...... _University Courses/MIT/MIT - Computers System Security 2014/
>f+++++++++ _University Courses/MIT/MIT - Computers System Security 2014/19-20. Mobile Phone Security.mp4
866,880,572 100% 28.03MB/s 0:00:29 (xfr#1, ir-chk=1003/53245)
>f+++++++++ _University Courses/MIT/MIT - Computers System Security 2014/20-21. Data Tracking.mp4
666,687,288 100% 27.70MB/s 0:00:22 (xfr#2, ir-chk=1002/53245)
>f+++++++++ _University Courses/MIT/MIT - Computers System Security 2014/21-22. Guest Lecture by MIT IS&T.mp4
716,033,570 100% 25.97MB/s 0:00:26 (xfr#3, ir-chk=1001/53245)
>f+++++++++ _University Courses/MIT/MIT - Computers System Security 2014/22-23. Security Economics.mp4
693,628,642 100% 25.63MB/s 0:00:25 (xfr#4, ir-chk=1000/53245)
>f+++++++++ _University Courses/MIT/MIT - Design and Analysis of Algorithms 2015/01-1. Course Overview, Interval Scheduling.mp4
1,069,906,667 100% 24.40MB/s 0:00:41 (xfr#5, ir-chk=1122/53368)
>f+++++++++ _University Courses/MIT/MIT - Design and Analysis of Algorithms 2015/02-2. Divide & Conquer - Convex Hull, Median Finding.mp4
919,036,317 100% 3.29MB/s 0:04:26 (xfr#6, ir-chk=1121/53368)
>f+++++++++ _University Courses/MIT/MIT - Design and Analysis of Algorithms 2015/03-R1. Matrix Multiplication and the Master Theorem.mp4
67,469,312 6% 4.71MB/s 0:03:23