pdp's questions

I have been measuring the throughput between two hosts connected via a fiber channel. The OS is CentOS 7.9.2009 and iperf3 version is 3.1.7 (RPM). One of the tests was the measurement of transfer rates when the transferred data gets written to file. This is relevant because our application is write heavy. The server was started by running:

$ [server] iperf3 -s -F zero.img -i 30

The client by:

$ [client] iperf3 -c server -F zero.img -i 30 -p 5201 -n 95G

A large file used in the transfers was created by doing:

$ [client] dd if=/dev/zero of=zero.img count=100000 bs=1000kB

dstat was using to monitor system resources:

$ [server] dstat -t -m -s -d -c -l -n 60 10

iperf3 transferred the data at just about 10 MB/s.

While scp was way faster (over 130 MB/s).

Could anyone please suggest what could be the reason?

We have several HP ProLiant BL460c Gen8, BIOS I31 06/01/2015 servers running CentOS kernel 2.6.32-358.2.1 and are observing a sudden spike in kernel log timestamp. This is from /var/log/messages:

May 19 05:31:58 NWBLWICZVIS-A-VZ-GVS2-01 kernel: [    0.001000] Detected 2600.024 MHz processor.
May 19 05:31:58 NWBLWICZVIS-A-VZ-GVS2-01 kernel: [18014398.554743] Mount-cache hash table entries: 256

The value shoots from nearly zero to 208 days! This has been seen on at least 10 systems. Curiously, the seconds part of the timestamp is always 18014398. The system freezes and has to be hard reset. When running normally, there is no jump in timestamp.

I am trying to format a 2 TB SAN based block device (/dev/mapper/mpathb). Even though, mke2fs has been running since over one hour without completing, iostat -z -x -m 10 output in a separate terminal shows a surprising thing: There are no completed write requests (w/s) even though number of sectors written to disk (wMB/s) is pretty high.

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util

dm-0      0.00     0.00    0.00    0.10     0.00   204.80 4194304.00     1.00 11036.00 10000.00 100.00
dm-2      0.00     0.00    0.00    0.10     0.00   204.80 4194304.00     1.00 11036.00 10000.00 100.00

Hardly anything changes in iostat output: write speed remains high while number of sectors written remains zero! Any ideas?

Use of the auditd system to watch a directory via a symlink hardly triggers any logs. Here is the situation:

# pwd
/home/root/serverfault

# ls -l 
total 4
drwxr-xr-x 2 root root 4096 Sep  1 15:12 dir
lrwxrwxrwx 1 root root    3 Sep  1 15:12 p -> dir

# auditctl -w /home/root/serverfault/p -p rwxa -k PX

I had the naive impression that anytime I would read/write a file inside the /home/root/serverfault/p directory, logs would appear in /var/log/audit/audit.log. But they don't.

Also, there is no audit log if I do:

# echo hello > /home/root/serverfault/p/hello.txt

I did some experimentation using strace to run ls on few directories and evaluated the outputs with the information in the auditctl(8) man page. One interesting line from strace output that makes me think that audit logs should have been generated is:

open("/home/root/serverfault/p/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3

May be it did not because open(2) follows symlinks by default. Logs get generated only when running ls on the parent directory (/home/root/serverfault in our case) of the path (argument given to -w switch in auditctl) being watched.

I realize that /home/root/serverfault/p is not really a directory but a symlink. But weren't symlinks meant to be used such that they did not appear to exist? Overall, it seems that auditing directory changes via a symlink is not supposed to be done. Is this really the case?

These steps are from a box running CentOS 6.5. I merely tried converting a RPM package to a CPIO archive and was surprised to find files listed in the RPM but absent from the archive. Here are the steps taken.

List out paths in the RPM:

$ rpm -qlp sssd-1.9.2-82.4.el6_4.x86_64.rpm | sort > rpm.lst

Convert the RPM to a CPIO archive:

$ rpm2cpio sssd-1.9.2-82.4.el6_4.x86_64.rpm > sssd-1.9.2-82.4.el6_4.x86_64.cpio

Save the list of files in the archive. The sed filter is required to remove leading dot entries from pathnames. This is not required for the RPM listing.

$ cpio -i -t < sssd-1.9.2-82.4.el6_4.x86_64.cpio | sed -e 's|^.||' | sort > cpio.lst

Finally, the diff showing missing files:

$ diff -u cpio.lst rpm.lst

--- cpio.lst        2015-07-16 19:54:06.020494348 +0530
+++ rpm.lst         2015-07-16 19:53:38.012494371 +0530
@@ -2,6 +2,7 @@
 /etc/rc.d/init.d/sssd
 /etc/rwtab.d/sssd
 /etc/sssd
+/etc/sssd/sssd.conf
 /usr/bin/sss_ssh_authorizedkeys
 /usr/bin/sss_ssh_knownhostsproxy
 /usr/lib64/ldb/modules/ldb/memberof.so
@@ -102,6 +103,8 @@
 /var/lib/sss
 /var/lib/sss/db
 /var/lib/sss/mc
+/var/lib/sss/mc/group
+/var/lib/sss/mc/passwd
 /var/lib/sss/pipes
 /var/lib/sss/pipes/private
 /var/lib/sss/pubconf

Why are there files present in the RPM but missing in the CPIO archive? My interest was especially in /etc/sssd/sssd.conf because I wanted to look at a sample config file. We are using a sssd package that is older then the one coming from CentOS. I repeated the steps above with the package from CentOS. The result was the same.

I needed to transfer a 20 GB KVM vdisk file, storing the root filesystem of a CentOS 6.5 VM, from one lab server to another. The large file size and the fact that I had once compressed such a vdisk file to a few hundred mega-bytes made me instinctively enable compression with scp but I was surprised to see a rather low transfer speed. Then I tried bzip2 in combination with ssh and cat and was startled. Here is the summary of methods and average throughput.

• scp -C vm1-root.img [email protected]:/mnt/vdisks/, 11 MB/s.
• bzip2 -c vm1-root.img | ssh -l root 192.168.161.62 "bzip2 -d -c > /mnt/vdisks/vm1-root.img", 5 MB/s. This even lower result prompted searching on the Net.
• scp -c arcfour -C vm1-root.img [email protected]:/mnt/vdisks/, 13 MB/s. This use of -c arcfour as was suggested in one answer on serverfault. It hardly helped. Finally, I disabled compression.
• scp vm1-root.img [email protected]:/mnt/vdisks/, 23 MB/s.

Shouldn't compression have been faster?

After receiving the ssh(1) man page tip from @sven, I tried a couple alternative methods of file transfer not involving compression, both with better results.

• cat vm1-root.img | ssh -l root 192.168.161.62 "cat > /mnt/vdisks/vm1-root.img", 26 MB/s.

• nc -l 5678 > /mnt/vdisks/vm1-root.img on the receiver and nc 192.168.161.62 5678 < vm1-root.img on the transmitter, 40 MB/s. The port 5678 is an arbitrary one that was available.

Using nc turned out to be the fastest copying method!

In the past, scp -C has worked very well whenever I thought it would. For example, when transferring syslogs (/var/log/messages*) of few GBs in size. An uncompressed transfer rate of few hundred KB/s would increase to 1-2 MB/s. This example does fall in the case of a slow connection as has been pointed out in the man page.

I have a case where, a newly created vdisk image for a 20 GB partition has a compressed size of just 200 MB. With a transfer rate of about 25 MB/s, we could do the copying in just 8 seconds instead of over 13 minutes! Clearly, scp without compression is inefficient in this case and scp -C is even worse.

I guess, the main lesson learned here is that, scp -C should be thought of as only being a convenience. If a file can be significantly compressed, then it is better to first compress it on the source, transfer the compressed form and finally decompress on the destination. Tools that do the compression and decompression quickly (e.g. pbzip2) will be of greater help.

We have a CentOS 6.4 based server attached to Hitachi HNAS 3080 storage and observed the kernel remount the filesystem in read-only mode:

May 16 07:31:03 GNS3-SRV-CMP-001 kernel: [1259725.675814] EXT3-fs (dm-1): error: remounting filesystem read-only

This happened after several I/O errors and all paths to the device reportedly going down:

May 16 07:31:03 GNS3-SRV-CMP-001 multipathd: mpatha: remaining active paths: 0

I have been looking at sar logs and can see few very large (2 seconds) await times:

07:40:00       dev8-0     17.91    112.04     98.03     11.73      0.00      0.20      0.07      0.12
07:40:00      dev8-16      0.23      1.85      0.00      8.00      0.00      3.71      3.71      0.09
07:40:00      dev8-32     91.50   8338.76   5292.93    148.98      8.38     91.60      9.76     89.35
07:40:00     dev252-0     91.27   8336.91   5292.93    149.34     17.79    194.88      9.79     89.38
07:40:00     dev252-1    674.80   8168.16   5292.93     19.95   1473.53   2183.60      1.32     88.98

The duration between 07:30:00-07:40:00 does happen the time when the filesystem got mounted read-only. However, even under normal conditions, one repeated observation is that the await time for underlying devices is much lower than that of the multipath device. For instance:

00:00:00          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
00:10:00       dev8-0     19.27    129.41     78.61     10.80      0.01      0.27      0.16      0.32
00:10:00      dev8-16      0.23      1.80      0.00      8.00      0.00      0.86      0.84      0.02
00:10:00      dev8-32     94.88  10285.16   3363.48    143.86      3.39     35.76      6.83     64.82
00:10:00     dev252-0     94.65  10283.34   3363.48    144.18      3.64     38.47      6.86     64.89
00:10:00     dev252-1    435.06  10087.12   3363.48     30.92    118.42    272.21      1.47     64.12

dev8-0 happens to be the local disk, while dev8-16 (/dev/sdb) and dev8-32 (/dev/sdc) are the underlying ones for dev252-0 (/dev/mapper/mpatha). dev252-1 (/dev/mapper/mpathap1) is a single partition spanning the whole of the multipath device. Here is output from multipath -ll:

mpatha (2521501cbffffffffe96773b50ec30020) dm-0 BlueArc,NAS Platform
size=10T features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=enabled
| `- 9:0:0:0 sdc 8:32 active ready running
`-+- policy='round-robin 0' prio=1 status=active
  `- 8:0:0:0 sdb 8:16 active ready running

Why should the await time for /dev/mapper/mpathap1 be so much higher than that of /dev/mapper/mpatha or even /dev/sdb or /dev/sdc?

These observations are from CentOS 6.3, 6.4 and 6.5. I have a multipath device with a single partition:

# ls -l /dev/mapper/
total 0
crw-rw---- 1 admin root 10, 57 Dec 22 12:52 control
lrwxrwxrwx 1 admin root      7 Dec 24 12:22 mpatha -> ../dm-0
lrwxrwxrwx 1 admin root      7 Dec 24 12:22 mpathap1 -> ../dm-1

# kpartx -l /dev/mapper/mpatha
mpatha1 : 0 5368719293 /dev/mapper/mpatha 34

Then, parted is used to remove the partition:

# parted /dev/mapper/mpatha
GNU Parted 1.7.1
Using /dev/dm-0
Welcome to GNU Parted! Type 'help' to view a list of commands.

(parted) unit GB                                                          
(parted) p                                                                

Disk /dev/dm-0: 2749GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      0.00GB  2749GB  2749GB  ext3         primary       

(parted) rm 1                                                             
(parted) q                                                                
Information: Don't forget to update /etc/fstab, if necessary.

However, the device special file /dev/mapper/mpathap1 does not get automatically removed. Running ls -l /dev/mapper/ produces exactly the same output as shown above, while kpartx -l /dev/mapper/mpatha, exits normally ($? = 0) without printing anything.

My question is related to this one. However, there the issue was with no device special file being created in /dev/ while mine is about removal and that too on multipath devices. I am doing certain operations in a script and have already instrumented checks on whether expected device files actually get created or removed and if not then I call kpartx -a -p p <device> (took that hint from /lib/udev/rules.d/40-multipath.rules) or kpartx -d <device> as the case may be. But I have not yet had a satisfying understanding.

When in general, creating partitions on an extra local disk or a portable hard disk using parted works perfectly fine why should anything more be required to be done on disks coming from a SAN? Any information will be appreciated. Thanks!