Michael Graff's questions

I am attempting to add a GPS-based clock source to my network, using a GPS which emits a 1-PPS clock pulse. This is connected to a GPIO pin.

After allowing ntpd to run for 30 minutes or so, the PPS clock is still not used, and marked as bad, but I cannot figure out why.

ntpq> opeer
     remote           local      st t when poll reach   delay   offset    disp
==============================================================================
-127.127.28.0    127.0.0.1        2 l    5   16  377    0.000   17.704   2.125
x127.127.22.0    127.0.0.1        0 l    4   16  377    0.000    6.546   0.260
+50.116.36.122   10.42.134.1      3 u   33   64  377   43.017    2.337   5.208
+52.0.56.137     10.42.134.1      3 u   33   64  377   52.022   -6.430   3.944
-2001:418:8405:4 _d58f:78d0:fd39  3 u   30   64  377   85.262   13.465   4.929
*104.156.99.30   10.42.134.1      2 u   32   64  377   63.671   -0.166   5.484

ntpq> assoc

ind assid status  conf reach auth condition  last_event cnt
===========================================================
  1  2139  933a   yes   yes  none   outlier    sys_peer  3
  2  2140  9134   yes   yes  none falsetick   reachable  3
  3  2141  9424   yes   yes  none candidate   reachable  2
  4  2142  9424   yes   yes  none candidate   reachable  2
  5  2143  9324   yes   yes  none   outlier   reachable  2
  6  2144  963a   yes   yes  none  sys.peer    sys_peer  3

ntpq> rv 2140
associd=2140 status=9134 conf, reach, sel_falsetick, 3 events, reachable,
srcadr=127.127.22.0, srcport=123, dstadr=127.0.0.1, dstport=123, leap=00,
stratum=0, precision=-20, rootdelay=0.000, rootdisp=0.000, refid=PPS,
reftime=da584969.00b5c330  Sun, Jan 31 2016  8:50:49.002,
rec=da584969.0f2551bb  Sun, Jan 31 2016  8:50:49.059, reach=377,
unreach=0, hmode=3, pmode=4, hpoll=4, ppoll=4, headway=0, flash=00 ok,
keyid=0, ttl=0, offset=9.551, delay=0.000, dispersion=0.260,
jitter=2.148,
filtdelay=     0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00,
filtoffset=    9.55    9.97   10.46   10.98   11.49   11.95   12.46   12.91,
filtdisp=      0.03    0.27    0.51    0.75    0.99    1.23    1.47    1.71

The PPS source itself appears to be very clean and regular:

# ppstest /dev/pps0
trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
ok, found 1 source(s), now start fetching data...
source 0 - assert 1454230928.995087804, sequence: 7552 - clear  0.000000000, sequence: 0
source 0 - assert 1454230929.995115719, sequence: 7553 - clear  0.000000000, sequence: 0
source 0 - assert 1454230930.995144056, sequence: 7554 - clear  0.000000000, sequence: 0
source 0 - assert 1454230931.995173388, sequence: 7555 - clear  0.000000000, sequence: 0
source 0 - assert 1454230932.995202464, sequence: 7556 - clear  0.000000000, sequence: 0
source 0 - assert 1454230933.995230660, sequence: 7557 - clear  0.000000000, sequence: 0
source 0 - assert 1454230934.995258966, sequence: 7558 - clear  0.000000000, sequence: 0
source 0 - assert 1454230935.995287693, sequence: 7559 - clear  0.000000000, sequence: 0
source 0 - assert 1454230936.995317675, sequence: 7560 - clear  0.000000000, sequence: 0
source 0 - assert 1454230937.995345684, sequence: 7561 - clear  0.000000000, sequence: 0
source 0 - assert 1454230938.995375052, sequence: 7562 - clear  0.000000000, sequence: 0
source 0 - assert 1454230939.995403851, sequence: 7563 - clear  0.000000000, sequence: 0
source 0 - assert 1454230940.995433228, sequence: 7564 - clear  0.000000000, sequence: 0
source 0 - assert 1454230941.995463497, sequence: 7565 - clear  0.000000000, sequence: 0
source 0 - assert 1454230942.995491843, sequence: 7566 - clear  0.000000000, sequence: 0

When creating a linux software raid device as a raid10 device, I am confused why it must be initialized. The same question applies for raid1 or raid0, really.

Ultimately most people would put a file system of some sort on top of it, and that filesystem should not assume any state of the disk's data. Each write will affect both disks in a raid10 or raid1 setup, where the N mirrors are written to. There should be no reason whatsoever for a raid10 to be initialized initially, as it will happen over time.

I can understand why for a raid5/6 setup where there is a parity requirement, but even then it seems like this could be done lazily.

Is it just so people feel better about it?

I have a machine with an 8 channel LSI SAS3008 controller chip, and individual drive testing shows I can write to any disk or all disks at between 174 MB/sec and 193 MB/sec with a sustained write speed:

This is the output from the command dd if=/dev/zero of=/dev/mapper/mpath?p1 bs=1G count=100 oflag=direct run in parallel to all 12 disks:

107374182400 bytes (107 GB) copied, 556.306 s, 193 MB/s
107374182400 bytes (107 GB) copied, 566.816 s, 189 MB/s
107374182400 bytes (107 GB) copied, 568.681 s, 189 MB/s
107374182400 bytes (107 GB) copied, 578.327 s, 186 MB/s
107374182400 bytes (107 GB) copied, 586.444 s, 183 MB/s
107374182400 bytes (107 GB) copied, 590.193 s, 182 MB/s
107374182400 bytes (107 GB) copied, 592.721 s, 181 MB/s
107374182400 bytes (107 GB) copied, 598.646 s, 179 MB/s
107374182400 bytes (107 GB) copied, 602.277 s, 178 MB/s
107374182400 bytes (107 GB) copied, 604.951 s, 177 MB/s
107374182400 bytes (107 GB) copied, 605.44 s, 177 MB/s

However, when I put these disks together as a software raid 10 device, I get around 500 MB/sec write speed. I expected to get about double that, since there is no penalty for accessing these disks at the same time.

I did notice the md10_raid10 process which I assume does the software raid itself is nearing 80%, and one core is always at 100% wait time, and 0% idle. Which core that is changes, however.

Additionally, the performance drops even further when using the buffer cache to write to the mounted EXT4 filesystem rather than using oflag=direct to bypass the cache. The disks report 25% busy (according to munin monitoring) but the disks are clearly not running hot, but I worry the md10 device itself may be.

Any suggestions on where to go next on this? I am attempting a hardware raid 10 config to compare, although I can only build a 10 disk unit it seems -- that said, I hope to get 900 MB/sec writes sustained. I'll update this question as I discover more.

Edit 1:

If I use put a dd command in a tight loop writing to an ext4 partition mounted on that device, and I do not use the buffer cache (oflag=direct) I can get upwards of 950 MB/sec at peak and 855 MB/sec sustained with some alterations to the mount flags.

If I also read with iflag=direct at the same tim, I can get 480 MB/sec writes and 750 MB/sec reads now.

If I write without oflag=direct, thus using the buffer cache, I get 230 MB/sec writes and 1.2 MB/sec reads, but the machine seems to be very sluggish.

So, the question is, why would using the buffer cache so seriously affect performance? I have tried various disk queueing strategies including using 'noop' at the drive level and putting 'deadline' or 'cfq' on the appropriate multi path dm device, or deadline on all, or none on the dm and deadline on the backing drive. It seems like the backing drive should have none, and the multi path device should be the one I want, but this affects performance not at all, at least in the buffer cache case.

I am having wildly different software raid10 performance and behavior on two otherwise identical machines.

I have two machines which are hardware identical, bought at the same time, with the same software versions, hardware versions, and firmware versions. Each has a SAS controller with 8 x 6 Gb/s channels going to a SAS enclosure which holds 12 SAS disks.

On machine 1, which is stable and seems to be working perfectly, each disk in the raid array behaves more or less identically: busy time is equal (about 33% across all disks in production load levels), and while the weekly software raid check runs, write and read performance is not degraded. The full raid check completes in about a day, using all available spare bandwidth to complete it as fast as possible. This amounts to about 200 MB/sec reads while this check completes.

Machine 2 is a problem child. The full raid check completes in basically never, although it is configured to also use all available disk bandwidth. While it is attempting to check, it plods along at 5 MB/sec, and write performance drops to about 30 MB/sec during this time. Also, four disks are at 35% busy, while the remaining ones are 22% busy on average.

After cancelling the raid check on machine 2, the write speed returns to about 160 MB/sec.

If I use dd to test each individual mpath device, on machine 1 I get most speeds around 145 MB/sec reading per drive, and the lowest of 119 MB/sec followed by 127 MB. The rest are all in the 145 MB/sec range.

On machine 2, I get speeds between 107 MB (x 3 disks) and the rest are all above 135 MB/sec, with the peak of 191 MB/sec (!) for one disk.

I admit to being well out of my comfort zone here, but I cannot find any evidence to draw a conclusion from. I have also checked SMART stats on each disk on both machines, and while there are a fair number of "read corrected" errors on all disks, there seems to be no correlation between the values and the read performance, nor between the busy% difference.

Nothing I can find explains the poor performance when performing a RAID check of the array on one box vs on the other. Suggestions on where to go next to debug this would be appreciated.

I am looking for suggestions on how to configure a Mikrotik router to log NAT session setup and ideally teardown. However, setup only is sufficient for my immediate needs.

I found a method to log all packets, and I might be able to reduce this to only logging the TCP SYN, FIN, and RST packets, but I have not yet found that method. This doesn't cover UDP, which I would also like to track. For UDP, it's fine if a session times out and re-appears as a new session again.

We have two machines with a multi path SAS controller, each with 12 physical disks attached.

We were looking into some I/O speed oddness, and noticed that on one machine, where /dev/mpath/mpath*p1 maps to dm-13 through dm-24 in various ways, dm-13 and dm-14 have an io scheduler of "cfq" in place. The other dm devices in that list have "none" and it appears to not be able to be changed.

I believe the other devices are presented as a different set of dm-* mappings, and those DO have cfq set.

On the other machine, none of the devices selected to be in /dev/mpath/mpath*p1 have corresponding dm devices with a scheduler set.

I'm sort of out of my element here, but since iostat DOES show that some of the requests are being merged on dm-13 and dm-14 of the first box, and none of the others on either box, I suspect we are paying some price for this.

Am I digging in the wrong hole, or is this an issue? If so, how can I fix it, since echo cfq > /sys/block/dm-15/queue/scheduler does not have any effect when "none" is the only currently listed option?