Laurentiu Soica's questions

I have this use case: Setup multiple k8s clusters that can communicate with each other. I also have one network per cluster for intra-cluster communication and another network for inter-cluster communication and for external access in general. Like below:

I am currently leveraging flannel as the network plugin. My understanding is that flannel does not support this use-case (assumption).

Is this something I can achieve with a different CNI plugin? Do I need multiple CNI plugins / plugin instances / a CNI multiplexer?

In case anyone implemented something similar, feedback is highly appreciated.

Thanks!

Update: Multus-cni is not an option as it does not support NodePort on secondary interfaces https://github.com/k8snetworkplumbingwg/multus-cni/issues/727

I have a system (centos 7.8) with k8s installed on top.

After a few days of normal operations (system load is ~30% and disk activity is around 60 iops, not saturated), the system goes to an unstable state where nothing is commited to disk anymore. As can be seen from an iostat -x 5, avgqu-sz freezes no more r/w are accepted.

The question is where should I look next in order to identify the root cause.

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    0,00     0,00     0,00     0,00   169,00    0,00    0,00    0,00   0,00 100,00
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00   186,00    0,00    0,00    0,00   0,00 100,00
dm-1              0,00     0,00    0,00    0,00     0,00     0,00     0,00     7,00    0,00    0,00    0,00   0,00 100,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11,36    0,00    1,71   86,94    0,00    0,00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    0,00     0,00     0,00     0,00   169,00    0,00    0,00    0,00   0,00 100,00
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00   186,00    0,00    0,00    0,00   0,00 100,00
dm-1              0,00     0,00    0,00    0,00     0,00     0,00     0,00     7,00    0,00    0,00    0,00   0,00 100,00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          11,41    0,00    1,58   87,01    0,00    0,00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0,00     0,00    0,00    0,00     0,00     0,00     0,00   169,00    0,00    0,00    0,00   0,00 100,00
dm-0              0,00     0,00    0,00    0,00     0,00     0,00     0,00   186,00    0,00    0,00    0,00   0,00 100,00
dm-1              0,00     0,00    0,00    0,00     0,00     0,00     0,00     7,00    0,00    0,00    0,00   0,00 100,00

The last log lines from /var/log/messages before a force system reset were:

kernel: ACPI Error: SMBus/IPMI/GenericSerialBus write requires Buffer of length 66, found length 32 (20130517/exfield-389)
kernel: ACPI Error: Method parse/execution failed [\_SB_.PMI0._PMM] (Node ffff99c2ba2513c0), AE_AML_BUFFER_LIMIT (20130517/psparse-536)
kernel: ACPI Exception: AE_AML_BUFFER_LIMIT, Evaluating _PMM (20130517/power_meter-339)

although, according to this RedHat thread, it should not be an issue https://access.redhat.com/discussions/3871951

LE 1: Occasionally I get similar freezes over small periods of time (less than a minute) and then it recovers. In the dmesg output I have:

[Lu aug 17 21:04:07 2020] hpsa 0000:06:00.0: scsi 0:1:0:0: resetting logical  Direct-Access     HP       LOGICAL VOLUME   RAID-1(+0) SSDSmartPathCap+ En+ Exp=1
[Lu aug 17 21:04:15 2020] hpsa 0000:06:00.0: device is ready.
[Lu aug 17 21:04:15 2020] hpsa 0000:06:00.0: scsi 0:1:0:0: reset logical  completed successfully Direct-Access     HP       LOGICAL VOLUME   RAID-1(+0) SSDSmartPathCap+ En+ Exp=1

LE 2: Managed to save a dmesg output when the disk doesn't recover anymore and a reset is required.

[Lu aug 24 13:00:18 2020] hpsa 0000:06:00.0: scsi 0:1:0:0: resetting logical  Direct-Access     HP       LOGICAL VOLUME   RAID-1(+0) SSDSmartPathCap+ En+ Exp=1
[Lu aug 24 13:03:20 2020] INFO: task scsi_eh_0:332 blocked for more than 120 seconds.
[Lu aug 24 13:03:20 2020] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Lu aug 24 13:03:20 2020] scsi_eh_0       D ffff8c603fc9acc0     0   332      2 0x00000000
[Lu aug 24 13:03:20 2020] Call Trace:

So hpsa enters this resetting logical disk-access procedure that never completes.

I have a k8s cluster on 4 VMs. 1 master and 3 workers. On each of the workers, I use rook to deploy a ceph OSD. The OSDs are using the same disk as the VM Operating System.

The VM disks are remote (the underlaying infrastructure is again a Ceph cluster).

This is the VM disk performance (similar for all 3 of them):

$ dd if=/dev/zero of=testfile bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 4.82804 s, 222 MB/s

And the latency (await) while idle is around 8ms.

If I mount an RBD volume inside a K8S POD, the performance is very poor:

$ dd if=/dev/zero of=testfile bs=1G count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 128.619 s, 8.3 MB/s 

During high load (100% util for the rbd volume), the latency of the RBD volume is greater than 30 seconds.

I know that my setup is not what ceph recommends and that dd is not the best tool to profile disk performance, but the penalty from having ceph on top of VM disks is still huge.

VM Operating system is

CentOS 7.7.1908.
Kernel 3.10.0-1062.12.1.el7.x86_64

Network bandwidth between worker nodes:

[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  2.35 GBytes  2.02 Gbits/sec

Network latency is less than 1 ms.

I'm looking for some hints into further troubleshooting the issue and improving performance.