Alex's questions

We have a number of AWS EC2 instances within the same AZ that transmit large amounts of network traffic to each other. In a small fraction of the connections, when a client on host A connects to a server on host B and sends a large amount of data (e.g. 20 GB) from A to B at a high rate, the TCP connection freezes or times out. I've investigated this and the symptoms aren't always the same, but typically it appears that when a connection is impacted by this problem the sender (on host A) stops receiving the ACKs that the receiving side (host B) is sending to A after some time. So at first all ACKs pass through, and then they get blocked in the middle of the connection. Also, VPC Flow Logs show that some packets returning from host B (the receiver) to host A (the sender) are rejected.

This happens on a number of EC2 instances (typically r5a.xlarge) that run Debian Linux 10 with Linux kernel 5.3.9 and the ENA AWS network driver that's shipped as part of Debian's kernel. They run Docker 18.09.1, installed via the docker.io Debian Buster package. Interestingly, I'm not able to reproduce the issue on Amazon Linux 2 (with Docker installed).

I've been able to reproduce it by letting the following simple experiment run in a loop for some time:

# Host B (server receiving data)
docker run -it --rm -p 20098:20098 debian:buster bash
apt-get update && apt-get -y install netcat-openbsd
while true; do date; nc -l -p 20098 | dd of=/dev/null bs=1M; done

# Host A (client sending data)
docker run -it --rm debian:buster bash
apt-get update && apt-get -y install netcat-openbsd
while sleep 1; do date; dd if=/dev/zero bs=1M count=20480 | nc -q 1 <server> 20098; done

The vast majority of times the experiment will succeed in sending 20 GB over the wire, but every once in a while (sometimes within minutes, sometimes within a few hours or even days) the transfer will get stuck or get cut short due to an unexpected disconnect/timeout. On some hosts I can reproduce the problem a lot more easily than on other hosts. The hosts where I can reproduce this more quickly tend to have more Docker containers and network activity, but I'm not sure yet if there's a causal relationship there. I was also able to reproduce the issue directly when running the above netcat experiment directly the host rather than within a Docker container, although it does seem a lot harder to reproduce this way. This happens on hosts within the same VPC, AZ, and even subnet so we can rule out cross-region/cross-AZ/cross-subnet connectivity issues as a cause.

Here's example tcpdump output that shows network activity when this happens. I'm skipping many successfully transmitted and ACK'ed TCP packets within this same connection. This information was captured with tcpdump -i eth0 -p -G 600 -s 80 -w ... host ... and port 20098. This is captured on the host's network interface, not inside the Docker network, so network address translations have already been applied.

Tcpdump output on host A (172.20.3.188, the sending client):

08:00:03.615061 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322435576:322444525, ack 1, win 491, options [nop,nop,TS val 4223113896 ecr 683441101], length 8949
08:00:03.615064 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322444525:322453474, ack 1, win 491, options [nop,nop,TS val 4223113896 ecr 683441101], length 8949
08:00:03.615066 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223113896 ecr 683441101], length 8949
08:00:03.615069 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322462423:322471372, ack 1, win 491, options [nop,nop,TS val 4223113896 ecr 683441101], length 8949
08:00:03.615071 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322471372:322480321, ack 1, win 491, options [nop,nop,TS val 4223113896 ecr 683441101], length 8949
08:00:03.615073 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322480321:322489270, ack 1, win 491, options [nop,nop,TS val 4223113896 ecr 683441101], length 8949
08:00:03.615076 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322489270:322498219, ack 1, win 491, options [nop,nop,TS val 4223113896 ecr 683441101], length 8949
08:00:03.615140 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322435576, win 256, options [nop,nop,TS val 683441101 ecr 4223113896], length 0
08:00:03.615178 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322453474, win 117, options [nop,nop,TS val 683441101 ecr 4223113896], length 0
08:00:03.824740 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223114105 ecr 683441101], length 8949
08:00:04.256748 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223114537 ecr 683441101], length 8949
08:00:05.084733 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223115365 ecr 683441101], length 8949
08:00:06.748724 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223117029 ecr 683441101], length 8949
08:00:10.108720 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223120389 ecr 683441101], length 8949
08:00:16.764722 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223127045 ecr 683441101], length 8949
08:00:30.076723 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223140357 ecr 683441101], length 8949
08:00:57.724718 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223168005 ecr 683441101], length 8949
08:01:50.972736 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223221253 ecr 683441101], length 8949
08:03:37.468722 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223327749 ecr 683441101], length 8949
08:05:38.304715 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223448585 ecr 683441101], length 8949
08:07:39.132913 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223569414 ecr 683441101], length 8949

Tcpdump output on host B (172.20.3.89, the receiving server):

08:00:03.615206 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322435576:322453474, ack 1, win 491, options [nop,nop,TS val 4223113896 ecr 683441101], length 17898
08:00:03.615225 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322498219, ack 1, win 491, options [nop,nop,TS val 4223113896 ecr 683441101], length 44745
08:00:03.615228 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322453474, win 117, options [nop,nop,TS val 683441101 ecr 4223113896], length 0
08:00:03.615256 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322498219, win 0, options [nop,nop,TS val 683441101 ecr 4223113896], length 0
08:00:03.615908 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322498219, win 1642, options [nop,nop,TS val 683441102 ecr 4223113896], length 0
08:00:03.616389 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322498219, win 3373, options [nop,nop,TS val 683441102 ecr 4223113896], length 0
08:00:03.618742 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322498219, win 6862, options [nop,nop,TS val 683441105 ecr 4223113896], length 0
08:00:03.621737 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322498219, win 13913, options [nop,nop,TS val 683441108 ecr 4223113896], length 0
08:00:03.824879 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223114105 ecr 683441101], length 8949
08:00:03.824905 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322498219, win 24576, options [nop,nop,TS val 683441311 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}], length 0
08:00:04.256895 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223114537 ecr 683441101], length 8949
08:00:04.256929 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322498219, win 24576, options [nop,nop,TS val 683441743 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}], length 0
08:00:05.084873 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223115365 ecr 683441101], length 8949
08:00:05.084908 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322498219, win 24576, options [nop,nop,TS val 683442571 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}], length 0
08:00:06.748872 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223117029 ecr 683441101], length 8949
08:00:06.748901 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322498219, win 24576, options [nop,nop,TS val 683444235 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}], length 0
08:00:10.108863 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223120389 ecr 683441101], length 8949
08:00:10.108889 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322498219, win 24576, options [nop,nop,TS val 683447595 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}], length 0
08:00:16.764877 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223127045 ecr 683441101], length 8949
08:00:16.764905 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322498219, win 24576, options [nop,nop,TS val 683454251 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}], length 0
08:00:30.076864 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223140357 ecr 683441101], length 8949
08:00:30.076881 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322498219, win 24576, options [nop,nop,TS val 683467563 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}], length 0
08:00:57.724863 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223168005 ecr 683441101], length 8949
08:00:57.724877 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322498219, win 24576, options [nop,nop,TS val 683495211 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}], length 0
08:01:50.972908 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223221253 ecr 683441101], length 8949
08:01:50.972922 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322498219, win 24576, options [nop,nop,TS val 683548459 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}], length 0
08:03:37.468882 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223327749 ecr 683441101], length 8949
08:03:37.468902 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322498219, win 24576, options [nop,nop,TS val 683654955 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}], length 0
08:05:38.304895 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223448585 ecr 683441101], length 8949
08:05:38.304942 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322498219, win 24576, options [nop,nop,TS val 683775791 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}], length 0
08:07:39.133073 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags [.], seq 322453474:322462423, ack 1, win 491, options [nop,nop,TS val 4223569414 ecr 683441101], length 8949
08:07:39.133092 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags [.], ack 322498219, win 24576, options [nop,nop,TS val 683896619 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}], length 0

Notice how host A stops receiving packets from host B after it receives the 08:00:03.615178 ... ack 322453474 packet.

Here is the output of VPC Flow Logs during a failed connection (captured at a different time than the tcpdump output above):

Given that Amazon Linux 2 doesn't seem to exhibit this problem I've tried to bring the network stack on Debian a bit more closely in line with Amazon Linux. I've tried to do the following on the Debian instances:

• Apply some of the network sysctl settings from Amazon Linux to Debian
• Upgrade the Linux kernel to 5.8.10
• Upgrade the ena driver to 2.2.11
• Upgrade Docker to 19.03.13
• Explicitly allow ingress and egress traffic to/from ephemeral ports (32768-65535) to/from all IPs within our VPC in the security group that these hosts use

None of these seem to resolve the issue I'm having. What could possibly cause these dropped/rejected packets?

I'm looking into setting up a shared filesystem/file server on AWS (EC2) infrastructure that offers replication and fairly painless failover. This filesystem would host potentially millions of files that are a few megs in size. Those files would be accessed (read/write) from several client VMs. If the primary file server fails I'd want the clients to be able to failover to the replica file server without losing any files (i.e. I want replication to be real-time). I've looked at a few options:

• Use S3 with s3fs. I'm concerned the latency of each request will be problematic when performing operations on thousands of files (e.g. when copying/moving files around). I've also heard some reports that make me question s3fs's stability—not sure if that's still the case.
• Setup an NFS server on an EC2 instance, using drbd to replicate blocks between two instances. Downsides:
  1. I've had reliability issues with drbd in the past, especially over high-latency links
  2. If the primary NFS server goes down it will take down the clients with it, requiring sysadmin intervention and/or reboots to get them to reconnect to the secondary server. There's no auto-failover.

Are there any better solutions?

I'm using Strongswan to handle IPsec connections, and need a way to support both Windows (IKEv2) and OS X (IKEv1) clients. I would prefer to use pure IPsec (i.e. avoid having to setup L2TP) unless there's a compelling reason to use L2TP/IPsec.

I know you can configure Strongswan in ipsec.conf to have some connections use IKEv1 whereas some other connections use IKEv2. However, in my case I can't easily setup separate connection configurations for each user because the IP address that they'll connect from won't be known in advance. I just use a single connection configuration for all of our users.

Is it possible to have some users connect via an IKEv1 connection configuration and others via an IKEv2 connection configuration, or setup a single connection configuration that'll handle both IKEv1 and IKEv2 connections? If not, what would be the easiest way to support both Windows and OS X built-in IPsec clients (with Strongswan or some other IPsec package)?

I'm considering virtualizing a number of guests onto a single server running a recent port of KVM to Illumos. It sounds like my two primary options will be OpenIndiana and SmartOS. The distribution I will end up using needs to meet the following requirements:

• Need to be able to manage and customize via CLI (e.g. change ZFS filesystem/zvol options, attach an external drive and copy data to it, or automatically replicate data to an offsite server using zfs send/receive).
• Need to implement automated ZFS snapshots (e.g. using zfs-auto-snapshot).
• Need to be able to setup automatic email notifications if the health of the server degrades. Essentially, setup periodic ZFS scrubbing, monitor zpool, fault manager, and/or SMART issues, and email when problems are detected, etc. Manually setting this up is OK, as long as the OS lets me.
• Should handle Debian, Ubuntu, and Windows 2008 Server R2 guests with good stability and reasonable performance. These guests will be used in production.
• There should be a reasonable expectation that future releases will continue to be delivered - I don't want to get stuck using a dead-end product.
• Would be nice if it was easy to setup and has some sort of GUI, but this is optional.

Based on these requirements, which distribution would you recommend?

You can assume that this environment won't be deployed until the upcoming OpenIndiana stable release is released. Also, you can assume the server will use a Sandy Bridge Xeon E3-1xxx CPU, so that should take care of KVM compatibility.

Also, how robust/stable is the KVM port to Illumos on either of these distributions? Should I even consider KVM/Illumos for a production environment for now?

I read in the XenServer Administrator's Guide in section 4.1.6.2 that you can map raw LUNs to VDIs if you're using a NetApp or EqualLogic SAN. Is it possible to do the same with Software iSCSI SANs (i.e. using a Nexenta/ZFS-based SAN as the iSCSI target)? I know I could setup an iSCSI SR and put VDIs on there, but I'd like to avoid it since I'd otherwise have to deal with the overhead of LVM, plus it'd make cloning more difficult. I'd handle the snapshotting and cloning from the SAN. So I'm basically just wanting to use a one-LUN-per-VDI mapping, without involving LVM inside the XenServer host.

I'm not opposed to setting this up through the CLI if necessary. I'd prefer something reliable that will persist across reboots of the XenServer host, but if I have to script something that will initialize the iSCSI sessions before the VMs start and the VDIs are loaded, I'm OK with that.

I'm planning on setting up a XenServer machine that uses a NexentaOS/ZFS-based SAN for storage of the Virtual Disk Images (VDIs) through iSCSI. I know I could just setup a big Storage Repository (SR) on the SAN and let XenServer take care of snapshotting and cloning disk images. However, I'd love to tap more into the power of ZFS and use that for snapshotting/cloning, for a few reasons:

1. I'm not sure how XenServer's snapshotting/cloning works, but if it's based on LVM, I'm concerned I'd run into issues when dealing with multiple snapshots. I've done some experiments a while ago with multiple LVM snapshots of the same data and performance seemed poor and the snapshots wasted a lot of space. It seems that ZFS snapshots are far superior to LVM snapshots.
2. The SAN would be taking automatic (and efficient) periodic ZFS snapshots that could go back in time a while, and I'd love to be able to revert a VM to such ZFS snapshot.

Would letting ZFS handle snapshotting/cloning instead of doing it through XenServer be advisable, and if so, what's the best way to go about it? If I put all VDIs inside a single large SR, and take ZFS snapshots of the entire SR, I would not be able to roll back an individual VM at a time. I could create one SR per VDI, and if I have to rollback a VDI, I'd carefully detach the SR, roll it back on the SAN, and re-attach it. However, I'm guessing that I'd run into problems when attaching a cloned SR if XenServer detects duplicate SR UUIDs. Are there any better ways to handle cloning or rolling back to previous snapshots from the SAN?

I would like to setup a diskless XenServer 5.6 FP 1 machine that boots from an iSCSI target. I think installation onto an iSCSI should be pretty easy: start the installer, open up a shell, use iscsiadm to connect to the iSCSI target, and have the installer install to it.

I'm not sure how to get the server to boot from the iSCSI target though. I'm thinking I could have the server's NIC perform a PXE boot to load gPXE, which in turn would load the boot loader from the iSCSI target, which would load Xen/Linux. However, how would I get Xen/Linux to mount the root filesystem from the iSCSI target?

Note that this server does not have an HBA iSCSI adapter, so I'm trying to boot it from a regular NIC.

I did a test installation inside a virtual machine. When using gPXE to boot from the iSCSI target, EXTLINUX just hangs. I then replaced the boot loader with GRUB, which was successful in booting the kernel. However, it was not able to mount the root filesystem. I see the following messages, which were printed after messages indicating that a DHCP lease was obtained successfully:

Attaching to iSCSI storage
iscsistart: transport class version 2.0-870. iscsid version 2.0-871
iscsistart: Portal Group not set. Exiting iscsistart
...
Waiting  for driver initialization.

I have a Cisco SPA942 phone, which is no longer supported by Cisco. It's running the latest firmware available for this phone: 6.1.5(a). Cisco replaced the SPA942 with the SPA504G, which is almost identical to the SPA942 hardware-wise. I noticed that the SPA504G has a newer firmware available (7.4.6) that adds some features that I'd like to have (a call ignore soft button and G.722 HD support).

I tried downloading and executing the 7.4.6 firmware upgrade tool, and going through the screens it didn't tell me I couldn't apply this SPA504G update to the SPA942. I stopped short of clicking the Upgrade button on the final screen before the upgrade, as I wasn't sure if this upgrade would brick the phone.

Is it safe to apply this SPA504G update to the SPA942?

I have two physical servers that I'd like to setup for virtualization. I'd like to set them up in such a way so that the following goals are met:

1. Allows live migration of guest OSes from one server to another, if I have to take one of the physical servers down for maintenance.
2. If the hardware of one of the servers dies, I can do a cold migration of its guest OSes to the other server (i.e. all data is sync at all times between the two physical servers).
3. Underlying storage for guest OSes can be efficiently snapshotted once every hour or once every day (e.g. using ZFS. I'm ruling out LVM, since taking multiple snapshots in LVM is inefficient).

Since OpenSolaris provides ZFS, I'm thinking using OpenSolaris/xVM as a dom0 would be a good fit. However, my understanding is that in order to allow live migration, you need to have another server (SAN) for shared storage, and I'd rather not add another server for cost reasons.

If it wasn't for the fact that ZFS isn't available for Linux (except through FUSE, which I'd rather avoid), I know I could set something like this up for DRBD. Could I do something similar to that with OpenSolaris? I've read about AVS, but I'm not sure if live migration can be made to work with that. Or if you have any other suggestions that use a different OS, I'm open to that as well.

Thanks!