TJ Zimmerman's questions

I am attempting to perform an OpenSCAP remediation through a chroot session. My command is structured as follows:

  oscap-chroot /mnt/chroot_fs \
        xccdf eval \
        --remediate \
        --results results.xml \
        --report report.html \
        --profile xccdf_org.ssgproject.content_profile_cui \
        ssg-rhel7-ds-1.2.xml

When the command executes, it performs a scan successfully indicating a handful of pass and fail results. Then, when it moves on to the remediation phase, it says:

 --- Starting Remediation ---
OpenSCAP Error: Can't perform remediation in offline mode: not implemented [/builddir/build/BUILD/openscap-1.3.3/src/XCCDF/xccdf_session.c:1690]

Is remediation through a chroot session not possible at this time? If not, how can you perform a remediation against a remote target?

I'm using Telmate's Terraform provider for Proxmox and trying to deploy Flatcar linux virtual machines using Cloud Init by passing files via cicustom. Based on their example, I have crafted the following Terraform file:

variable "pve_user" {
}
variable "pve_password" {
}
variable "pve_host" {
}

provider "proxmox" {
  pm_tls_insecure = true
  pm_api_url      = "https://SNIP/api2/json"
  pm_user         = "SNIP"
  pm_password     = "SNIP"
  pm_parallel     = 4
}

resource "null_resource" "cloud_init_config_files" {
  connection {
    type     = "ssh"
    user     = var.pve_user
    password = var.pve_password
    host     = var.pve_host
  }

  provisioner "file" {
    source      = "./templates/cloud-config.yml"
    destination = "/var/lib/vz/snippets/cloud-config.yml"
  }
}

resource "proxmox_vm_qemu" "k8s-masters" {
  depends_on = [
    null_resource.cloud_init_config_files
  ]

  count      = 1
  name       = "VM-${count.index}"
  clone      = "VM-Template"
  full_clone = true

  target_node = "192.168.20.10"
  pool        = "VM"

  cores   = 2
  sockets = 1
  memory  = 10240

  network {
    id     = 0
    model  = "virtio"
    bridge = "vmbr0"
    tag    = 50
  }

  disk {
    id           = 0
    type         = "scsi"
    size         = 30
    storage      = "Pool"
    storage_type = "zfspool"
    backup       = true
    iothread     = true
  }

  onboot = true
  agent  = 1

  os_type         = "cloud-init"
  ssh_user        = "core"
  cicustom  = "user=local:snippets/cloud-config.yml"
  ipconfig0 = "ip=192.168.50.10/24,gw=192.168.50.1"

  sshkeys = "ssh-rsa SNIP"
  ssh_private_key = <<EOF
-----BEGIN RSA PRIVATE KEY-----
SNIP
-----END RSA PRIVATE KEY-----
EOF
}

After running terraform apply, the VM is successfully created and Flatcar is bootstrapped. If you connect to the console via Proxmox, however, the VM shows that the IP Address is one received via DHCP instead of the one I provided. Furthermore, the SSH key does not work so I am unable to connect to the VM to troubleshoot.

If I comment out the cicustom line, and simply rely on ipconfig0 and the other normal options, the VM comes up and my SSH key does in fact work. However, the specified IP address is still not used; the VM just uses one provided by DHCP instead. Despite being able to access the VM, I still want to use a custom Cloud Init config file so I can have access to more powerful configuration options.

I've tried various combinations of my cloud-config.yml file. Including things as simple as:

hostname: test

to things more detailed like:

storage:
  files:
    - path: /opt/file1
      filesystem: root
      contents:
        inline: Hello, world!
      mode: 0644
      user:
        id: 500
      group:
        id: 501
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-rsa SNIP

Despite all of this, my SSH keys never work after I pass something to cicustom and, based on the fact that the hostname is never manipulated, I'm assuming Cloud Init just outright isn't receiving the custom config file at all. I've tried passing in both Ignition and Container Linux Config formatted files.

Is Flatcar Linux broken with Proxmox/cicustom? Searching Google for things like "flatcar" "proxmox", "coreos" "proxmox", "container linux" "proxmox", "flatcar" "cicustom", etc don't seem to turn up much results. I guess there aren't a lot of people out there bridging Cloud Native operating systems with bare metal? hehehe

What does come up, however, is this interesting script. Lines 104-132 discuss creating a Flatcar/CoreOS template for Proxmox. So someone else has done this before, at least? Who knows if they passed in a custom cloud init config file afterwards?

Any ideas?

EDIT: ------------------------------------------------------------

Added logs from user-configdrive systemd unit when not passing cicustom

Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Checking availability of "cloud-drive"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Fetching user-data from datasource of type "cloud-drive"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Attempting to read from "/media/configdrive/openstack/latest/user_data"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 line 6: warning: unrecognized key "chpasswd"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 line 9: warning: incorrect type for "users[0]" (want struct)
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 line 10: warning: unrecognized key "package_upgrade"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Fetching meta-data from datasource of type "cloud-drive"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Attempting to read from "/media/configdrive/openstack/latest/meta_data.json"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Attempting to read from "/media/configdrive/openstack/content/0000"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Parsing user-data as cloud-config
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Merging cloud-config from meta-data and user-data
Aug 20 16:22:17 VM-0 coreos-cloudinit[831]: 2020/08/20 16:22:17 Set hostname to VM-0
Aug 20 16:22:17 VM-0 coreos-cloudinit[831]: 2020/08/20 16:22:17 Authorized SSH keys for core user
Aug 20 16:22:17 VM-0 coreos-cloudinit[831]: 2020/08/20 16:22:17 Failed to apply cloud-config: Invalid option to manage_etc_hosts
Aug 20 16:22:17 VM-0 systemd[1]: user-configdrive.service: Main process exited, code=exited, status=1/FAILURE

EDIT 2: ----------------------------------------------------------

Added the contents of the files referenced by the above Systemd unit.

core@VM-0 ~ $ cat /media/configdrive/openstack/latest/user_data
#cloud-config
hostname: VM-0
manage_etc_hosts: true
ssh_authorized_keys:
  - REDACTED
chpasswd:
  expire: False
users:
  - default
package_upgrade: true

As you pointed out, manage_etc_hosts is set to true here by Proxmox.

core@VM-0 ~ $ cat /media/configdrive/openstack/content/0000
auto lo
iface lo inet loopback

        dns_nameservers 192.168.1.100
        dns_search example.com
auto eth0
iface eth0 inet static
        address 192.168.50.10
        netmask 255.255.255.0
        gateway 192.168.50.1

This section contains all of the correct network configuration, however it is all overridden by DHCP so it's ignored.

core@VM-0 ~ $ cat /media/configdrive/openstack/latest/meta_data.json
{
     "uuid": "e61563c9057e9162c4e14d111fea171379170532",
     "network_config": { "content_path": "/content/0000" }
}

EDIT 3: ----------------------------------------------------------

Added my attempt at a custom Cloud Init file

manage_etc_hosts: false
hostname: "test"
ssh_authorized_keys:
  - REDACTED

EDIT 4: ----------------------------------------------------------

I followed this guide to basically accomplish the same thing directly with qemu instead of using Proxmox. Using:

This cloud-config.ign:

{
  "ignition": { "version": "2.2.0" },
  "passwd": {
    "users": [
      {
        "name": "core",
        "sshAuthorizedKeys": [
          "REDACTED"
        ]
      }
    ]
  },
  "storage": {
    "files": [{
      "filesystem": "root",
      "path": "/etc/hostname",
      "mode": 420,
      "contents": { "source": "data:,test" }
    }]
  }
}

OR this cloud-config.yml:

hostname: test
passwd:
  users:
    - name: tj
      ssh_authorized_keys:
        - REDACTED

both produce the same result. An unconnectable virtual machine that spins up with the hostname localhost.

• Here are the logs that were produced using cloud-config.ign.
• Here are the logs that were produced using cloud-config.yml.

EDIT 5: ----------------------------------------------------------

These logs are produced by user-configdrive.service when using Telmate's provider against a Flatcar template.

Aug 21 21:41:50 localhost coreos-cloudinit[795]: 2020/08/21 21:41:50 Checking availability of "cloud-drive"
Aug 21 21:41:50 localhost coreos-cloudinit[795]: 2020/08/21 21:41:50 Fetching user-data from datasource of type "cloud-drive"
Aug 21 21:41:50 localhost coreos-cloudinit[795]: 2020/08/21 21:41:50 Attempting to read from "/media/configdrive/openstack/latest/user_data"
Aug 21 21:41:50 localhost coreos-cloudinit[795]: 2020/08/21 21:41:50 Fetching meta-data from datasource of type "cloud-drive"
Aug 21 21:41:50 localhost coreos-cloudinit[795]: 2020/08/21 21:41:50 Attempting to read from "/media/configdrive/openstack/latest/meta_data.json"
Aug 21 21:41:50 localhost coreos-cloudinit[795]: 2020/08/21 21:41:50 Attempting to read from "/media/configdrive/openstack/content/0000"
Aug 21 21:41:50 localhost coreos-cloudinit[795]: Detected an Ignition config. Exiting...
Aug 21 21:41:50 localhost systemd[1]: Started Load cloud-config from /media/configdrive.

EDIT 6: ----------------------------------------------------------

Here is the entire Systemd journal from the first output after using the cloud-config.ign in EDIT 4 with Telmate's provider.

EDIT 7: ----------------------------------------------------------

These qm commands produce a working Flatcar VM using cicustom.

qm create 101 --name test --cores 2 --memory 2048 --net0 "virtio,bridge=vmbr0"

qm set 101 --net0 "virtio,bridge=vmbr0,tag=50"

qm importdisk 101 /mnt/RAIDPool_Templates/template/iso/flatcar_production_qemu_image.img FlashPool

qm set 101 --scsihw virtio-scsi-pci --scsi0 FlashPool:vm-101-disk-0 --ide2 FlashPool:cloudinit --boot c --bootdisk scsi0 --serial0 /dev/tty0 --ipconfig0 ip=dhcp --citype configdrive2

qm set 101 --cicustom user=RAIDPool_Templates:snippets/user-data.yml

qm start 101
ssh -i ~/.ssh/sol.milkyway.kubernetes [email protected]

user-data.yml:

#cloud-config
hostname: test
manage_etc_hosts: true
ssh_authorized_keys:
  - ssh-rsa REDACTED
chpasswd:
  expire: False
users:
  - default
package_upgrade: true

Passing the same user-data.yml to cicustom within the proxmox_vm_qemu resource doesn't seem to be working though.

TL;DR: I'm having some performance issues with my hypervisor storage. here are a bunch of test results from fio. Skip to the Results section to read about them and see my questions.

Summary

I recently purchased an R730xd so before I migrate over to it I wanted to be sure that storage was performing opmtimally. I've been running some benchmark tests with fio and have found some alarming results. Using a combination of those results and fio-plot, I've amassed a pretty large collection of graphs and charts that demonstrate the issues across my various storage backends.

However, I'm having a hard time turning it into usable information because I don't have anything to compare it to. And I think I'm having some very strange performance issues.

Disk Configuration

Here are the four types of storage exposed to my hypervisor (Proxmox):

╔═══════════╦════════════════════════════════╦═════════════╦════════════════════════════╗
║  Storage  ║            Hardware            ║ Filesystem  ║        Description         ║
╠═══════════╬════════════════════════════════╬═════════════╬════════════════════════════╣
║ SATADOM   ║ 1x Dell K9R5M SATADOM          ║ LVM/XFS     ║ Hypervisor filesystem      ║
║ FlashPool ║ 2x Samsung 970 EVO M.2 SSD     ║ ZFS RAID 1  ║ Hypervisor Compute Storage ║
║ DataPool  ║ 6x HGST 7200RPM HDD            ║ ZFS RAID 10 ║ Redundant Data Storage     ║
║ RAIDPool  ║ 6x Seagate/Hitachi 7200RPM HDD ║ HW RAID 10  ║ General Purpose Storage    ║
╚═══════════╩════════════════════════════════╩═════════════╩════════════════════════════╝

Storage Details

Here is a more detailed breakdown for each storage backend:

1. SATADOM: The SATADOM is managed directly by Proxmox via LVM. Here is the output from lvdisplay pve. The SATADOM is connected to the server via the internal DVD-ROM SATA port as it is unused in the R730xd model.

2. FlashPool: The FlashPool is a simple ZFS RAID 1 comprised of dual NVMe SSDs. The goal is to use this as backing storage for my VMs. Here are the outputs for:

    zpool list  
    zpool status  
    zfs get all
   

   Each of the SSDs in the FlashPool are connected to the server via PCI-E -> M.2 adapters installed in x16 PCIe slots. I recognize that these are x4 PCIe adapters. However, I'm pretty sure NVMe only operates at that speed so faster adapters are not manufactured.

3. DataPool: The DataPool is the only pre-existing dataset. It is a couple years old and was previously used for both Data and VM storage to the detriment of performance. It is also managed by Proxmox as a ZFS RAID 10.

   It was originally comprised of 6x 4TB HGST Ultrastar 7K4000 7200RPM disks. However, as they started to fail I decided to replace them with higher density disks. As a result, the array now consists of:

    2x 6TB HGST Ultrastar He6 7200RPM  
    4x 4TB HGST Ultrastar 7K4000 7200RPM 
   

   I obviously intend to eventually move entirely to 6TB disks as the older ones continue to fail. Here are the outputs for the same commands posted above for the FlashPool.

   These 6 disks are connected to the server via the first 6 bays on the backplane. This backplane is connected to a Dell H730 Mini PERC RAID Controller.

4. RAIDPool: The RAIDPool is an experimental storage backend. I've never worked with hardware RAID before so I was excited for the opportunity now that I have a proper RAID Controller. Similar to the DataPool, these disks are installed in the last 6 bays on the backplane. However, instead of being passed through to Proxmox, they are managed by the PERC. They are presented to Proxmox as a single disk which is then managed by LVM and presented to the OS via logical volumes as XFS filesystems. Here is the output from lvdisplay RAIDPool.

RAID Controller Configuration

So, you may have just noticed that both the DataPool and RAIDPool are installed and managed by the H730 RAID Controller. However, the DataPool is managed by Proxmox via ZFS and the RAIDPool is managed by the actual controller.

Here is a screenshot of the topology of the physical disks. The H730 is capable of passing disks directly through to the OS and simultaneously managing other disks. As you can see, The first 6 disks are configured in Non-RAID mode and the last 6 disks are configured in Online mode.

• Here are the configured properties for the controller from within the iDRAC UI.
• Disk Cache is enabled for both Write Back and Read Ahead on the Virtual Disk (RAIDPool). Since this is configured specifically for the VD, it should not impact the ZFS drives.
• Dick Cache for Non-RAID disks (ZFS DataPool) is set to Disable.
• The Link Speed for all of the drives is set to auto.

Also, after stepping through all of the settings once more, I enabled Write Cache for the Embedded SATA Controller. So this might improve performance on the SATADOM from what is seen in the benchmarks below.

Benchmarking:

I have benchmarked all of these storage backends in two ways. For both tests, I ran a series of fio-plot commands in a small shell script that dumped the results in a few folders.

If you're crazy and want to parse through the raw results on your own, here they are. You will need to massage my scripts a bit to rerun then since I moved around the directory structure to organize it prior to uploading it.

In a nutshell, they ran a series of tests against each storage backend that evaluated its RANDOM bandwidth, IOPS, and latency. It then plotted these results on graphs. Some graphs compare multiple backends. Other graphs simply show results from individual backends. I did not perform any SEQUENTIAL tests. In all cases, the default block size was used for the test.

Test 1) From within Proxmox, I mounted all of the storage backends into the /mnt directory. The ZFS Pool were simply imported into the OS and both the RAIDPool and the SATADOM were presented to the OS via LVM. Each had a Logical Volume formatted as an XFS partition that was used for benchmarking. NOTE: I ran these benchmarks from the live OS so the performance of the SATADOM will be affected accordingly.

• Comparison Graphs
• Individual Graphs

The log files were generated using these commands:

./bench_fio --target /mnt/SATADOM_Data/bm --type directory --size 450M --mode randread randwrite --output SATADOM
./bench_fio --target /mnt/RAIDPool_Data/bm --type directory --size 1G --mode randread randwrite --output RAIDPOOL
./bench_fio --target /mnt/DataPool/bm/ --type directory --size 1G --mode randread randwrite --output DATAPOOL
./bench_fio --target /mnt/FlashPool/bm/ --type directory --size 1G --mode randread randwrite --output FLASHPOOL

Test 2) I created three VMs in Proxmox. Each of which used a different backing storage from the FlashPool, DataPool, and RAIDPool. The FlashPool and DataPool VMs ran in their own ZFS dataset. The RAIDPool VM ran on its own thick-provisioned Logical Volume. All three VMs were given 4 vCPUs and 40GB of memory.

• Comparison Graphs
• Individual Graphs

The log files were generated using these commands:

./bench_fio     --target /fio     --type file     --size 1G     --mode randread randwrite     --duration 600     --output DATAPOOL_VM
./bench_fio     --target /fio     --type file     --size 1G     --mode randread randwrite     --duration 600     --output RAIDPOOL_VM
./bench_fio     --target /fio     --type file     --size 1G     --mode randread randwrite     --duration 600     --output FLASHPOOL_VM

Results:

The graphs in the above Imgur links should all be in the same order. The results from the two benchmarks are quite a bit different. But that is to be expected when you factor in the overhead from virtualization. What is NOT expected to me, is that they all seem to behave about the same.

• For example, this chart shows that when fio was run from within a VM, the average write bandwidth was somewhere around 125 MB/s. Shouldn't the two NVMe SSDs in RAID 1 (FlashPool) MASSIVELY outperform the SATADOM? Instead, you can see that the FlashPool VM took the LONGEST amount of time to complete the test and had the slowest average write bandwidth. The same situation can be seen for the Write IOPS comparison -- The average IOPS were around 3,000 and the FlashPool VM took the longest to execute the test!

• Stepping away from the benchmarks taken from WITHIN a VM, and instead looking at those taken by directly interacting with the storage from the hypervisor, we can see some different behavior. For example, in this test the write bandwidth for the FlashPool and DataPool was as high as 400MB/s. However, the performance for the RAIDPool averaged around 10MB/s. Which coincidentally, was about the same as the SATADOM? Surely, the RAIDPool should have performed compatible to, if not better than, the DataPool? Given that they are comprised of similar disks present in the same RAID Controller? Similar to above, the Write IOPS show the same bizarre story.

• The Write Latency from the Hypervisor tests also appears to be unusual. The RAIDPool appears to be experiencing up to ten times worse latency than the ZFS Pools? However, if you flip over to the VM tests, the latency for the three storage backends seems to congregate around 300us. Which is pretty similar to what we were seeing in the WORST cast for the RAIDPool. Why does this smooth effect happen to write latency when the tests are run from VMs instead of hypervisor? Why does the latency for the ZFS Pools suddenly become so much worse, and comparable to, the RAIDPool?

• Looking at the read Bandwidth, IOPS, and latency shows a similar story. All metrics are equally slow, despite having massively different hardware configurations, when benchmarked from within a VM. However, once benchmarked from the hypervisor, the ZFS Pools suddenly greatly outperform everything else?

  • Read Bandwidth 1

  • Read Bandwidth 2

  • Read IOPS 1

  • Read IOPS 2

  • Read Latency 1

  • Read Latency 2

Questions:

1. These results are abnormal... right? This benchmark from this website shows a 970 EVO achieving upwards of 900MB/s random write speeds. Why are mine only coming in at 150MB/s on the hypervisor and 10MB/s in a VM? Why are these speeds so different when benchmarked from the hypervisor and from a VM?

2. Why does the RAIDPool suddenly become abnormally slow when benchmarked from the Hypervisor? Here we see that the Read Bandwidth in a VM averages 20MB/s. However, from the hypervisor, it instead reports 4MB/s. Just like the benchmark tests I showed in question 1, shouldn't these read speeds be closer to 900MB/s?

3. Why do the ZFS Pools suddenly perform significantly worse when benchmarked from within a VM instead of the hypervisor? For example, here we can see that the read IOPS averaged around 200,000 and latency under 650us. However, when benchmarked from within a VM, we can suddenly see that the read IOPS average around 2,500 and the latency more than quadrupled? Shouldn't the performance in both situations be about the same?

Summary

I am attempting to bootstrap a Kubernetes cluster on AWS using Kubeadm. Please before you suggest them, I am not interested in using EKS or another bootstrapping solution like Kops, Kubespray, etc.

It appears that there is a lot of inaccurate information about the proper procedures out there due to a schism with respect to Cloud Provider integrations not being managed out of tree rather in-tree. So I've been struggling to get a clear picture in my head about how to properly set this integration up.

The Requirements

The official repo indicates three requirements.

1) You must initialize kubelet, kube-apiserver, and kube-controller-manager with the --cloud-provider=external argument. If I understand things correctly, this allows you to use the out of tree provider. Using aws here instead would use the in-tree provider which is on a deprecation timeline.

2) You must create two IAM policies, associate them with IAM Instance Profiles, and launch your Kubernetes nodes with said policy attached.

3) Each node in the cluster must have the same hostname that is associated with the underlying EC2 instance as its Private DNS name.

In addition to this, I believe it was once required to attach the following Tags to your EC2 instances, Route Tables, Security Groups, and Subnets. Which I have done for good measure as well:

"kubernetes.io/cluster/${var.K8S_CLUSTER_NAME}" = "kubernetes.io/cluster/${var.K8S_CLUSTER_NAME}"

The Problem

Despite this, however, when my worker nodes come online after bootstrapping they have the following taint applied:

node.cloudprovider.kubernetes.io/uninitialized: true

This obviously implies that the nodes have not been initialized by the Cloud Provider. I'm not really sure where to go from here. There is an open request for additional instructions on how to use the Cloud Provider integration with AWS but it is currently unsatisfied.

My Configuration

You might have noticed I left a comment on that issue detailing my issue as well. Here is a summary of the details of my environment showing that I should be in compliance with the listed requirements.

1) My Kubeadm config files set the cloud provider to external in four places

KubeletConfiguration and InitConfiguration

nodeRegistration:
  kubeletExtraArgs:
    cloud-provider: external

ClusterConfiguration

apiServer:
  extraArgs:
    cloud-provider: external

ClusterConfiguration

controllerManager:
  extraArgs:
    cloud-provider: external

2) My EC2 instances were launched with an instance profile with the IAM policies outlined in the README:

$> aws ec2 describe-instances --instance-ids INSTANCE.ID | jq '.Reservations[].Instances[].IamInstanceProfile[]'
"arn:aws-us-gov:iam::ACCOUNT.ID:instance-profile/PROFILE-NAME"

3) The hostnames are the EC2 Private DNS names:

$> hostname -f
ip-10-0-10-91.us-gov-west-1.compute.internal

4) The EC2 instances as well as my route tables, subnets, etc are tagged with:

"kubernetes.io/cluster/${var.K8S_CLUSTER_NAME}" = "kubernetes.io/cluster/${var.K8S_CLUSTER_NAME}"

As a result, it looks like I am in compliance with all of the requirements so I am unsure why my nodes are still left with that Taint. Any help would be greatly appreciated!

EDIT

I have updated the tags on each instance to:

"kubernetes.io/cluster/${var.K8S_CLUSTER_NAME}" = "owned"

And added this tag to each Subnet:

"kubernetes.io/role/internal-elb" = 1

This has not resolved the situation, however.

EDIT 2

A user elsewhere suggested that the issue may be that I didn't apply the RBAC and DaemonSet resources present in the manifests directory of the cloud-provider-aws repo. After doing so using this image, I can confirm that this has NOT resolved my issue since the aws-cloud-controller-manager appears to expect you to be using aws not external` as per the logs produced by the pod on startup:

Generated self-signed cert in-memory

Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.

Version: v0.0.0-master+$Format:%h$

WARNING: aws built-in cloud provider is now deprecated. The AWS provider is deprecated and will be removed in a future release

Building AWS cloudprovider

Zone not specified in configuration file; querying AWS metadata service

Cloud provider could not be initialized: could not init cloud provider "aws": clusterID tags did not match: "example-14150" vs "True"

EDIT 3

I built a new image using the repo as of commit 6a14c81. It can be found here. It appears to also be using the aws provider by default?

Cloud provider could not be initialized: could not init cloud provider "aws": clusterID tags did not match: "example-14150" vs "True"

I recently purchased a Dell R730xd that came with an H730 RAID card. I would like to bring an existing 6 disk ZFS pool to this server as well as create an additional 6 disk hardware RAID.

Is it possible to configure an H730 RAID card to operate in both HBA and RAID mode simultaneously? And then only manage 6 of the disks in a hardware RAID and allow the OS to manage the remaining 6 disks in a software RAID?

If not, can a Dell R730xd use both an H730 RAID card and an LSI 9207-8i HBA card simultaneously? Would you simply plug one SAS cable from each card into each part of the Backplane? Will I need to purchase any new cables to connect the LSI 9207-8i to the backplane in my R730xd?

Is there a better card to use for ZFS in an R730xd using spinning disks than the LSI 9207-8i?

I have provisioned a Kubernetes cluster on RHEL 7 and deployed Calico to it. The calic-node pods running on the master and worker nodes do not become ready until I SSH into the master and temporarily disable iptables with systemctl stop iptables.

I have configured iptables on the master to allow incoming access over TCP to 6443, 2379:2380, and 10250:10252. So it should be configured correctly as per the documented required ports. After all, the nodes are able to join the cluster via kubeadm join which requires the ability to network with the master over 6443.

I'm trying to bootstrap a Kubernetes cluster on RHEL 7.8 but I'm having some issues with my firewall.

nftables is not supported in Kubernetes and iptables-legacy must be installed instead. While the iptables-legacy package exists in distros like Debian Buster, it does not seem to be available for RHEL 7. However, this article mentions installing iptables-services, disabling firewalld, and enabling iptables. The relevant material from the article is:

yum install iptables-services.x86_64 -y
systemctl stop firewalld.service
systemctl disable firewalld.service
systemctl mask firewalld.service
systemctl start iptables
systemctl enable iptables
systemctl unmask iptables
iptables -F
service iptables save

After this, if I run iptables --version on the server, I can see that 1.4.2 is installed. Since this is older than 1.8 as implied by the GitHub issue linked above, this version should be fine.

Before running kubeadm join from my worker nodes, the following Ansible tasks run against my master to configure iptables:

- iptables:
    chain: INPUT
    destination_port: "{{ port }}"
    jump: ACCEPT
    protocol: tcp
  loop:
    - 6443
    - 2379:2380
    - 10250:10252
  loop_control:
    loop_var: port

- command: service iptables save

- systemd:
    name: iptables
    state: restarted

And this against my nodes to configure iptables:

- iptables:
    chain: INPUT
    destination_port: "{{ port }}"
    jump: ACCEPT
    protocol: tcp
  loop:
    - 10250
    - 30000:32767
  loop_control:
    loop_var: port

- command: service iptables save

- systemd:
    name: iptables
    state: restarted

After this, I can confirm that the rule is present in memory:

$> iptables -S | grep 6443
-A INPUT -p tcp -m tcp --dport 6443 -j ACCEPT

Then, when I run kubeadm join from the worker node, it fails to connect:

I0406 22:07:19.205714    5715 token.go:73] [discovery] Created cluster-info discovery client, requesting info from "https://192.168.50.10:6443"
I0406 22:07:19.206720    5715 token.go:78] [discovery] Failed to request cluster info: [Get https://192.168.50.10:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s: dial tcp 192.168.50.10:6443: connect: no route to host]
I0406 22:07:19.206749    5715 token.go:191] [discovery] Failed to connect to API Server "192.168.50.10:6443": Get https://192.168.50.10:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s: dial tcp 192.168.50.10:6443: connect: no route to host
I0406 22:07:24.207648    5715 token.go:188] [discovery] Trying to connect to API Server "192.168.50.10:6443

However, if I systemctl stop iptables on the master then the worker nodes can join without any issues. Indicating to me that the firewall on the master is misconfigured?

I have PowerDNS Authoritative & Recursor running on a server in my environment. When I use this node as my a DNS Server, it responds to A record queries without any issues. However, when I attempt to perform a reverse lookup it fails. Any idea why?

Here is a screenshot of some DNS Lookups exhibiting the behavior.

Here is a screenshot depicting the A record configuration for Mars in the Forward Lookup Zone.

Here is a screenshot depicting an automatically created PTR record for Mars in the Reverse Lookup Zone.

Here is the pdns.conf file.

Here is the recursor.conf file.

EDIT: I should mention that Pi-Hole sits in front & relays requests to PowerDNS. However, if I instruct dig to perform lookups directly against PowerDNS the results are the same:

Forward:

$> dig mars.sol.milkyway @192.168.1.110

; <<>> DiG 9.10.6 <<>> mars.sol.milkyway @192.168.1.110
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19842
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;mars.sol.milkyway.     IN  A

;; ANSWER SECTION:
mars.sol.milkyway.  2812    IN  A   192.168.30.10

;; Query time: 3 msec
;; SERVER: 192.168.1.110#53(192.168.1.110)
;; WHEN: Sun Oct 06 18:35:50 PDT 2019
;; MSG SIZE  rcvd: 62

Reverse:

$> dig -x 192.168.30.10 @192.168.1.110

; <<>> DiG 9.10.6 <<>> -x 192.168.30.10 @192.168.1.110
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 32029
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;10.30.168.192.in-addr.arpa.    IN  PTR

;; AUTHORITY SECTION:
168.192.in-addr.arpa.   1687    IN  SOA localhost. root. 1 604800 86400 2419200 604800

;; Query time: 4 msec
;; SERVER: 192.168.1.110#53(192.168.1.110)
;; WHEN: Sun Oct 06 18:31:53 PDT 2019
;; MSG SIZE  rcvd: 104

I have a home Kubernetes cluster that runs in 4 VMs on top of Proxmox. Proxmox is tagged to VLAN 20, the Kubernetes VMs are tagged to VLAN 40.

The Kubernetes VMs are BGP neighbors of my router so that I can tag pods to then run on one of two other VLANs that are designated as DMZ spaces, 50 and 60. In short, the network looks like this:

- VLAN1: Networking Hardware
    - VLAN20: Physical Machines
        - VLAN40: Kubernetes VMs
            - VLAN50: Internal Kubernetes Deployments
            - VLAN60: External Kubernetes Deployments

This works great, everything is able to communicate with one-another and the internet just fine. With one exception, performance.

My Proxmox server also acts as my storage server by advertising a ZFS pool as an NFS server. This works great, and is capable of some pretty fast reads and writes for a home storage server. Upwards of 6Gb/s reads, for example.

When I used to run Docker containers directly on my Proxmox server, virtual switching allowed the containers to interact with the NFS server hosted by Proxmox by hostname at nearly that speed.

Furthermore, before I set up VLANs, the Kubernetes VMs used to run on the same VLAN (1) as Proxmox itself. And any pods that were deployed on Kubernetes were also able to interact with the NFS server hosted by Proxmox by hostname at nearly that speed.

However, now that I have configured VLANs and use BGP to provision my Kubernetes pods on separate VLANs from the hosts, networking has been capped at 1Gb/s, if not worse than that.

My Ubiquiti Edgerouter Lite and Unifi Switch 8 are both 1Gb devices, so it makes sense. However, this is starting to feel very painful in my lab. For example, cover art in Plex Media Server takes upwards of 10 seconds to load when I scroll in my library because Kubernetes volume mounts the database on the NFS server. Similarly, Deluge is acting incredibly poorly. The web interface crashes frequently and any sort of action such as opening the Preferences panel or trying to see the Details section of a new torrent can take several minutes! Deluge's cache settings are set to use 4GB of memory, but I'm unsure if these performance issues are because of my network or because Deluge just doesn't scale well to 1100 torrents. Lastly, sometimes my Kubernetes deployments that interact heavily with a database (Plex, Jira, etc) end up with a corrupted database after a few weeks of running. This is presumably because of network latency, but I'm not sure.

I'm looking for a few questions to be answered with this post:

1. I know my network is complex, especially for a homelab. However, my homelab is used pretty much entirely for learning for my job. And the hobby is fun for me, especially when I cater to obscene levels of complexity. However, I'm just curious if everything seems like it is configured correctly to you, given the fact that I am okay with the complexity.

2. Would purchasing a 10Gb switch resolve this issue or would it also be necessary to purchase a 10Gb router since the Edgerouter is a BGP neighbor of the Kubernetes nodes?

3. If it would be necessary to purchase both a Switch and a Router, would it instead be possible to purchase a 10Gb switch with BGP capabilities?

4. What hardware would you recommend I purchase to resolve this issue? Ideally I would like to keep the total cost under $500-1,000 but it doesn't look like that would be possible given the incredibly high cost of 10Gb routers.

5. Would it be possible to use a different Kubernetes Storage Class for storing the data directly on the nodes? What would this look like?

6. Would you recommend a different solution to my problem?

I am having what appears to be a DNS related issue that I would appreciate some assistance resolving.

I'm using Ansible to provision a Kubernetes cluster on my Proxmox server. The project works in two ways, by letting the user modify the site.yml to deploy using Linux Containers (LXC) or Virtual Machines from a CentOS7 qcow2 image.

When deploying with LXC, the project experiences no issues and correctly bootstraps a Kubernetes cluster. However, when using the qcow2 image, I encounter what appears to be a DNS related issue. This occurs when the changeover happens between the playbook that provisions my virtual machines, and the one that connects to them for the first time to prepare them.

What happens, is that the Gathering Facts stage eventually timeouts out and Ansible throws the following error:

TASK [Gathering Facts] *******************************************************************************************************************************************************************************************************************************************************
fatal: [pluto.sol.milkyway]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host pluto.sol.milkyway port 22: Operation timed out\r\n", "unreachable": true}
fatal: [ceres.sol.milkyway]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host ceres.sol.milkyway port 22: Operation timed out\r\n", "unreachable": true}
fatal: [eris.sol.milkyway]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host eris.sol.milkyway port 22: Operation timed out\r\n", "unreachable": true}
fatal: [haumea.sol.milkyway]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host haumea.sol.milkyway port 22: Operation timed out\r\n", "unreachable": true}

If, after this occurs, I try to manually SSH into the servers, I can verify that SSH is taking a very long time to connect. I would like to remind you at this point that this does NOT occur with the LXC instances that use the same exact hostnames, IP addresses, and name servers.

The issue can then be resolved by setting the UseDNS no directive in my sshd_config file on each of the servers. And running the playbook again after restarting the sshd.service.

So, naturally, this looks like a DNS issue. However, since it doesn't occur with LXC I'm skeptical. So here are a few more data points about my DNS configuration.

1) The DNS server that they're all using is BIND and is installed on a server named IO.Sol.Milkyway at 192.168.1.10. There are no VNets or Subnets or anything in my homelab, and the Gateway is correctly set to my router, 192.168.1.1 so there are no routing issues to this server.

2) Here are the relevant parts of the DNS zones on my BIND server.

• Forward Lookup Zone
• Reverse Lookup Zone

3) Here are some nslookups performed from the Proxmox server and appended with the time command to demonstrate that my BIND server responds correctly in <= .01 seconds.

$> time nslookup pluto.sol.milkyway
Server:     192.168.1.100
Address:    192.168.1.100#53

Name:   pluto.sol.milkyway
Address: 192.168.1.170

nslookup pluto.sol.milkyway  0.00s user 0.02s system 39% cpu 0.042 total

-and-

$> time nslookup 192.168.1.170
Server:     192.168.1.100
Address:    192.168.1.100#53

170.1.168.192.in-addr.arpa  name = pluto.sol.milkyway.

nslookup 192.168.1.170  0.01s user 0.01s system 96% cpu 0.013 total

4) And, lastly, you can see that my nameservers are correctly configured on the VMs via cloud-init lines 104, 115, 126, & 137 here. Which reference the variables defined here.

-----EDITS BELOW-----

5) I'm able to successfully perform a forward and reverse nslookup from the following. Each response takes < 1.5 seconds:

• My personal workstation (Executes Ansible)
• My Proxmox Server (Runs Ansible Commands & VMs)
• The 4 Virtual Machines

Here is an example from what would be the Kubernetes Master server.