Questions[calico](server)

My current setup involves an EKS Cluster with multiple namespaces (multi-tenant) across many different EKS nodes in private subnets. I would like the egress traffic from the pods to have a dedicated EIP per namespace. AFAIK there are no off the shelf solutions available for this problem. I have searched long and hard on the internet but in vain. Here are some of the solutions that I have tired, but eventually hit a roadblock.

Istio Egress Gateway Allows you to direct all outbound traffic from pods via a dedicated egress gateway pod. But the egress gateway pod will assume the ip of the node on which it is running, which will not work for my use case. Also, I have not found decent documentation around how I can setup multiple egress gateway pods across multiple nodes.

Calico Egress Gateway Very similar to Istio Egress Gateway solution and the same constraint applies

Custom Solution I have started implementing a custom solution, where I setup multiple Gateway Nodes in public subnet each with a dedicated EIP. I can now modify the IP routes/gateways in the private EKS node, to route traffic via a specific gateway node based on the pod source IP. This solution feels very kludgy and the operational overhead of such a solution is very high.

I have looked at solutions like this, but have not had any luck with them.

Is there a better approach/solution to this problem?

P.S. My production cluster is extremely large, I cannot afford to stand up a cluster for each namespace.

I have cross posted this question on stackoverflow as well. I am really not sure which forum is a better place this question. Happy to delete it where appropriate.

I have this use case: Setup multiple k8s clusters that can communicate with each other. I also have one network per cluster for intra-cluster communication and another network for inter-cluster communication and for external access in general. Like below:

I am currently leveraging flannel as the network plugin. My understanding is that flannel does not support this use-case (assumption).

Is this something I can achieve with a different CNI plugin? Do I need multiple CNI plugins / plugin instances / a CNI multiplexer?

In case anyone implemented something similar, feedback is highly appreciated.

Thanks!

Update: Multus-cni is not an option as it does not support NodePort on secondary interfaces https://github.com/k8snetworkplumbingwg/multus-cni/issues/727

We are about to start setting up a new kubernetes cluster on bare metal at our own datacenter. The documentation for the k8s moduls and services is great, however I was not able to find any comprehensive top view documentation on the components necessary to meet our requirements:

Pods need to be reachable via IPv4 and IPv6 Pods need to be able to move between hosts and still be reachable on both protocols Pods need to reach outside resources through IPv4 and IPv6 I know that one can use MetalLb for ingress traffic management. But would that also work when a pod tries to reach an external v4/v6 resource?

What would be necessary, overall, to satisfy the requirements?

kube-system   pod/calico-node-9czgm                                  0/1     Running     3          42d
kube-system   pod/calico-node-msfjk                                  0/1     Running     0          5m37s

...is what I get when a let "moon" to join the cluster as outlined below...

NAMESPACE   NAME        STATUS   ROLES                  AGE     VERSION
            node/moon   Ready    <none>                 5m38s   v1.20.5
            node/sky    Ready    control-plane,master   276d    v1.21.2

Whereby pod/calico-node-9czgm is sky's calico node that is running just fine (1/1 Ready) before moon joins and after moon is deleted from the k8s cluster.

❯ kubectl logs pod/calico-node-msfjk -n kube-system -f Error from server: Get "https://192.168.178.23:10250/containerLogs/kube-system/calico-node-msfjk/calico-node?follow=true": dial tcp 192.168.178.23:10250: i/o timeout

pod/calico-node-9czgm tells me:

2021-07-04 20:39:38.999 [INFO][72] felix/ipsets.go 760: Doing full IP set rewrite family="inet" numMembersInPendingReplace=2 setID="all-hosts-net"
bird: Reconfiguration requested by SIGHUP
bird: Reconfiguring
bird: device1: Reconfigured
bird: direct1: Reconfigured
bird: Reconfigured
2021-07-04 20:39:39.023 [INFO][76] confd/resource.go 277: Target config /etc/calico/confd/config/bird.cfg has been updated due to change in key: /calico/bgp/v1/host
2021-07-04 20:39:43.185 [INFO][76] confd/client.go 897: Recompute BGP peerings: HostBGPConfig(node=moon; name=ip_addr_v4) updated; HostBGPConfig(node=moon; name=network_v4) updated
bird: Reconfiguration requested by SIGHUP
bird: Reconfiguring
bird: device1: Reconfigured
bird: direct1: Reconfigured
bird: Adding protocol Mesh_192_168_178_23
bird: Mesh_192_168_178_23: Initializing
bird: Mesh_192_168_178_23: Starting
bird: Mesh_192_168_178_23: State changed to start
bird: Reconfigured
2021-07-04 20:39:43.198 [INFO][76] confd/resource.go 277: Target config /etc/calico/confd/config/bird.cfg has been updated due to change in key: /calico/bgp/v1/host
2021-07-04 20:39:53.134 [INFO][72] felix/summary.go 100: Summarising 13 dataplane reconciliation loops over 1m1.6s: avg=14ms longest=77ms ()
2021-07-04 20:40:11.444 [INFO][74] monitor-addresses/startup.go 774: Using autodetected IPv4 address on interface eth0: 88.99.99.99/32

The IP 192.168.178.23 belongs to my notebook's WiFi interface. So apparently I need to override the local' interface IP with my public IP which I try to do via --node-ip=93.93.17.11 in /etc/systemd/system/kubelet.service.d/kubeadm.conf but everything remains broken and the kubelet additionally reports:

Jul 04 22:58:43 moon kubelet[92613]: E0704 22:58:43.610441 92613 kubelet_node_status.go:586] Failed to set some node status fields: failed to validate nodeIP: node IP: "93.93.17.11" not found in the host's network interfaces

I am using K8s v1.20.5 with cri-o v1.20 on Fedora 34.

i have trouble to add a CNI to a kubernetes master node, the CNI plugin does not have access to certain files or folders. The logs from Calico and Flannel say that certain files or folders are not accessable (In the post I only refer to Calico).

I found the same problem for kubectl, kubeadm and kubelet with version v1.19.4 and v1.19.3. Docker is on version 19.03.13-ce and use overlay2 with an ext4 filesystem and systemd as cgroupdriver. Swap is disabled.

The only thing that goes in that direction what I found on stackoverflow is this: Kubernetes Cluster with Calico - Containers are not coming up & failing with FailedCreatePodSandBox

In the first step I setup the cluster with kubeadm (CIDR for calico):

# kubeadm init --apiserver-advertise-address=192.168.178.33 --pod-network-cidr=192.168.0.0/16

This is workinThis is working correctly, in the kubelet logs is the message that a CNI is required. After this I am applying the CNI calico:

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

After waiting some time the master node will remain in the following state:

❯ kubectl get pods --all-namespaces                 
NAMESPACE     NAME                                       READY   STATUS              RESTARTS   AGE
kube-system   calico-kube-controllers-5c6f6b67db-zdksz   0/1     ContainerCreating   0          7m47s
kube-system   calico-node-sc42z                          0/1     CrashLoopBackOff    5          7m47s
kube-system   coredns-f9fd979d6-4zrcj                    0/1     ContainerCreating   0          8m11s
kube-system   coredns-f9fd979d6-wf9r2                    0/1     ContainerCreating   0          8m11s
kube-system   etcd-hs-0                                  1/1     Running             0          8m20s
kube-system   kube-apiserver-hs-0                        1/1     Running             0          8m20s
kube-system   kube-controller-manager-hs-0               1/1     Running             0          8m20s
kube-system   kube-proxy-t6ngd                           1/1     Running             0          8m11s
kube-system   kube-scheduler-hs-0                        1/1     Running             0          8m20sere

For me, the information I got from the following command:

kubectl describe pods calico-node-sc42z --namespace kube-system

is inconsistent with the next code: The calico-node pod has a mounted volume but also that the pod has no access to it (look ad the volumes and the events).

❯ kubectl describe pods calico-node-sc42z --namespace kube-system
Name:                 calico-node-sc42z
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 hs-0/192.168.178.48
Start Time:           Sat, 14 Nov 2020 00:58:36 +0100
Labels:               controller-revision-hash=5f678767
                      k8s-app=calico-node
                      pod-template-generation=1
Annotations:          <none>
Status:               Running
IP:                   192.168.178.48
IPs:
  IP:           192.168.178.48
Controlled By:  DaemonSet/calico-node
Init Containers:
  upgrade-ipam:
    Container ID:  docker://29c6cf8b73ecb98ee18169db0f6ffe8b141a8a6e10b2c839fc5bf05177f066ac
    Image:         calico/cni:v3.16.5
    Image ID:      docker-pullable://calico/cni@sha256:e05d0ee834c2004e8e7c4ee165a620166cd16e3cb8204a06eb52e5300b46650b
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/calico-ipam
      -upgrade
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 14 Nov 2020 00:58:48 +0100
      Finished:     Sat, 14 Nov 2020 00:58:48 +0100
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      KUBERNETES_NODE_NAME:        (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:  <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
    Mounts:
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/lib/cni/networks from host-local-net-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-tzhr4 (ro)
  install-cni:
    Container ID:  docker://4435863e0d2f3ab4535aa6ca49ff95d889e71614861f3c7c0e4213d8c333f4db
    Image:         calico/cni:v3.16.5
    Image ID:      docker-pullable://calico/cni@sha256:e05d0ee834c2004e8e7c4ee165a620166cd16e3cb8204a06eb52e5300b46650b
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/install
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 14 Nov 2020 00:58:49 +0100
      Finished:     Sat, 14 Nov 2020 00:58:49 +0100
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      CNI_CONF_NAME:         10-calico.conflist
      CNI_NETWORK_CONFIG:    <set to the key 'cni_network_config' of config map 'calico-config'>  Optional: false
      KUBERNETES_NODE_NAME:   (v1:spec.nodeName)
      CNI_MTU:               <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      SLEEP:                 false
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-tzhr4 (ro)
  flexvol-driver:
    Container ID:   docker://ca03f59013c1576a4a605a6d737af78ec3e859376aa11a301e56f0ffdacbc8db
    Image:          calico/pod2daemon-flexvol:v3.16.5
    Image ID:       docker-pullable://calico/pod2daemon-flexvol@sha256:7b20fd9cc36c7196dd24d56cc1e89ac573c634856ee020334b0b30cf5b8a3d3b
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 14 Nov 2020 00:58:56 +0100
      Finished:     Sat, 14 Nov 2020 00:58:56 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /host/driver from flexvol-driver-host (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-tzhr4 (ro)
Containers:
  calico-node:
    Container ID:   docker://96bbc7f4adf1d5cb9a927aedc18e16da7b5ed4b0ff1290179a8dd4a51c115ab8
    Image:          calico/node:v3.16.5
    Image ID:       docker-pullable://calico/node@sha256:43c145b2bd837611d8d41e70631a8f2cc2b97b5ca9d895d66ffddd414dab83c5
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Sat, 14 Nov 2020 01:04:51 +0100
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Sat, 14 Nov 2020 01:03:41 +0100
      Finished:     Sat, 14 Nov 2020 01:04:51 +0100
    Ready:          False
    Restart Count:  5
    Requests:
      cpu:      250m
    Liveness:   exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=1s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      DATASTORE_TYPE:                     kubernetes
      WAIT_FOR_DATASTORE:                 true
      NODENAME:                            (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:          <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
      CLUSTER_TYPE:                       k8s,bgp
      IP:                                 autodetect
      CALICO_IPV4POOL_IPIP:               Always
      CALICO_IPV4POOL_VXLAN:              Never
      FELIX_IPINIPMTU:                    <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_VXLANMTU:                     <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_WIREGUARDMTU:                 <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      CALICO_DISABLE_FILE_LOGGING:        true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      FELIX_IPV6SUPPORT:                  false
      FELIX_LOGSEVERITYSCREEN:            info
      FELIX_HEALTHENABLED:                true
    Mounts:
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /sys/fs/ from sysfs (rw)
      /var/lib/calico from var-lib-calico (rw)
      /var/run/calico from var-run-calico (rw)
      /var/run/nodeagent from policysync (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-tzhr4 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:  
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:  
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/calico
    HostPathType:  
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  sysfs:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/
    HostPathType:  DirectoryOrCreate
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:  
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:  
  host-local-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/cni/networks
    HostPathType:  
  policysync:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/nodeagent
    HostPathType:  DirectoryOrCreate
  flexvol-driver-host:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
    HostPathType:  DirectoryOrCreate
  calico-node-token-tzhr4:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  calico-node-token-tzhr4
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     :NoScheduleop=Exists
                 :NoExecuteop=Exists
                 CriticalAddonsOnly op=Exists
                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists
                 node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                 node.kubernetes.io/unreachable:NoExecute op=Exists
                 node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  6m52s                  default-scheduler  Successfully assigned kube-system/calico-node-sc42z to hs-0
  Normal   Pulling    6m51s                  kubelet            Pulling image "calico/cni:v3.16.5"
  Normal   Pulled     6m40s                  kubelet            Successfully pulled image "calico/cni:v3.16.5" in 10.618669742s
  Normal   Started    6m40s                  kubelet            Started container upgrade-ipam
  Normal   Created    6m40s                  kubelet            Created container upgrade-ipam
  Normal   Created    6m39s                  kubelet            Created container install-cni
  Normal   Pulled     6m39s                  kubelet            Container image "calico/cni:v3.16.5" already present on machine
  Normal   Started    6m39s                  kubelet            Started container install-cni
  Normal   Pulling    6m38s                  kubelet            Pulling image "calico/pod2daemon-flexvol:v3.16.5"
  Normal   Started    6m32s                  kubelet            Started container flexvol-driver
  Normal   Created    6m32s                  kubelet            Created container flexvol-driver
  Normal   Pulled     6m32s                  kubelet            Successfully pulled image "calico/pod2daemon-flexvol:v3.16.5" in 6.076268177s
  Normal   Pulling    6m31s                  kubelet            Pulling image "calico/node:v3.16.5"
  Normal   Pulled     6m19s                  kubelet            Successfully pulled image "calico/node:v3.16.5" in 12.051211859s
  Normal   Created    6m19s                  kubelet            Created container calico-node
  Normal   Started    6m19s                  kubelet            Started container calico-node
  Warning  Unhealthy  5m32s (x5 over 6m12s)  kubelet            Readiness probe failed: calico/node is not ready: BIRD is not ready: Failed to stat() nodename file: stat /var/lib/calico/nodename: no such file or directory
  Warning  Unhealthy  109s (x23 over 6m9s)   kubelet            Liveness probe failed: calico/node is not ready: bird/confd is not live: exit status 1

Further I have the logs of the calico-node, but I do not understand how to benefit from this additional information: Unfortunately I don't know if datastore is referring to the file system, meaning this is the error I already know or if it is something additional.

❯ kubectl logs calico-node-sc42z -n kube-system -f
2020-11-14 01:42:55.536 [INFO][8] startup/startup.go 376: Early log level set to info
2020-11-14 01:42:55.536 [INFO][8] startup/startup.go 392: Using NODENAME environment for node name
2020-11-14 01:42:55.536 [INFO][8] startup/startup.go 404: Determined node name: hs-0
2020-11-14 01:42:55.539 [INFO][8] startup/startup.go 436: Checking datastore connection
2020-11-14 01:43:25.539 [INFO][8] startup/startup.go 451: Hit error connecting to datastore - retry error=Get "https://10.96.0.1:443/api/v1/nodes/foo": dial tcp 10.96.0.1:443: i/o timeout
2020-11-14 01:43:56.540 [INFO][8] startup/startup.go 451: Hit error connecting to datastore - retry error=Get "https://10.96.0.1:443/api/v1/nodes/foo": dial tcp 10.96.0.1:443: i/o timeout

Maybe someone can give me a hint how to solve this problem or where to read about this topic. Greetings, Kokos Bot.