Home / server / Questions / 1108493
In Process
Chris G.
Asked: 2022-08-19 04:40:18 +0800 CST

Calico HA cluster - not-ready:NoSchedule

  • 772

Installed Calico on a K8s HA cluster

Note: same installation worked on single node installation, removing taint. But in a HA Cluster it does only have taint: node.kubernetes.io/not-ready:NoSchedule

  kubectl create -f https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml
  curl https://projectcalico.docs.tigera.io/manifests/custom-resources.yaml -O
  kubectl create -f custom-resources.yaml

On worker node I have:

kubectl describe nodes <worker node> | grep -i taint
Taints:             node.kubernetes.io/not-ready:NoSchedule

I then did:

kubectl taint nodes <worker node> node.kubernetes.io/not-ready:NoSchedule-

This gets me from pending to ContainerCreating - problem is that it never finishes creating?

kube-system       coredns-6d4b75cb6d-mlshg           0/1     ContainerCreating   0          35m

On worker node - describe pod:

kubectl describe pod coredns-6d4b75cb6d-89ggn -n kube-system
Name:                 coredns-6d4b75cb6d-89ggn
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 master01worker01/192.168.64.7
Start Time:           Thu, 18 Aug 2022 14:47:22 +0200
Labels:               k8s-app=kube-dns
                      pod-template-hash=6d4b75cb6d
Annotations:          <none>
Status:               Pending
IP:                   
IPs:                  <none>
Controlled By:        ReplicaSet/coredns-6d4b75cb6d
Containers:
  coredns:
    Container ID:  
    Image:         k8s.gcr.io/coredns/coredns:v1.8.6
    Image ID:      
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tbb27 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  kube-api-access-tbb27:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule
                             node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  2m57s                 default-scheduler  0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Normal   Scheduled         2m15s                 default-scheduler  Successfully assigned kube-system/coredns-6d4b75cb6d-89ggn to master01worker01
  Warning  NetworkNotReady   98s (x18 over 2m10s)  kubelet            network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
  Warning  FailedMount       98s (x7 over 2m10s)   kubelet            MountVolume.SetUp failed for volume "config-volume" : object "kube-system"/"coredns" not registered

if I delete the pods, I get:

Warning FailedScheduling 76s default-scheduler 0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling

And then again if I try to un-taint using:

kubectl taint nodes <conrtrol plane> node.kubernetes.io/not-ready:NoSchedule-

Then I am back to Events:

  Type     Reason            Age              From               Message
  ----     ------            ----             ----               -------
  Warning  FailedScheduling  4m14s            default-scheduler  0/2 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
  Normal   Scheduled         3s               default-scheduler  Successfully assigned kube-system/coredns-6d4b75cb6d-gglk2 to master01worker01
  Warning  NetworkNotReady   1s (x2 over 3s)  kubelet            network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
  Warning  FailedMount       1s (x3 over 2s)  kubelet            MountVolume.SetUp failed for volume "config-volume" : object "kube-system"/"coredns" not registered
kubernetes

0 Answers