Home / server / Questions / 1076702
Accepted
Dean Schulze
Asked: 2021-09-06 11:32:45 +0800 CST

Only have connectivity to nginx pod from the node its running on

  • 772

I've installed kubernetes master and one node v 1.20. I deployed nginx with

kubectl run nginxpod --image=nginx


$ kubectl get pods -o wide
NAME       READY   STATUS    RESTARTS   AGE   IP              NODE         NOMINATED NODE   READINESS GATES
nginxpod   1/1     Running   0          19s   192.168.2.195   xps15-9560   <none>           <none>

On master when I curl on master it times out:

$ curl 192.168.2.195
curl: (7) Failed to connect to 192.168.2.195 port 80: Connection timed out

On the node it works. I've tried from other hosts on my network and they timeout too. Why can I only connect from the node the pod is actually running on?

----Edit----

The calico-nodes are running but they are not ready. I don't know what this means:

$ kubectl get pods -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
default       nginxpod                                   1/1     Running   0          64m
kube-system   calico-kube-controllers-5f6cfd688c-wk5jp   1/1     Running   0          69m
kube-system   calico-node-t47kf                          0/1     Running   0          45m
kube-system   calico-node-vqj6m                          0/1     Running   0          68m
kube-system   calico-node-wzwzb                          0/1     Running   0          69m
kube-system   coredns-74ff55c5b-mb2vj                    1/1     Running   0          69m
kube-system   coredns-74ff55c5b-pvsgz                    1/1     Running   0          69m
kube-system   etcd-ubuntu-18-extssd                      1/1     Running   0          69m
kube-system   kube-apiserver-ubuntu-18-extssd            1/1     Running   0          69m
kube-system   kube-controller-manager-ubuntu-18-extssd   1/1     Running   0          69m
kube-system   kube-proxy-5fq9b                           1/1     Running   0          68m
kube-system   kube-proxy-bxhfm                           1/1     Running   0          69m
kube-system   kube-proxy-pp9sb                           1/1     Running   0          45m
kube-system   kube-scheduler-ubuntu-18-extssd            1/1     Running   0          69m

--------Edit 2------------------

Calico nodes NotFound:

$ kubectl describe pod calico-node-t47kf
Error from server (NotFound): pods "calico-node-t47kf" not found
$ kubectl describe pod calico-node-vqj6m
Error from server (NotFound): pods "calico-node-vqj6m" not found
$ kubectl describe pod calico-node-*****
Error from server (NotFound): pods "calico-node-*****" not found

----Edit 3---- Output for describe pod calico-node-t47kf:

$ kubectl -n kube-system describe pod calico-node-t47kf
Name:                 calico-node-t47kf
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 xps15-9560/192.168.0.71
Start Time:           Sun, 05 Sep 2021 16:51:37 -0600
Labels:               controller-revision-hash=b8998dcb
                      k8s-app=calico-node
                      pod-template-generation=1
Annotations:          <none>
Status:               Running
IP:                   192.168.0.71
IPs:
  IP:           192.168.0.71
Controlled By:  DaemonSet/calico-node
Init Containers:
  upgrade-ipam:
    Container ID:  docker://3d393316548badf75bb2c2ad881ffd7a4d2c37a1762d84ec973715c8a398072e
    Image:         docker.io/calico/cni:v3.20.0
    Image ID:      docker-pullable://calico/cni@sha256:9906e2cca8006e1fe9fc3f358a3a06da6253afdd6fad05d594e884e8298ffe1d
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/calico-ipam
      -upgrade
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 05 Sep 2021 16:51:38 -0600
      Finished:     Sun, 05 Sep 2021 16:51:38 -0600
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      KUBERNETES_NODE_NAME:        (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:  <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
    Mounts:
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/lib/cni/networks from host-local-net-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-qfrcw (ro)
  install-cni:
    Container ID:  docker://9e3b23f12657fe343117ec0cf54e104a8eb69c6133fe2dd10c1aabbc9260189f
    Image:         docker.io/calico/cni:v3.20.0
    Image ID:      docker-pullable://calico/cni@sha256:9906e2cca8006e1fe9fc3f358a3a06da6253afdd6fad05d594e884e8298ffe1d
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/install
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 05 Sep 2021 16:51:39 -0600
      Finished:     Sun, 05 Sep 2021 16:51:40 -0600
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      CNI_CONF_NAME:         10-calico.conflist
      CNI_NETWORK_CONFIG:    <set to the key 'cni_network_config' of config map 'calico-config'>  Optional: false
      KUBERNETES_NODE_NAME:   (v1:spec.nodeName)
      CNI_MTU:               <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      SLEEP:                 false
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-qfrcw (ro)
  flexvol-driver:
    Container ID:   docker://090008276cf33c2fd64aa141405e90feff00fc70f55769372655e45bf2e2dc92
    Image:          docker.io/calico/pod2daemon-flexvol:v3.20.0
    Image ID:       docker-pullable://calico/pod2daemon-flexvol@sha256:c17e3e9871682bed00bfd33f8d6f00db1d1a126034a25bf5380355978e0c548d
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 05 Sep 2021 16:51:40 -0600
      Finished:     Sun, 05 Sep 2021 16:51:40 -0600
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /host/driver from flexvol-driver-host (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-qfrcw (ro)
Containers:
  calico-node:
    Container ID:   docker://2c1456bb2f346564f88b3e8deba3baef1669f0aa61aa0e654d52c915eaf462cf
    Image:          docker.io/calico/node:v3.20.0
    Image ID:       docker-pullable://calico/node@sha256:7f9aa7e31fbcea7be64b153f8bcfd494de023679ec10d851a05667f0adb42650
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Sun, 05 Sep 2021 16:51:41 -0600
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      250m
    Liveness:   exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=10s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=10s period=10s #success=1 #failure=3
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      DATASTORE_TYPE:                     kubernetes
      WAIT_FOR_DATASTORE:                 true
      NODENAME:                            (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:          <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
      CLUSTER_TYPE:                       k8s,bgp
      IP:                                 autodetect
      CALICO_IPV4POOL_IPIP:               Always
      CALICO_IPV4POOL_VXLAN:              Never
      FELIX_IPINIPMTU:                    <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_VXLANMTU:                     <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_WIREGUARDMTU:                 <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      CALICO_DISABLE_FILE_LOGGING:        true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      FELIX_IPV6SUPPORT:                  false
      FELIX_HEALTHENABLED:                true
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /sys/fs/ from sysfs (rw)
      /var/lib/calico from var-lib-calico (rw)
      /var/log/calico/cni from cni-log-dir (ro)
      /var/run/calico from var-run-calico (rw)
      /var/run/nodeagent from policysync (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-qfrcw (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:  
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:  
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/calico
    HostPathType:  
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  sysfs:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/
    HostPathType:  DirectoryOrCreate
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:  
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:  
  cni-log-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/calico/cni
    HostPathType:  
  host-local-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/cni/networks
    HostPathType:  
  policysync:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/nodeagent
    HostPathType:  DirectoryOrCreate
  flexvol-driver-host:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
    HostPathType:  DirectoryOrCreate
  calico-node-token-qfrcw:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  calico-node-token-qfrcw
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     :NoSchedule op=Exists
                 :NoExecute op=Exists
                 CriticalAddonsOnly op=Exists
                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists
                 node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                 node.kubernetes.io/unreachable:NoExecute op=Exists
                 node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                      From     Message
  ----     ------     ----                     ----     -------
  Warning  Unhealthy  93s (x41181 over 4d18h)  kubelet  (combined from similar events): Readiness probe failed: 2021-09-10 17:16:42.497 [INFO][1620263] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 192.168.55.1,192.168.0.12

------Edit 4-------

Tried deleting and reinstalling calico and that made things worse.

nginx kubernetes calico

1 Answers

  1. Best Answer

    Apparently there is something wrong with the calico.yaml. Here is the solution that got my calico nodes up:

    Run:
kubectl set env daemonset/calico-node -n kube-system IP_AUTODETECTION_METHOD=can-reach=www.google.com

Then the pods became ready within a few seconds.

Reference:
https://docs.projectcalico.org/networking/ip-autodetection#change-the-autodetection-method
    • 1