Home / user-844549

Thagor's questions

Martin Hope
Thagor
Asked: 2021-10-14 03:14:37 +0800 CST
  • 0

I have a microk8s cluster on a Centos8 host. The host uses iptables-legacy and firewalld. I have troubles getting the cluster to talk with the internet, and recently found that these two tools are standing in my way. For iptables I can do $ sudo iptables -P FORWARD ACCEPT' on my host, but for the firewalld the only solution right now is to disable it, which is less than ideal. I want to configure the firewall so that it lets traffic to my cluster trough, but I'm not that familiar with firewalld. How to configure it in a way so that it forwards all the traffic that originates from and to my microk8s cluster? (in microk8s, master and node are on the same system)

networking port-forwarding kubernetes firewalld
Martin Hope
Thagor
Asked: 2021-09-23 00:03:40 +0800 CST
  • 1

My microk8s cluster is running on a centos8 vm and I have some issues with DNS resolution of my pods. Nameserver are at x.x.x.101 and x.x.x.100 both are ping able from within the pods and I can also ping 8.8.8.8

nslookup from within a pod looks like this:

root@debug-7857894f66-mnklp:/# nslookup kubernetes.default
Server:         10.152.183.10
Address:        10.152.183.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.152.183.1

coredns config looks like this:

apiVersion: v1
data:
  Corefile: ".:53 {\n    errors\n    health {\n      lameduck 5s\n    }\n    ready\n
    \   log . {\n      class error\n    }\n    kubernetes cluster.local in-addr.arpa
    ip6.arpa {\n      pods insecure\n      fallthrough in-addr.arpa ip6.arpa\n    }\n
    \   prometheus :9153\n    forward .  x.x.x.101 x.x.x.100 \n    cache 30\n
    \   loop\n    reload\n    loadbalance\n}\n"
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"Corefile":".:53 {\n    errors\n    health {\n      lameduck 5s\n    }\n    ready\n    log . {\n      class error\n    }\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods insecure\n      fallthrough in-addr.arpa ip6.arpa\n    }\n    prometheus :9153\n    forward .  x.x.x.101 x.x.x.100 \n    cache 30\n    loop\n    reload\n    loadbalance\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"EnsureExists","k8s-app":"kube-dns"},"name":"coredns","namespace":"kube-system"}}
  creationTimestamp: "2021-08-31T08:57:27Z"
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    k8s-app: kube-dns
  name: coredns
  namespace: kube-system
  resourceVersion: "2420090"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: 471b258a-253d-4b51-aaf7-7e934ab300d1

the /etc/resolv.conf in my pods looks like this:

search default.svc.cluster.local svc.cluster.local cluster.local xxx.xxxxx
nameserver 10.152.183.10
options ndots:5

when I look at the logs of my kube-dns with $ microk8s kubectl logs --namespace=kube-system -l k8s-app=kube-dns I get the following response:

[INFO] 10.1.107.105:47549 - 5288 "AAAA IN www.google.com. udp 36 false 512" NOERROR - 0 0.000256103s
[ERROR] plugin/errors: 2 www.google.com. AAAA: read udp 10.1.107.127:51486->x.x.x.101:53: read: no route to host

DNS service is up:

$ microk8s kubectl get svc --namespace=kube-system
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
[...]
kube-dns                    ClusterIP   10.152.183.10    <none>        53/UDP,53/TCP,9153/TCP   21d

DNS endpoints are exposed:

$ microk8s kubectl get endpoints kube-dns --namespace=kube-system
NAME       ENDPOINTS                                           AGE
kube-dns   10.1.107.127:53,10.1.107.127:53,10.1.107.127:9153   21d

The virtual machine the cluster is running on is using the same nameserver without a problem. Is this something I need to bother our admins about, or am I missing something that is wrongly configured?

EDIT:

Okay I tried out a bit more and there is some weird behavior that I think has to do with me incorrectly creating the coredns config file. I completely reset the cluster and tried again, still same error when I use 8.8.8.8 or 8.8.4.4 as DNS servers. But, when I run the following command: $ microk8s enable dns:x.x.x.101,x.x.x.100 finally it works. Then I tried to configure for this with $ microk8s kubectl -n kube-system edit configmap/coredns and I add both DNS to the config it stops working, and I again get the error: [ERROR] plugin/errors: 2 www.google.com. AAAA: read udp 10.1.107.127:51486->x.x.x.101:53: read: no route to host so what is wrong with my configuration that is automatically correctly set with $ microk8s enable dns:x.x.x.101,x.x.x.100 ?

2nd Edit

I tried to use dig here even when I specify a DNS server it is not working. Is there any reason ping could go through but DNS is blocked by the system? It only happens on the microk8s cluster, host system runs fin docker runs fine ... here are the printouts: this is from within the pod:

root@debug-865cb7fb4-wfhw4:/# dig www.google.com

; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> www.google.com
;; global options: +cmd
;; connection timed out; no servers could be reached
root@debug-865cb7fb4-wfhw4:/# dig @x.x.x.101 www.google.com

; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @x.x.x.101 www.google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
root@debug-865cb7fb4-wfhw4:/# dig @8.8.8.8 www.google.com

; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @8.8.8.8 www.google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
root@debug-865cb7fb4-wfhw4:/# dig @x.x.x.100 www.google.com

; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @x.x.x.100 www.google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

and this is from the host system:

$ dig www.google.com

; <<>> DiG 9.11.20-RedHat-9.11.20-5.el8_3.1 <<>> www.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25735
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;www.google.com.                        IN      A

;; ANSWER SECTION:
www.google.com.         113     IN      A       142.250.185.228

;; Query time: 0 msec
;; SERVER: x.x.x.101#53(x.x.x.101)
;; WHEN: Fri Oct 08 15:10:21 CEST 2021
;; MSG SIZE  rcvd: 59



$ dig @8.8.8.8 www.google.com

; <<>> DiG 9.11.20-RedHat-9.11.20-5.el8_3.1 <<>> @8.8.8.8 www.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3924
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;www.google.com.                        IN      A

;; ANSWER SECTION:
www.google.com.         300     IN      A       142.250.185.228

;; Query time: 34 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Oct 08 15:10:49 CEST 2021
;; MSG SIZE  rcvd: 59

$ dig @x.x.x.101 www.google.com

; <<>> DiG 9.11.20-RedHat-9.11.20-5.el8_3.1 <<>> @x.x.x.101 www.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60305
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;www.google.com.                        IN      A

;; ANSWER SECTION:
www.google.com.         70      IN      A       142.250.185.228

;; Query time: 0 msec
;; SERVER: x.x.x.101#53(x.x.x.101)
;; WHEN: Fri Oct 08 15:11:04 CEST 2021
;; MSG SIZE  rcvd: 59

I have no idea what seems to be going on ....

domain-name-system kubernetes microk8s