curl & wget cannot resolve internal dns names within the aks cluster but nslookup , host , dig work fine

I have a managed kubernetes instance on azure. I am very sure that the core dns is working and the dns pods are healthy.

I have a couple of services

1. frontend-service with one pod - Image [nginx-alpine] which has the static frontend files.

2. backend-service , with one pod - Image [ubuntu:20.04] which has the nodejs code.

I am unable to resolve the internal dns service names like frontend-service OR frontend-service.default.svc.cluster.local from the pods of the backend but nslookup , host , dig of the internal dns names resolve to the correct address. The backend pods are also able to resolve the external dns names like google.com.

curl http://frontend-service
curl: (6) Could not resolve host: frontend-service

curl http://frontend-service.default.svc.cluster.local
curl: (6) Could not resolve host: frontend-service.default.svc.cluster.local

wget frontend-service
--2020-08-31 23:36:43--  http://frontend-service
Resolving frontend-service (frontend-service)... failed: Name or service not known.
wget: unable to resolve host address 'frontend-service'

/etc/nsswitch.conf shows the below :

passwd:         files
group:          files
shadow:         files
gshadow:        files

hosts:          files dns
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

Everything works fine while trying to resolve the backend-service internal dns name from the pods of frontend service.

After some debugging and looking at the logs of coredns and the strace , I see that no call is happening to the coredns pods while doing a curl , but I can see the entry while doing an nslook up.

I also. verified that the /etc/resolv.conf has the correct configuration.

nameserver 10.3.0.10
search default.svc.cluster.local svc.cluster.local cluster.local tdghymxumodutbxfnz5m2elcog.bx.internal.cloudapp.net
options ndots:5

strace does not show any entry to search for /etc/resolv.conf , so curl is not checking for /etc/resolv.conf.

Edit 1

From the backend service pod :
dig frontend-service [It is able to resolve to the correct name server.]


; <<>> DiG 9.16.1-Ubuntu <<>> frontend-service
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 13441
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; OPT=65436: 87 a1 ee 81 04 d8 5a 49 be 0e c4 ed 1d d8 27 41 ("......ZI......'A")
;; QUESTION SECTION:
;frontend-service.            IN      A

;; AUTHORITY SECTION:
.                       30      IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2020083101 1800 900 604800 86400

;; Query time: 20 msec
;; SERVER: 10.3.0.10#53(10.3.0.10)
;; WHEN: Tue Sep 01 10:48:00 IST 2020
;; MSG SIZE  rcvd: 142

nslookup frontend-service

Server:         10.3.0.10
Address:        10.3.0.10#53

Name:   frontend-service.default.svc.cluster.local
Address: 10.3.0.30

host frontend-service     
frontend-service.default.svc.cluster.local has address 10.3.0.30

Edit 2

I wanted to test the deployment step by step with the same ubuntu:20.04 image , so I did the following.

Approach 1

I created an ephemeral pod in the cluster as below.

kubectl run -it --rm test-ubuntu --image=ubuntu:20.04 --restart=Never

Installed curl (7.68) and ran the curl http://frontend-service – This is successful.

This puzzled me , so I have removed all my build steps from Dockerfile and used only the below commands.

Approach 2

Dockerfile

FROM ubuntu:20.04
 
EXPOSE 3688
CMD [ "sleep", "infinity" ]

Pushed the image to acr and deployed the backend pods again.

kubectl exec -it <pod-name> /bin/bash

I installed curl (7.68) and ran the curl http://frontend-service – Same error – unable to resolve host.

This is surprising , same image with same content – running through kubectl run and deploying through Dockerfile , has different behaviour while running curl of same version (7.68).

I wanted to see the flow in strace in both the appraches. Please find the strace links from RUN and EXEC

strace from running curl from the ephemeral pod. https://pastebin.com/NthHQacW

strace from running curl from the pod deployed through Dockerfile https://pastebin.com/6LCE5NXu

After analysing the probing paths by running

cat strace-log | grep open

I found that the strace log from the approach 2 is missing the below lines.

2844  openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 7
2844  openat(AT_FDCWD, "/etc/host.conf", O_RDONLY|O_CLOEXEC <unfinished...>
2844  <... openat resumed>)             = 7
2844  openat(AT_FDCWD, "/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 7
2844  openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 7
2844  openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 7
2844  openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 7
2844  openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC <unfinished ...>
2844  <... openat resumed>)             = 7
2844  openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = 7

So the curl command within the pod is not looking at either /etc/resolv.conf OR /etc/nsswitch.conf.

I am puzzled why the curl's behaviour within two pods with same image and same curl version in the same cluster is different.