Questions[aks](server)

I've created a private cluster on AKS and deployed some workloads to it, but I'm not sure how to connect to the services. They are all NodePort services, both TCP & UDP.

Initially, I thought that the endpoint AKS provides could be used to interact with the cluster as a whole, but this isn't the case -- this only exposes the Kubernetes API (probably why they call it the API endpoint)

I also tried using the VMSS public IP but that didn't work either. Using the instance IP works, but the IPs would change over time, right?

Ultimately I decided to use an ingress controller and proxy the traffic, but because I'm using mixed protocols I would need two. Not to mention this seems too complex to be the reasonable solution to this problem.

How to expose services on a private AKS cluster? This should be a pretty common problem, I think, but I haven't found a clear solution

I have two subscriptions under my account. When I check today, the balance is low in my subscription where I deployed the cluster.

So, I click on change subscription option and selected the second subscription in my account.

But, getting below error.

{"code":"ResourceMoveValidationFailed","message":"The resource batch move request has '1' validation errors. Diagnostic information: timestamp '20210708T074027Z', tracking Id 'bae55f37-137e-4388-ba57-a46456b5efb9', request correlation Id '2ffbd2bc-1198-42ec-b8c5-60cbaed7a160'.","details":[{"code":"ResourceMoveNotSupported","target":"/subscriptions/f733ab10-e6a2-406d-9c23-cafe4ec0e71e/resourceGroups/testmart/providers/Microsoft.ContainerService/managedClusters/testmart","message":"Resource move is not supported for resource types 'Microsoft.ContainerService/managedClusters'."}]}

How to move the cluster then?

The pods in my application scale with 1 pod per user (each user gets their own pod). I have the limits for the application container set up like so:

  resources:
    limits:
      cpu: 250m
      memory: 768Mi
    requests:
      cpu: 100m
      memory: 512Mi

The nodes in my nodepool have 8GB of memory each. I started up a bunch of user instances to begin testing, and watched my resource metrics go up as I started each one:

CPU:

Memory:

At 15:40, I saw the event logs show this error (note: the first node is excluded using a taint):

0/2 nodes are available: 1 Insufficient memory, 1 node(s) didn't match node selector.

Why did this happen when the memory/cpu requests were still well below the total capacity (~50% for cpu, ~60% mem)?

Here is some relevant info from kubectl describe node:

Non-terminated Pods:          (12 in total)
  Namespace                   Name                                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                   ----                                                               ------------  ----------  ---------------  -------------  ---
  ide                         theia-deployment--ac031811--football-6b6d54ddbb-txsd4              110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    13m
  ide                         theia-deployment--ac031811--footballteam-6fb7b68794-cv4c9          110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    12m
  ide                         theia-deployment--ac031811--how-to-play-football-669ddf7c8cjrzl    110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    14m
  ide                         theia-deployment--ac031811--packkide-7bff98d8b6-5twkf              110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    9m54s
  ide                         theia-deployment--ac032611--static-website-8569dd795d-ljsdr        110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    16m
  ide                         theia-deployment--aj090111--spiderboy-6867b46c7d-ntnsb             110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    2m36s
  ide                         theia-deployment--ar041311--tower-defenders-cf8c5dd58-tl4j9        110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    14m
  ide                         theia-deployment--np091707--my-friends-suck-at-coding-fd48ljs7z    110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    4m14s
  ide                         theia-deployment--np091707--topgaming-76b98dbd94-fgdz6             110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    5m17s
  kube-system                 csi-azurefile-node-nhbpg                                           30m (1%)      400m (21%)  60Mi (1%)        400Mi (7%)     12d
  kube-system                 kube-proxy-knq65                                                   100m (5%)     0 (0%)      0 (0%)           0 (0%)         12d
  lens-metrics                node-exporter-57zp4                                                10m (0%)      200m (10%)  24Mi (0%)        100Mi (1%)     6d20h

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                       Requests      Limits
  --------                       --------      ------
  cpu                            1130m (59%)   3750m (197%)
  memory                         4836Mi (90%)  7988Mi (148%)
  ephemeral-storage              0 (0%)        0 (0%)
  hugepages-1Gi                  0 (0%)        0 (0%)
  hugepages-2Mi                  0 (0%)        0 (0%)
  attachable-volumes-azure-disk  0             0

I have a managed kubernetes instance on azure. I am very sure that the core dns is working and the dns pods are healthy.

I have a couple of services

1. frontend-service with one pod - Image [nginx-alpine] which has the static frontend files.

2. backend-service , with one pod - Image [ubuntu:20.04] which has the nodejs code.

I am unable to resolve the internal dns service names like frontend-service OR frontend-service.default.svc.cluster.local from the pods of the backend but nslookup , host , dig of the internal dns names resolve to the correct address. The backend pods are also able to resolve the external dns names like google.com.

curl http://frontend-service
curl: (6) Could not resolve host: frontend-service

curl http://frontend-service.default.svc.cluster.local
curl: (6) Could not resolve host: frontend-service.default.svc.cluster.local

wget frontend-service
--2020-08-31 23:36:43--  http://frontend-service
Resolving frontend-service (frontend-service)... failed: Name or service not known.
wget: unable to resolve host address 'frontend-service'

/etc/nsswitch.conf shows the below :

passwd:         files
group:          files
shadow:         files
gshadow:        files

hosts:          files dns
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

Everything works fine while trying to resolve the backend-service internal dns name from the pods of frontend service.

After some debugging and looking at the logs of coredns and the strace , I see that no call is happening to the coredns pods while doing a curl , but I can see the entry while doing an nslook up.

I also. verified that the /etc/resolv.conf has the correct configuration.

nameserver 10.3.0.10
search default.svc.cluster.local svc.cluster.local cluster.local tdghymxumodutbxfnz5m2elcog.bx.internal.cloudapp.net
options ndots:5

strace does not show any entry to search for /etc/resolv.conf , so curl is not checking for /etc/resolv.conf.

Edit 1

From the backend service pod :
dig frontend-service [It is able to resolve to the correct name server.]


; <<>> DiG 9.16.1-Ubuntu <<>> frontend-service
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 13441
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; OPT=65436: 87 a1 ee 81 04 d8 5a 49 be 0e c4 ed 1d d8 27 41 ("......ZI......'A")
;; QUESTION SECTION:
;frontend-service.            IN      A

;; AUTHORITY SECTION:
.                       30      IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2020083101 1800 900 604800 86400

;; Query time: 20 msec
;; SERVER: 10.3.0.10#53(10.3.0.10)
;; WHEN: Tue Sep 01 10:48:00 IST 2020
;; MSG SIZE  rcvd: 142

nslookup frontend-service

Server:         10.3.0.10
Address:        10.3.0.10#53

Name:   frontend-service.default.svc.cluster.local
Address: 10.3.0.30

host frontend-service     
frontend-service.default.svc.cluster.local has address 10.3.0.30

Edit 2

I wanted to test the deployment step by step with the same ubuntu:20.04 image , so I did the following.

Approach 1

I created an ephemeral pod in the cluster as below.

kubectl run -it --rm test-ubuntu --image=ubuntu:20.04 --restart=Never

Installed curl (7.68) and ran the curl http://frontend-service – This is successful.

This puzzled me , so I have removed all my build steps from Dockerfile and used only the below commands.

Approach 2

Dockerfile

FROM ubuntu:20.04
 
EXPOSE 3688
CMD [ "sleep", "infinity" ]

Pushed the image to acr and deployed the backend pods again.

kubectl exec -it <pod-name> /bin/bash

I installed curl (7.68) and ran the curl http://frontend-service – Same error – unable to resolve host.

This is surprising , same image with same content – running through kubectl run and deploying through Dockerfile , has different behaviour while running curl of same version (7.68).

I wanted to see the flow in strace in both the appraches. Please find the strace links from RUN and EXEC

strace from running curl from the ephemeral pod. https://pastebin.com/NthHQacW

strace from running curl from the pod deployed through Dockerfile https://pastebin.com/6LCE5NXu

After analysing the probing paths by running

cat strace-log | grep open

I found that the strace log from the approach 2 is missing the below lines.

2844  openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 7
2844  openat(AT_FDCWD, "/etc/host.conf", O_RDONLY|O_CLOEXEC <unfinished...>
2844  <... openat resumed>)             = 7
2844  openat(AT_FDCWD, "/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 7
2844  openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 7
2844  openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 7
2844  openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 7
2844  openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC <unfinished ...>
2844  <... openat resumed>)             = 7
2844  openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = 7

So the curl command within the pod is not looking at either /etc/resolv.conf OR /etc/nsswitch.conf.

I am puzzled why the curl's behaviour within two pods with same image and same curl version in the same cluster is different.

I've set up an Azure Application Gateway with Azure Kubernetes Service using the Azure Application Gateway Ingress Controller (AGIC) and confirmed that it's working correctly using the sample guestbook app.

I then used almost the exact configuration to deploy a Golang app that uses the gRPC-gateway to the same AKS cluster.

The default liveness and readiness health checks fail so I've configured custom exec based (not fit for prod) so that they pass the checks and are added to the backend pool.

The gateway (REST) pod is healthy however the pod that uses gRPC is still unhealthy and has the following error message.

Cannot connect to server. Check whether any NSG/UDR/Firewall is blocking 
access to server. Check if application is running on correct port.

I'm guessing that the gateway doesn't support gRPC/binary communication but I'm not positive and cannot find anything in the documentation.

Setup Details: - Greenfield - Application Gateway Ingress Controller tutorial to build the AKS cluster

I've submitted a help ticket to Microsoft however I'm still waiting for a reply. Has anyone else tried to host gRPC apps using the Application Gateway?