uday's questions

I want to run amazon linux commands as part of gitlab pipeline.

So, trying to use docker image as runner, amazonlinux:latest

So, connected to docker container and ran below command.

yum -yq install aws-cli

It installed aws-cli

Then, configured aws cli.

aws configure set region $AWS_REGION
aws configure set aws_access_key_id $AWS_ACCESS_KEY
aws configure set aws_secret_access_key $AWS_SECRET_KEY
aws configure set plugins.bolt awscli-plugin-bolt

Then ran below command to check identity, but got error as aws-cli-plugin-bolt not there.

aws sts get-caller-identity

Then ran python pip install for the same and got python 2.7 deprecation error and module not found.

pip install awscli-plugin-bolt
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip install --user` instead.
ERROR: Could not find a version that satisfies the requirement awscli-plugin-bolt (from versions: none)
ERROR: No matching distribution found for awscli-plugin-bolt

So changed the python to python 3 as default version using below commands and the reference page.

amazon-linux-extras enable python3.8
yum install python3.8
update-alternatives --install /usr/bin/python python /usr/bin/python3.8 1

And then installed the bolt pluging using pip3.

yum install python3-pip
pip3 install awscli-plugin-bolt

But still, below command still looks for python 2 and failing.

aws sts get-caller-identity

So, deleted that container and created a new one and in that as first step, changed the python version 3 as default one.

But this time, yum installation of aws-cli itself failing.

 bash-4.2# yum -yq install aws-cli
  File "/usr/bin/yum", line 30
    except KeyboardInterrupt, e:
                            ^
SyntaxError: invalid syntax

Please suggest how to access aws cli commands from the amazonlinux docker image.

It seems it is working if I change the image to

  image: 
    name: amazon/aws-cli
    entrypoint: [""]

So that no need to install aws cli on my own. But it is coming with 3.7 by default, how to change that as part of image. And major question, in gitlab above image is working, with docker run I wasn't able to use the image directly. Only individual aws commands it is accepting and closing the container immediately

With the azure policy addon enabled(as per organization policy), we can’t create privileged containers on the aks, azure kubernetes.

Our application is set to security context as below.

    securityContext:
        allowPrivilegeEscalation: false
        runAsNonRoot: true
        runAsUser: 999

So our app can create without privileged access. But, when linked with consul(through annotations), the consul init containers are failing to create.

 Warning  FailedCreate  6s (x15 over 90s)  replicaset-controller  Error creating: admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-psp-container-no-privilege-esc-30132221bc21e5b724da] Privilege escalation container is not allowed: envoy-sidecar
[azurepolicy-psp-container-no-privilege-esc-30132221bc21e5b724da] Privilege escalation container is not allowed: consul-sidecar
[azurepolicy-psp-container-no-privilege-esc-30132221bc21e5b724da] Privilege escalation container is not allowed: consul-connect-inject-init

I have deleted the old clusters by deleting the resource group completely. But when I try to create a new cluster, I am getting error as limit exceeded.

As you can see in the below error.

{"code":"InvalidTemplateDeployment","message":"The template deployment 'microsoft.aks-20210803135741' is not valid according to the validation procedure. The tracking id is '979df053-50fa-4201-9643-5f3029e0b43d'. See inner errors for details.","details":[{"code":"QuotaExceeded","message":"Provisioning of resource(s) for container service testingdelete in resource group testingdelete failed. Message: {\n "code": "QuotaExceeded",\n "message": "Provisioning of resource(s) for container service testingdelete in resource group testingdelete failed. Message: Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus2, Current Limit: 10, Current Usage: 10, Additional Required: 6, (Minimum) New Limit Required: 16. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/?#create/Microsoft.Support/Parameters/%7B%22subId%22:******************************************************************************************************************************** by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/regional-quota-requests.. Details: "\n }. Details: "}]}

How to cleanup the old details which are left?

I have a configmap like below.Which I will link to a config file in our application.

apiVersion: v1
kind: ConfigMap
metadata:
  name: database-configmap
data:
  config: |
   dbport=5432
   dcname=
   dbssl=false
   locktime=300
   domainuser=
   dattserverhost=localhost
   conntype=ON-PREM
   dbinstance=
   dattusessl=false
   dbpwd=VrjvBguQ
   iisport=80
   docountupgrade=false
   doreportupgrade=false
   dbname=dattdatabase
   dattuseiis=false
   dbtype=POSTGRESQL
   dbusername=postg
   dbserver=tgres.database.azure.com
   domainpwd=

How to save this in azurekeyvault and get from there during deployment? Individual key/value pairs are stored there in keyvault. But how to achieve this?

I have tried the option provided in the link but it is not showing as normal kubernetes secret. Please see the output below:

kubectl get azurekeyvaultsecret.spv.no/secret-sync -n akv-test
NAME          VAULT           VAULT OBJECT   SECRET NAME   SYNCHED
secret-sync   testingvaultd   sample1
kubectl get secrets -n akv-test
NAME                  TYPE                                  DATA   AGE
default-token-9bsk8   kubernetes.io/service-account-token   3      5m31s

I have followed the steps mentioned in the link

Came till the deployment part, last before step.

After deployment, the pods status is stuck at creation.

kubectl get pods --watch
NAME                                READY   STATUS     RESTARTS   AGE
devwebapp                           0/2     Init:0/1   0          2m11s
nginx-6799fc88d8-9xnqv              1/1     Running    1          98m
vault-0                             1/1     Running    0          25m
vault-agent-injector-c5f9f8-zcv6q   1/1     Running    0          25m

So I ran the describe command and found nothing.

osboxes@osboxes:~$ kubectl describe pod devwebapp
Name:         devwebapp
Namespace:    default
Priority:     0
Node:         uday1-control-plane/172.19.0.2
Start Time:   Tue, 27 Jul 2021 15:50:56 -0400
Labels:       app=devwebapp
Annotations:  vault.hashicorp.com/agent-inject: true
              vault.hashicorp.com/agent-inject-secret-credentials.txt: secret/data/martwebapp/config
              vault.hashicorp.com/agent-inject-status: injected
              vault.hashicorp.com/role: martweb-app
Status:       Pending
IP:           10.244.0.10
IPs:
  IP:  10.244.0.10
Init Containers:
  vault-agent-init:
    Container ID:  containerd://a125495c63dc63e605146b9dd67d1e0e731c43c28e4130156d261efca2aaf54c
    Image:         vault:1.7.3
    Image ID:      docker.io/library/vault@sha256:6085e96fa42c2524eef7bf9af0cf5199da0b16964003b3f88e2b8195b6acb52b
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -ec
    Args:
      echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
    State:          Running
      Started:      Tue, 27 Jul 2021 15:50:57 -0400
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:     250m
      memory:  64Mi
    Environment:
      VAULT_LOG_LEVEL:   info
      VAULT_LOG_FORMAT:  standard
      VAULT_CONFIG:      eyJhdXRvX2F1dGgiOnsibWV0aG9kIjp7InR5cGUiOiJrdWJlcm5ldGVzIiwibW91bnRfcGF0aCI6ImF1dGgva3ViZXJuZXRlcyIsImNvbmZpZyI6eyJyb2xlIjoibWFydHdlYi1hcHAifX0sInNpbmsiOlt7InR5cGUiOiJmaWxlIiwiY29uZmlnIjp7InBhdGgiOiIvaG9tZS92YXVsdC8udmF1bHQtdG9rZW4ifX1dfSwiZXhpdF9hZnRlcl9hdXRoIjp0cnVlLCJwaWRfZmlsZSI6Ii9ob21lL3ZhdWx0Ly5waWQiLCJ2YXVsdCI6eyJhZGRyZXNzIjoiaHR0cDovL3ZhdWx0LmRlZmF1bHQuc3ZjOjgyMDAifSwidGVtcGxhdGUiOlt7ImRlc3RpbmF0aW9uIjoiL3ZhdWx0L3NlY3JldHMvY3JlZGVudGlhbHMudHh0IiwiY29udGVudHMiOiJ7eyB3aXRoIHNlY3JldCBcInNlY3JldC9kYXRhL21hcnR3ZWJhcHAvY29uZmlnXCIgfX17eyByYW5nZSAkaywgJHYgOj0gLkRhdGEgfX17eyAkayB9fToge3sgJHYgfX1cbnt7IGVuZCB9fXt7IGVuZCB9fSIsImxlZnRfZGVsaW1pdGVyIjoie3siLCJyaWdodF9kZWxpbWl0ZXIiOiJ9fSJ9XX0=
    Mounts:
      /home/vault from home-init (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9k5qp (ro)
      /vault/secrets from vault-secrets (rw)
Containers:
  devwebapp:
    Container ID:
    Image:          jweissig/app:0.0.1
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9k5qp (ro)
      /vault/secrets from vault-secrets (rw)
  vault-agent:
    Container ID:
    Image:         vault:1.7.3
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -ec
    Args:
      echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:     250m
      memory:  64Mi
    Environment:
      VAULT_LOG_LEVEL:   info
      VAULT_LOG_FORMAT:  standard
      VAULT_CONFIG:      eyJhdXRvX2F1dGgiOnsibWV0aG9kIjp7InR5cGUiOiJrdWJlcm5ldGVzIiwibW91bnRfcGF0aCI6ImF1dGgva3ViZXJuZXRlcyIsImNvbmZpZyI6eyJyb2xlIjoibWFydHdlYi1hcHAifX0sInNpbmsiOlt7InR5cGUiOiJmaWxlIiwiY29uZmlnIjp7InBhdGgiOiIvaG9tZS92YXVsdC8udmF1bHQtdG9rZW4ifX1dfSwiZXhpdF9hZnRlcl9hdXRoIjpmYWxzZSwicGlkX2ZpbGUiOiIvaG9tZS92YXVsdC8ucGlkIiwidmF1bHQiOnsiYWRkcmVzcyI6Imh0dHA6Ly92YXVsdC5kZWZhdWx0LnN2Yzo4MjAwIn0sInRlbXBsYXRlIjpbeyJkZXN0aW5hdGlvbiI6Ii92YXVsdC9zZWNyZXRzL2NyZWRlbnRpYWxzLnR4dCIsImNvbnRlbnRzIjoie3sgd2l0aCBzZWNyZXQgXCJzZWNyZXQvZGF0YS9tYXJ0d2ViYXBwL2NvbmZpZ1wiIH19e3sgcmFuZ2UgJGssICR2IDo9IC5EYXRhIH19e3sgJGsgfX06IHt7ICR2IH19XG57eyBlbmQgfX17eyBlbmQgfX0iLCJsZWZ0X2RlbGltaXRlciI6Int7IiwicmlnaHRfZGVsaW1pdGVyIjoifX0ifV19
    Mounts:
      /home/vault from home-sidecar (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9k5qp (ro)
      /vault/secrets from vault-secrets (rw)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-9k5qp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
  home-init:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  home-sidecar:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  vault-secrets:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      Memory
    SizeLimit:   <unset>
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  7m24s  default-scheduler  Successfully assigned default/devwebapp to uday1-control-plane
  Normal  Pulled     7m24s  kubelet            Container image "vault:1.7.3" already present on machine
  Normal  Created    7m24s  kubelet            Created container vault-agent-init
  Normal  Started    7m23s  kubelet            Started container vault-agent-init

osboxes@osboxes:~$ kubectl logs devwebapp -c vault-agent-init
==> Vault agent started! Log data will stream in below:
2021-07-27T19:50:57.835Z [INFO]  sink.file: creating file sink
2021-07-27T19:50:57.836Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2021-07-27T19:50:57.837Z [INFO]  template.server: starting template server
[INFO] (runner) creating new runner (dry: false, once: false)

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.7.3
             Version Sha: 5d517c864c8f10385bf65627891bc7ef55f5e827

[INFO] (runner) creating watcher
2021-07-27T19:50:57.844Z [INFO]  sink.server: starting sink server
2021-07-27T19:50:57.844Z [INFO]  auth.handler: starting auth handler
2021-07-27T19:50:57.845Z [INFO]  auth.handler: authenticating
2021-07-27T19:51:57.847Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1s
2021-07-27T19:51:58.847Z [INFO]  auth.handler: authenticating
2021-07-27T19:52:58.851Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1.91s
2021-07-27T19:53:00.774Z [INFO]  auth.handler: authenticating
2021-07-27T19:54:00.789Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=2.93s
2021-07-27T19:54:03.723Z [INFO]  auth.handler: authenticating
2021-07-27T19:55:03.724Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=4.87s
2021-07-27T19:55:08.595Z [INFO]  auth.handler: authenticating
2021-07-27T19:56:09.043Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=8.89s
2021-07-27T19:56:17.940Z [INFO]  auth.handler: authenticating
2021-07-27T19:57:17.942Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=13.86s
2021-07-27T19:57:31.811Z [INFO]  auth.handler: authenticating
2021-07-27T19:58:31.813Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=21.7s
2021-07-27T19:58:53.516Z [INFO]  auth.handler: authenticating
2021-07-27T19:59:53.521Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=39.73s
2021-07-27T20:00:33.254Z [INFO]  auth.handler: authenticating
2021-07-27T20:01:33.255Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1m7.23s
2021-07-27T20:02:40.492Z [INFO]  auth.handler: authenticating
2021-07-27T20:03:40.493Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=2m7.25s
2021-07-27T20:05:47.752Z [INFO]  auth.handler: authenticating
2021-07-27T20:06:47.756Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=3m42.25

Any suggestion on how to resolve this?

With azure vault and csi driver, able to create secrets and access them as single files in container.

I followed this approach to create basic secrets.

Can accessible the secrets from the container as secret files inside it.

But, when I tried to create a secret from it and use the same for imagePull, it is not working..

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: azure-kvname
  namespace: default
spec:
  provider: azure
  secretObjects:
  - secretName: acr-test
    type: kubernetes.io/dockerconfigjson
    data:
     - objectName: martrepo
       key: .dockerconfigjson
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "false"
    userAssignedIdentityID: ""
    keyvaultName: "secret-store-dummy"
    objects: |
      array:
        - |
          objectName: secret1              
          objectType: secret
          objectVersion: ""
        - |
          objectName: martrepo              
          objectType: secret
          objectVersion: ""    
    tenantId: "f33abe27-86cd-46d6-ae2b-b918362ab160"

---
kind: Pod
apiVersion: v1
metadata:
  name: busybox-secrets-store-inline
spec:
  containers:
  - name: busybox
    image: k8s.gcr.io/e2e-test-images/busybox:1.29
    command:
      - "/bin/sleep"
      - "10000"
    volumeMounts:
    - name: secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  volumes:
    - name: secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-kvname"
        nodePublishSecretRef:                       # Only required when using service principal mode
          name: secrets-store-creds 

The above acr-test secret, I tried to use in deployment of an app at imagePullSecrets, but it didn't wotk as the events showing issues with pulling the image.

Please guide me if the approach is wrong.

And also, how to use these secrets as a part of existing configmap?

I have followed the walkthrough provided in this link

And at step 6, I have tried to create a pod with the volume linked to the keyvault.

But it is not able to access the volume.

kubectl get pods
NAME                                         READY   STATUS              RESTARTS   AGE
busybox-secrets-store-inline                 0/1     ContainerCreating   0          27s
csi-csi-secrets-store-provider-azure-2fwqd   1/1     Running             0          45m
csi-secrets-store-csi-driver-d6zjv           3/3     Running             0          45m


 kubectl describe pod busybox-secrets-store-inline
Name:         busybox-secrets-store-inline
Namespace:    default
Priority:     0
Node:         aks-agentpool-22716505-vmss000001/10.240.0.5
Start Time:   Mon, 26 Jul 2021 21:45:02 +0530
Labels:       <none>
Annotations:  <none>
Status:       Pending
IP:
IPs:          <none>
Containers:
  busybox:
    Container ID:
    Image:         k8s.gcr.io/e2e-test-images/busybox:1.29
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sleep
      10000
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /mnt/secrets-store from secrets-store-inline (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-g7shm (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  secrets-store-inline:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            secrets-store.csi.k8s.io
    FSType:
    ReadOnly:          true
    VolumeAttributes:      secretProviderClass=azure-kvname
  default-token-g7shm:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-g7shm
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age    From               Message
  ----     ------       ----   ----               -------
  Normal   Scheduled    3m50s  default-scheduler  Successfully assigned default/busybox-secrets-store-inline to aks-agentpool-22716505-vmss000001
  Warning  FailedMount  110s   kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Warning  FailedMount  107s   kubelet            Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[secrets-store-inline default-token-g7shm]: timed out waiting for the condition

How to fix this?

I have a 4 node cluster in AWS, which 2 nodes are continuosly getting diconnected and sometimes rebooting works and sometimes need to reboot all the nodes in the cluster to get all back.

[ec2-user@ip-172-31-7-235 ~]$ docker node ls
ID                            HOSTNAME                                      STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
xhei85m3mjp6wikz81phl01sx *   ip-172-31-7-235.us-west-2.compute.internal    Ready     Active         Leader           20.10.4
a63wole6vosq1t5s25wib8ggu     ip-172-31-36-138.us-west-2.compute.internal   Down      Active                          19.03.13-ce
guw26oul1i2fb60f5shud8xif     ip-172-31-47-112.us-west-2.compute.internal   Ready     Active         Reachable        19.03.13-ce
ex996ixxqo3s0mcig1zfzankg     ip-172-31-47-251.us-west-2.compute.internal   Ready     Active                          19.03.13-ce

And the output of inspect command:

[ec2-user@ip-172-31-7-235 ~]$ docker node inspect ip-172-31-36-138.us-west-2.compute.internal
[
    {
        "ID": "a63wole6vosq1t5s25wib8ggu",
        "Version": {
            "Index": 212444
        },
        "CreatedAt": "2021-02-10T13:25:54.271879167Z",
        "UpdatedAt": "2021-07-23T07:36:17.078000983Z",
        "Spec": {
            "Labels": {},
            "Role": "worker",
            "Availability": "active"
        },
        "Description": {
            "Hostname": "ip-172-31-36-138.us-west-2.compute.internal",
            "Platform": {
                "Architecture": "x86_64",
                "OS": "linux"
            },
            "Resources": {
                "NanoCPUs": 2000000000,
                "MemoryBytes": 8362287104
            },
            "Engine": {
                "EngineVersion": "19.03.13-ce",
                "Plugins": [
                    {
                        "Type": "Log",
                        "Name": "awslogs"
                    },
                    {
                        "Type": "Log",
                        "Name": "fluentd"
                    },
                    {
                        "Type": "Log",
                        "Name": "gcplogs"
                    },
                    {
                        "Type": "Log",
                        "Name": "gelf"
                    },
                    {
                        "Type": "Log",
                        "Name": "journald"
                    },
                    {
                        "Type": "Log",
                        "Name": "json-file"
                    },
                    {
                        "Type": "Log",
                        "Name": "local"
                    },
                    {
                        "Type": "Log",
                        "Name": "logentries"
                    },
                    {
                        "Type": "Log",
                        "Name": "splunk"
                    },
                    {
                        "Type": "Log",
                        "Name": "syslog"
                    },
                    {
                        "Type": "Network",
                        "Name": "bridge"
                    },
                    {
                        "Type": "Network",
                        "Name": "host"
                    },
                    {
                        "Type": "Network",
                        "Name": "ipvlan"
                    },
                    {
                        "Type": "Network",
                        "Name": "macvlan"
                    },
                    {
                        "Type": "Network",
                        "Name": "null"
                    },
                    {
                        "Type": "Network",
                        "Name": "overlay"
                    },
                    {
                        "Type": "Volume",
                        "Name": "local"
                    }
                ]
            },
            "TLSInfo": {
                "TrustRoot": "-----BEGIN CERTIFICATE-----\nMIIBajCCARCgAwIBAgIUCi5JL30BEEaYOmlbrp9A+Rivul0wCgYIKoZIzj0EAwIw\nEzERMA8GA1UEAxMIc3dhcm0tY2EwHhcNMjEwMjEwMTMwMjAwWhcNNDEwMjA1MTMw\nMjAwWjATMREwDwYDVQQDEwhzd2FybS1jYTBZMBMGByqGSM49AgEGCCqGSM49AwEH\nA0IABFqgXKora10w8BODSxg9O4N9UveYhsitjwz+pHSi/6BB0j7YBu+4RADv4ZjK\nitIYTCLZZKbOx9saQ2YeB8sBxFajQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMB\nAf8EBTADAQH/MB0GA1UdDgQWBBTETORYsVN1OwUTjtYJHSJtGx55QzAKBggqhkjO\nPQQDAgNIADBFAiEA7qNRnsq0LUFenYODEah4Rku1YYpHBCHIid4W4Hy7MVcCICQF\n9BTfuQsAp5uQ72ycyWQfyQziFzbG+Sb/zQ8NzCRf\n-----END CERTIFICATE-----\n",
                "CertIssuerSubject": "MBMxETAPBgNVBAMTCHN3YXJtLWNh",
                "CertIssuerPublicKey": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWqBcqitrXTDwE4NLGD07g31S95iGyK2PDP6kdKL/oEHSPtgG77hEAO/hmMqK0hhMItlkps7H2xpDZh4HywHEVg=="
            }
        },
        "Status": {
            "State": "down",
            "Message": "heartbeat failure for node in \"unknown\" state",
            "Addr": "172.31.36.138"
        }
    }
]

Please suggest how to backtrace and fix this issue? The issue comes back even after replacing with a new node.

With this we can create a configmap linked to external file.

kubectl create configmap database-config --from-file=database.properties

But when we edit the yaml, it will show that complete content of file is dumped there.

Is there anyway to just point to a file in yaml also so that I will keep the properties file in the same folder as the configmap yaml and appy the folder all the yamls with:

kubectl apply -f target_folder\

Please suggest.

I am trying to delete kind cluster using.

kind delete cluster --name samples

And I got error after waiting for 15 min.

Deleting cluster "samples" ...
ERROR: failed to delete cluster "samples": failed to delete nodes: command "docker rm -f -v samples-control-plane" failed with error: exit status 1
Command Output: Error response from daemon: removal of container samples-control-plane is already in progress

But this is the only command I ran to delete, not before.

So tried to delete the container of kind cluster using.

docker ps

CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS
  NAMES
5d0fa7222662   postgres:latest        "docker-entrypoint.s…"   51 minutes ago   Up 51 minutes   0.0.0.0:5432->5432/tcp, :::5432->5432/tcp   postgres_database
e6acb3781680   kindest/node:v1.21.1   "/usr/local/bin/entr…"   33 hours ago     Up 2 hours      127.0.0.1:44151->6443/tcp
  samples-control-plane

And tried deleting stopping it first.

docker stop e6

But stuck there for almost an hour but no response till now.

So I pressed ctrl+c and ran the command with -t.

docker stop e6 -t 2

Same result, stuck there and kept for 3 hours and more, but still there only. How to delete this?

With single replicaset, the application is working. So I changed the replicaset to 3.

Then, the other two copies of pods are not starting.: kubectl get pods

 customservice-c8645cd6-7gghm                                   0/3     Init:CrashLoopBackOff   8          17m
    customservice-c8645cd6-f5nbn                                   3/3     Running                 0          6h50m
    customservice-c8645cd6-fh57n                                   0/3     Init:CrashLoopBackOff   8          17m


 kubectl describe pod customservice-c8645cd6-fh57n  

 Name:         customservice-c8645cd6-7gghm
Namespace:    default
Priority:     0
Node:         ip-192-168-93-234.us-west-2.compute.internal/192.168.93.234
Start Time:   Tue, 20 Jul 2021 19:34:48 +0530
Labels:       app=customservice
              consul.hashicorp.com/connect-inject-status=injected
              pod-template-hash=c8645cd6
              service=customservice
Annotations:  consul.hashicorp.com/connect-inject: true
              consul.hashicorp.com/connect-inject-status: injected
              consul.hashicorp.com/connect-service: customservice
              consul.hashicorp.com/connect-service-port: 18170
              consul.hashicorp.com/connect-service-upstreams: dashboard:9002
              kubernetes.io/psp: eks.privileged
              prometheus.io/path: /metrics
              prometheus.io/port: 20200
              prometheus.io/scrape: true
Status:       Pending
IP:           192.168.93.88
IPs:
  IP:           192.168.93.88
Controlled By:  ReplicaSet/customservice-c8645cd6
Init Containers:
  consul-connect-inject-init:
    Container ID:  docker://a9bf6bb490f5c21637c18aff681d49d53692f09a3333bf34adb2080816953e26
    Image:         hashicorp/consul:1.9.7
    Image ID:      docker-pullable://hashicorp/consul@sha256:37c7a001af46a68f8e3513bd8180e7f84133d428b0e4ce5cf385d3e54f894760
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -ec

      export CONSUL_HTTP_ADDR="${HOST_IP}:8500"
      export CONSUL_GRPC_ADDR="${HOST_IP}:8502"

      # Register the service. The HCL is stored in the volume so that
      # the preStop hook can access it to deregister the service.
      cat <<EOF >/consul/connect-inject/service.hcl
      services {
        id   = "${SERVICE_ID}"
        name = "customservice"
        address = "${POD_IP}"
        port = 18170
        meta = {
          pod-name = "${POD_NAME}"
          k8s-namespace = "${POD_NAMESPACE}"
        }
      }

      services {
        id   = "${PROXY_SERVICE_ID}"
        name = "customservice-sidecar-proxy"
        kind = "connect-proxy"
        address = "${POD_IP}"
        port = 20000
        meta = {
          pod-name = "${POD_NAME}"
          k8s-namespace = "${POD_NAMESPACE}"
        }

        proxy {
        config {
          envoy_prometheus_bind_addr = "0.0.0.0:20200"
        }
          destination_service_name = "customservice"
          destination_service_id = "${SERVICE_ID}"
          local_service_address = "127.0.0.1"
          local_service_port = 18170
          upstreams {
            destination_type = "service"
            destination_name = "dashboard"
            local_bind_port = 9002
          }
        }

        checks {
          name = "Proxy Public Listener"
          tcp = "${POD_IP}:20000"
          interval = "10s"
          deregister_critical_service_after = "10m"
        }

        checks {
          name = "Destination Alias"
          alias_service = "${SERVICE_ID}"
        }
      }
      EOF

      /bin/consul services register \
        /consul/connect-inject/service.hcl

      # Generate the envoy bootstrap code
      /bin/consul connect envoy \
        -proxy-id="${PROXY_SERVICE_ID}" \
        -prometheus-scrape-path="/metrics" \
        -prometheus-backend-port="20100" \
        -bootstrap > /consul/connect-inject/envoy-bootstrap.yaml

      # Copy the Consul binary
      cp /bin/consul /consul/connect-inject/consul
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 20 Jul 2021 19:51:19 +0530
      Finished:     Tue, 20 Jul 2021 19:51:23 +0530
    Ready:          False
    Restart Count:  8
    Limits:
      cpu:     50m
      memory:  150Mi
    Requests:
      cpu:     50m
      memory:  25Mi
    Environment:
      HOST_IP:            (v1:status.hostIP)
      POD_IP:             (v1:status.podIP)
      POD_NAME:          customservice-c8645cd6-7gghm (v1:metadata.name)
      POD_NAMESPACE:     default (v1:metadata.namespace)
      SERVICE_ID:        $(POD_NAME)-customservice
      PROXY_SERVICE_ID:  $(POD_NAME)-customservice-sidecar-proxy
    Mounts:
      /consul/connect-inject from consul-connect-inject-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from customservice-token-4xf6t (ro)
Containers:
  customservice:
    Container ID:
    Image:          customserverlinux.azurecr.io/custom:latest
    Image ID:
    Port:           18170/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      DASHBOARD_CONNECT_SERVICE_HOST:  127.0.0.1
      DASHBOARD_CONNECT_SERVICE_PORT:  9002
    Mounts:
      /home/spring/AppData/Local/erwin/custom Server/ from custom-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from customservice-token-4xf6t (ro)
  envoy-sidecar:
    Container ID:
    Image:         envoyproxy/envoy-alpine:v1.16.0
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      envoy
      --config-path
      /consul/connect-inject/envoy-bootstrap.yaml
      -l
      debug
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      HOST_IP:            (v1:status.hostIP)
      CONSUL_HTTP_ADDR:  $(HOST_IP):8500
    Mounts:
      /consul/connect-inject from consul-connect-inject-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from customservice-token-4xf6t (ro)
  consul-sidecar:
    Container ID:
    Image:         hashicorp/consul-k8s:0.25.0
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      consul-k8s
      consul-sidecar
      -service-config
      /consul/connect-inject/service.hcl
      -consul-binary
      /consul/connect-inject/consul
      -enable-metrics-merging=true
      -merged-metrics-port=20100
      -service-metrics-port=18170
      -service-metrics-path=/metrics
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     20m
      memory:  50Mi
    Requests:
      cpu:     20m
      memory:  25Mi
    Environment:
      HOST_IP:            (v1:status.hostIP)
      CONSUL_HTTP_ADDR:  $(HOST_IP):8500
    Mounts:
      /consul/connect-inject from consul-connect-inject-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from customservice-token-4xf6t (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  custom-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  customservice-token-4xf6t:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  customservice-token-4xf6t
    Optional:    false
  consul-connect-inject-data:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:   <unset>
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  19m                   default-scheduler  Successfully assigned default/customservice-c8645cd6-7gghm to ip-192-168-93-234.us-west-2.compute.internal
  Normal   Pulled     17m (x5 over 19m)     kubelet            Container image "hashicorp/consul:1.9.7" already present on machine
  Normal   Created    17m (x5 over 19m)     kubelet            Created container consul-connect-inject-init
  Normal   Started    17m (x5 over 19m)     kubelet            Started container consul-connect-inject-init
  Warning  BackOff    3m50s (x68 over 18m)  kubelet            Back-off restarting failed container

As the init container is failing,logs for that container:

kubectl logs customservice-c8645cd6-64j2j -c consul-connect-inject-init

Registered service: customservice
Registered service: customservice-sidecar-proxy
flag provided but not defined: -prometheus-scrape-path
Usage:
  -address value
        LAN address to advertise in the gateway service registration
  -admin-access-log-path string
        The path to write the access log for the administration server. If no access log is desired specify "/dev/null". By default it will use "/dev/null". (default "/dev/null")
  -admin-bind string
        The address:port to start envoy's admin server on. Envoy requires this but care must be taken to ensure it's not exposed to an untrusted network as it has full control over the secrets and config of the proxy. (default "localhost:19000")
  -bind-address <name>=<ip>:<port>
        Bind address to use instead of the default binding rules given as <name>=<ip>:<port> pairs. This flag may be specified multiple times to add multiple bind addresses.
  -bootstrap
        Generate the bootstrap.json but don't exec envoy
  -ca-file value
        Path to a CA file to use for TLS when communicating with Consul. This can also be specified via the CONSUL_CACERT environment variable.
  -ca-path value
        Path to a directory of CA certificates to use for TLS when communicating with Consul. This can also be specified via the CONSUL_CAPATH environment variable.
  -client-cert value
        Path to a client cert file to use for TLS when 'verify_incoming' is enabled. This can also be specified via the CONSUL_CLIENT_CERT environment variable.
  -client-key value
        Path to a client key file to use for TLS when 'verify_incoming' is enabled. This can also be specified via the CONSUL_CLIENT_KEY environment variable.
  -deregister-after-critical string
        The amount of time the gateway services health check can be failing before being deregistered (default "6h")
  -envoy-binary string
        The full path to the envoy binary to run. By default will just search $PATH. Ignored if -bootstrap is used.
  -envoy-version string
        Sets the envoy-version that the envoy binary has. (default "1.16.4")
  -expose-servers
        Expose the servers for WAN federation via this mesh gateway
  -gateway string
        The type of gateway to register. One of: terminating, ingress, or mesh
  -grpc-addr string
        Set the agent's gRPC address and port (in http(s)://host:port format). Alternatively, you can specify CONSUL_GRPC_ADDR in ENV. (default "192.168.93.234:8502")
  -http-addr address
        The address and port of the Consul HTTP agent. The value can be an IP address or DNS address, but it must also include the port. This can also be specified via the CONSUL_HTTP_ADDR environment variable. The default value is http://127.0.0.1:8500. The scheme can also be set to HTTPS by setting the environment variable CONSUL_HTTP_SSL=true.
  -mesh-gateway
        Configure Envoy as a Mesh Gateway.
  -namespace default
        Specifies the namespace to query. If not provided, the namespace will be inferred from the request's ACL token, or will default to the default namespace. Namespaces are a Consul Enterprise feature.
  -no-central-config
        By default the proxy's bootstrap configuration can be customized centrally. This requires that the command run on the same agent as the proxy will and that the agent is reachable when the command is run. In cases where either assumption is violated this flag will prevent the command attempting to resolve config from the local agent.
  -omit-deprecated-tags
        In Consul 1.9.0 the format of metric tags for Envoy clusters was updated from consul.[service|dc|...] to consul.destination.[service|dc|...]. The old tags were preserved for backward compatibility,but can be disabled with this flag.
  -proxy-id string
        The proxy's ID on the local agent.
  -register
        Register a new gateway service before configuring and starting Envoy
  -service string
        Service name to use for the registration
  -sidecar-for string
        The ID of a service instance on the local agent that this proxy should become a sidecar for. It requires that the proxy service is registered with the agent as a connect-proxy with Proxy.DestinationServiceID set to this value. If more than one such proxy is registered it will fail.
  -tls-server-name value
        The server name to use as the SNI host when connecting via TLS. This can also be specified via the CONSUL_TLS_SERVER_NAME environment variable.
  -token value
        ACL token to use in the request. This can also be specified via the CONSUL_HTTP_TOKEN environment variable. If unspecified, the query will default to the token of the Consul agent at the HTTP address.
  -token-file value
        File containing the ACL token to use in the request instead of one specified via the -token argument or CONSUL_HTTP_TOKEN environment variable. This can also be specified via the CONSUL_HTTP_TOKEN_FILE environment variable.
  -wan-address value
        WAN address to advertise in the gateway service registration. For ingress gateways, only an IP address (without a port) is required.

Any suggestion on how to fix this?

I have deployed application in kubernetes. It is working if we expose to a default load balancer service in azure kubernetes. But, after setting a virtual service linked with istio ingress gateway, it is launching only the home page, none of the links are working like /admin /login.

I have added entries for /admin and /login already in virtual service but it is not opening those pages.

kiali launched using

istioctl dashboard kiali

When I check logs in kiali.

[2021-07-18T08:19:51.304Z] "GET /admin/ HTTP/1.1" 404 NR route_not_found - "-" 0 0 0 - "10.244.1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36" "bc1ce347-4151-947b-b3d5-4c08003889a2" "20.85.0.248" "-" - - 10.244.1.9:8080 10.244.1.1:4441 - -

[2021-07-18T08:19:53.400Z] "GET /login/ HTTP/1.1" 404 NR route_not_found - "-" 0 0 0 - "10.244.1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36" "78f9d59e-6126-94f6-99e3-cbffe98e274f" "20.85.0.248" "-" - - 10.244.1.9:8080 10.244.1.1:4441 - -

[2021-07-18T08:19:56.286Z] "GET /login HTTP/1.1" 301 - via_upstream - "-" 0 0 4 4 "10.244.1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36" "be2ff6b8-751d-9150-9d6a-936a436f7fef" "20.85.0.248" "10.244.1.50:3000" outbound|8080||node-service.default.svc.cluster.local 10.244.1.9:59678 10.244.1.9:8080 10.244.1.1:4441 - -

[2021-07-18T08:19:56.525Z] "GET /login/ HTTP/1.1" 404 NR route_not_found - "-" 0 0 0 - "10.244.1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36" "f821db0b-cc9d-9b8c-9769-71ac7636a87e" "20.85.0.248" "-" - - 10.244.1.9:8080 10.244.1.1:4441 - -

[2021-07-18T08:20:00.198Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 2804 4 4 "10.244.1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36" "43c37e75-f0ce-9474-bc4d-ea4459772f6d" "20.85.0.248" "10.244.1.50:3000" outbound|8080||node-service.default.svc.cluster.local 10.244.1.9:59678 10.244.1.9:8080 10.244.1.1:4441 - -

[2021-07-18T08:20:00.457Z] "GET /static/todo/logo.png HTTP/1.1" 404 NR route_not_found - "-" 0 0 0 - "10.244.1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36" "a26d3710-3868-96b6-9d6d-7f6ee05dc131" "20.85.0.248" "-" - - 10.244.1.9:8080 10.244.1.1:4441 - -

[2021-07-18T08:20:34.337Z] "GET /login/ HTTP/1.1" 404 NR route_not_found - "-" 0 0 0 - "10.244.1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36" "e8e73933-f95e-90a7-bdf4-ad39352982c5" "20.85.0.248" "-" - - 10.244.1.9:8080 10.244.1.1:4441 - -

[2021-07-18T08:20:37.506Z] "GET /login/ HTTP/1.1" 404 NR route_not_found - "-" 0 0 0 - "10.244.1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36" "bee90116-2d78-99dd-a4b7-fede0641fed8" "20.85.0.248" "-" - - 10.244.1.9:8080 10.244.1.1:4441 - -

[2021-07-18T08:20:40.191Z] "GET /login/ HTTP/1.1" 404 NR route_not_found - "-" 0 0 0 - "10.244.1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36" "f9df634e-743f-9699-a85d-40926db41ca4" "20.85.0.248" "-" - - 10.244.1.9:8080 10.244.1.1:4441 - -

Please suggest how to fix this?

The sample app docker link.

The kubernetes yaml files used:

Deployment.yaml

apiVersion: v1
kind: Service
metadata:
  name: node-service
  labels:
    app: nodeapp
    service: node-service
spec:
  selector:
    app: nodeapp
  ports:
    - name: http
      port: 8080
      targetPort: 3000 #80

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nodeapp-details
  labels:
    account: nodeapp-details
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nodeapp-deployment
  labels:
    app: nodeapp
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nodeapp
      version: v1
  template:
    metadata:
      labels:
        app: nodeapp
        version: v1
    spec:
      serviceAccountName: nodeapp-details
      containers:
        - name: nginx
          image: uday1kiran/todoapp:1.0 #nginx
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000 #80
              name: nodeapp-port
          #securityContext:
          #runAsUser: 1000

virtualservice.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nodeserver
spec:
  hosts:
    - "*"
  gateways:
    - node-gateway
  http:
    - match:
        - uri:
            exact: /
        - uri:
            exact: /admin
        - uri:
            exact: /login
      route:
        - destination:
            host: node-service
            port:
              number: 8080

gateway.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: node-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"

The yaml files are in this repository

I have tried to deploy a sample nginx app and it working. So, I replaced the same with another nodejs as this contains mutliple sub pages. So, I can test the routing.

Here, with LoadBalancer default service type, the page is loading as below.

But, with istio gateway, it is just showing an empty page.

Please suggest how to fix that?

My gateway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: sampleserver-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"

My virtualservice:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginxserver
spec:
  hosts:
  - "*"
  gateways:
  - sampleserver-gateway
  http:
  - match:
    - uri:
        exact: /
    route:
    - destination:
        host: nginx-service
        port:
          number: 8080

My deployment and service file:

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  labels:
    app: nginx
    service: nginx
spec:
  selector:
    app: nginx
  ports:
    - name: http 
      port: 8080
      targetPort: 3000 #80

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-details
  labels:
    account: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
      version: v1
  template:
    metadata:
      labels:
        app: nginx
        version: v1
    spec:
      serviceAccountName: nginx-details
      containers:
        - name: nginx
          image: uday1kiran/getting-started:1.0 #nginx
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000 #80
              name: nginx-port
          #securityContext:
              #runAsUser: 1000

For gateway, I set the default kubernetes service as ClusterIP. But, as it is not showing, for testing purpose I changed it to LoadBalancer to check any issue with app, but throug normal LoadBalancer service it is loading but not with istio ingresss gateway.

The sample app is public image, you can test directly

The source code of application is in app folder of repo

I created an nginx deployment and got below error.

root@kmaster:~# kubectl describe pod nginx
Name:           nginx-6799fc88d8-ltnc5
Namespace:      default
Priority:       0
Node:           kworker1/10.8.0.70
Start Time:     Wed, 14 Jul 2021 16:26:32 +0000
Labels:         app=nginx
                pod-template-hash=6799fc88d8
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/nginx-6799fc88d8
Containers:
  nginx:
    Container ID:   
    Image:          nginx
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ngqhw (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-ngqhw:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                    From               Message
  ----     ------                  ----                   ----               -------
  Normal   Scheduled               7m27s                  default-scheduler  Successfully assigned default/nginx-6799fc88d8-ltnc5 to kworker1
  Warning  FailedCreatePodSandBox  7m27s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "bd465da8133b3b31cea3ccf75b0bbf718336f4a06fea40aeef5ddf9a2ba74b96": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  7m16s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "eb61496041afad0a51a989e1fe1898db37a7356cd1047e0e9bad73d5bfcc9abc": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  7m5s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a56c7fe7d8b102f38c4a84d514e18f5b995210c8efa3e945f6f12d51fc27b72b": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  6m54s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1ebe31cebdac61145dbacbca661c2595ca267f4112fe77f69e7e1cd1f04b97bf": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  6m42s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3de40514e0b210f27e5fe3e3c70afafef5418a23e1faed886358766d2454d434": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  6m29s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1fba1c9980010c57d1dbb8ed8c6554ca16957f3b1773874fa8d652442212f6ed": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  6m17s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f1296e109c11689f7f01a0c5cb8483b6d47269092dc0851afce21bb17d873dca": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  6m6s                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "23bfe01e93b6639f4953ee8590576e4c3784a01372afcf609affbd03d5b26398": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  5m51s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "2a589a2995a530c7d9b11abd51e7315852285078d05717aefa6c8f4c0e928866": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  119s (x17 over 5m37s)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "cd97f555abf734ef27d19ab7a13f1935dbf2e4fd2fca9e9cbb986061f174a46b": open /run/flannel/subnet.env: no such file or directory

I applied this to create flannel network Please suggest how to fix this?

lxc containers:

dminuser@testing:~/Desktop/kubernetes/lxd-provisioning$ lxc list
+----------+---------+-------------------+-----------------------------------------------+-----------+-----------+
|   NAME   |  STATE  |       IPV4        |                     IPV6                      |   TYPE    | SNAPSHOTS |
+----------+---------+-------------------+-----------------------------------------------+-----------+-----------+
| kmaster  | RUNNING | 10.8.0.129 (eth0) | fd42:666f:471d:3d53:216:3eff:fedb:8af1 (eth0) | CONTAINER | 0         |
+----------+---------+-------------------+-----------------------------------------------+-----------+-----------+
| kworker1 | RUNNING | 10.8.0.229 (eth0) | fd42:666f:471d:3d53:216:3eff:fe93:ca5e (eth0) | CONTAINER | 0         |
+----------+---------+-------------------+-----------------------------------------------+-----------+-----------+
| kworker2 | RUNNING | 10.8.0.152 (eth0) | fd42:666f:471d:3d53:216:3eff:fe23:b3a4 (eth0) | CONTAINER | 0         |
+----------+---------+-------------------+-----------------------------------------------+-----------+-----------+

some output related to pods:

oot@kmaster:~# kubectl get nodes
NAME       STATUS   ROLES                  AGE     VERSION
kmaster    Ready    control-plane,master   3m44s   v1.21.0
kworker1   Ready    <none>                 114s    v1.21.0
kworker2   Ready    <none>                 99s     v1.21.0

oot@kmaster:~# kubectl get pods -n kube-system
NAME                              READY   STATUS              RESTARTS   AGE
coredns-558bd4d5db-569mm          0/1     ContainerCreating   0          4m1s
coredns-558bd4d5db-pzlfh          0/1     ContainerCreating   0          4m1s
etcd-kmaster                      1/1     Running             0          4m4s
kube-apiserver-kmaster            1/1     Running             0          4m4s
kube-controller-manager-kmaster   1/1     Running             0          4m4s
kube-flannel-ds-8mfsv             0/1     Error               2          2m14s
kube-flannel-ds-jv9rg             1/1     Running             4          4m1s
kube-flannel-ds-wms5n             0/1     CrashLoopBackOff    2          2m29s
kube-proxy-k8mnl                  0/1     CrashLoopBackOff    5          4m1s
kube-proxy-mw8ff                  0/1     CrashLoopBackOff    4          2m14s
kube-proxy-qkpzl                  0/1     CrashLoopBackOff    4          2m29s
kube-scheduler-kmaster            1/1     Running             0          4m4s
root@kmaster:~# 


oot@kmaster:~# kubectl logs kube-flannel-ds-8mfsv 
Error from server (NotFound): pods "kube-flannel-ds-8mfsv" not found
root@kmaster:~# kubectl logs kube-flannel-ds-8mfsv -n kube-system
I0715 08:13:32.125253       1 main.go:520] Determining IP address of default interface
I0715 08:13:32.125810       1 main.go:533] Using interface with name eth0 and address 10.8.0.152
I0715 08:13:32.125874       1 main.go:550] Defaulting external address to interface address (10.8.0.152)
W0715 08:13:32.125954       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
E0715 08:14:02.128488       1 main.go:251] Failed to create SubnetManager: error retrieving pod spec for 'kube-system/kube-flannel-ds-8mfsv': Get "https://10.96.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-8mfsv": dial tcp 10.96.0.1:443: i/o timeout

For testing purpose, I have installed ubuntu 21 on vmware esxi server. On that machine, spinned up kubernetes using lxc containers following this repository LXC is spinned up and running.

adminuser@testing:~/Desktop$ lxc list
+----------+---------+-------------------+-----------------------------------------------+-----------+-----------+
|   NAME   |  STATE  |       IPV4        |                     IPV6                      |   TYPE    | SNAPSHOTS |
+----------+---------+-------------------+-----------------------------------------------+-----------+-----------+
| kmaster  | RUNNING | 10.8.0.217 (eth0) | fd42:666f:471d:3d53:216:3eff:fe54:dce6 (eth0) | CONTAINER | 0         |
+----------+---------+-------------------+-----------------------------------------------+-----------+-----------+
| kworker1 | RUNNING | 10.8.0.91 (eth0)  | fd42:666f:471d:3d53:216:3eff:fee4:480e (eth0) | CONTAINER | 0         |
+----------+---------+-------------------+-----------------------------------------------+-----------+-----------+
| kworker2 | RUNNING | 10.8.0.124 (eth0) | fd42:666f:471d:3d53:216:3eff:fede:3c9d (eth0) | CONTAINER | 0         |
+----------+---------+---------------

Then started deploying metallb on this cluster using the steps mentioned in this link. And applied this configmap for routing. GNU nano 4.8 k8s-metallb-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 10.8.0.240-10.8.0.250

But the metallb pods are not running.

kubectl get pods -n metallb-system
NAME                          READY   STATUS                       RESTARTS   AGE
controller-6b78bff7d9-cxf2z   0/1     ContainerCreating            0          38m
speaker-fpvjt                 0/1     CreateContainerConfigError   0          38m
speaker-mbz7b                 0/1     CreateContainerConfigError   0          38m
speaker-zgz4d                 0/1     CreateContainerConfigError   0          38m

I checked the logs.

kubectl describe pod controller-6b78bff7d9-cxf2z -n metallb-system
Name:           controller-6b78bff7d9-cxf2z
Namespace:      metallb-system
Priority:       0
Node:           kworker1/10.8.0.91
Start Time:     Wed, 14 Jul 2021 20:52:10 +0530
Labels:         app=metallb
                component=controller
                pod-template-hash=6b78bff7d9
Annotations:    prometheus.io/port: 7472
                prometheus.io/scrape: true
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/controller-6b78bff7d9
Containers:
  controller:
    Container ID:  
    Image:         quay.io/metallb/controller:v0.10.2
    Image ID:      
    Port:          7472/TCP
    Host Port:     0/TCP
    Args:
      --port=7472
      --config=config
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:
      METALLB_ML_SECRET_NAME:  memberlist
      METALLB_DEPLOYMENT:      controller
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j76kg (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-j76kg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                 From               Message
  ----     ------                  ----                ----               -------
  Normal   Scheduled               32m                 default-scheduler  Successfully assigned metallb-system/controller-6b78bff7d9-cxf2z to kworker1
  Warning  FailedCreatePodSandBox  32m                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a8a6fa54086b9e65c42c8a0478dcac0769e8b278eeafe11eafb9ad5be40d48eb": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31m                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "264ee423734139b712395c0570c888cff0b7b526e5154da0b7ccbdafe5bd9ba3": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31m                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "1a3cb9e20a2a015adc7b4924ed21e0b50604ee9f9fae52170c03298dff0d6a78": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31m                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "56dd906cdadc8ef50db3cc725d988090539a0818c2579738d575140cebbec71a": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  31m                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8ddcfa704da9867c3a68030f0dc59f7c0d04bdc3a0b598c98a71aa8787585ca6": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  30m                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "50431bbc89188799562c48847be90e243bbf49a2c5401eb2219a0c4745cfcfb6": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  30m                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "da9ad1d418d3aded668c53f5e3f98ddfac14af638ed7e8142b904e12a99bfd77": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  30m                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4dc6109c696ee410c58a0894ac70e5165a56bab99468ee42ffe88b2f5e33ef2f": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  30m                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a8f1cad2ce9f8c278c07c924106a1b6b321a80124504737a574bceea983a0026": open /run/flannel/subnet.env: no such file or directory
  Warning  FailedCreatePodSandBox  2m (x131 over 29m)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f5e93b893275afe5309eddd9686c0ecfeb01e91141259164082cb99c1e2c1902": open /run/flannel/subnet.env: no such file or directory

And the speaker container.

kubectl describe pod  speaker-zgz4d -n metallb-system
Name:         speaker-zgz4d
Namespace:    metallb-system
Priority:     0
Node:         kmaster/10.8.0.217
Start Time:   Wed, 14 Jul 2021 20:52:10 +0530
Labels:       app=metallb
              component=speaker
              controller-revision-hash=7668c5cdf6
              pod-template-generation=1
Annotations:  prometheus.io/port: 7472
              prometheus.io/scrape: true
Status:       Pending
IP:           10.8.0.217
IPs:
  IP:           10.8.0.217
Controlled By:  DaemonSet/speaker
Containers:
  speaker:
    Container ID:  
    Image:         quay.io/metallb/speaker:v0.10.2
    Image ID:      
    Ports:         7472/TCP, 7946/TCP, 7946/UDP
    Host Ports:    7472/TCP, 7946/TCP, 7946/UDP
    Args:
      --port=7472
      --config=config
    State:          Waiting
      Reason:       CreateContainerConfigError
    Ready:          False
    Restart Count:  0
    Environment:
      METALLB_NODE_NAME:       (v1:spec.nodeName)
      METALLB_HOST:            (v1:status.hostIP)
      METALLB_ML_BIND_ADDR:    (v1:status.podIP)
      METALLB_ML_LABELS:      app=metallb,component=speaker
      METALLB_ML_SECRET_KEY:  <set to the key 'secretkey' in secret 'memberlist'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-l2gzm (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-l2gzm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node-role.kubernetes.io/master:NoSchedule op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    41m                  default-scheduler  Successfully assigned metallb-system/speaker-zgz4d to kmaster
  Warning  FailedMount  41m                  kubelet            MountVolume.SetUp failed for volume "kube-api-access-l2gzm" : failed to sync configmap cache: timed out waiting for the condition
  Warning  Failed       39m (x12 over 41m)   kubelet            Error: secret "memberlist" not found
  Normal   Pulled       78s (x185 over 41m)  kubelet            Container image "quay.io/metallb/speaker:v0.10.2" already present on machine

container state after setting the value from null to 0.

kube-apiserver-kmaster            1/1     Running             0          27m
kube-controller-manager-kmaster   1/1     Running             0          27m
kube-flannel-ds-7f5b7             0/1     CrashLoopBackOff    1          76s
kube-flannel-ds-bs9h5             0/1     Error               1          72s
kube-flannel-ds-t9rpf             0/1     Error               1          71s
kube-proxy-ht5fk                  0/1     CrashLoopBackOff    3          76s
kube-proxy-ldhhc                  0/1     CrashLoopBackOff    3          75s
kube-proxy-mwrkc                  0/1     CrashLoopBackOff    3          76s
kube-scheduler-kmaster            1/1     Running             0          2

I have two subscriptions under my account. When I check today, the balance is low in my subscription where I deployed the cluster.

So, I click on change subscription option and selected the second subscription in my account.

But, getting below error.

{"code":"ResourceMoveValidationFailed","message":"The resource batch move request has '1' validation errors. Diagnostic information: timestamp '20210708T074027Z', tracking Id 'bae55f37-137e-4388-ba57-a46456b5efb9', request correlation Id '2ffbd2bc-1198-42ec-b8c5-60cbaed7a160'.","details":[{"code":"ResourceMoveNotSupported","target":"/subscriptions/f733ab10-e6a2-406d-9c23-cafe4ec0e71e/resourceGroups/testmart/providers/Microsoft.ContainerService/managedClusters/testmart","message":"Resource move is not supported for resource types 'Microsoft.ContainerService/managedClusters'."}]}

How to move the cluster then?

I have followed the steps of installation from metallb website.

And the metallb containers are also running.

After that I deployed an nginx service for testing with service as LoadBalancer. The LoadBalancer is not getting IP from metallb.

Any suggestions on how to fix this?

With Azure, when we connect to a cloud machine, we can install Hyper-V on top of it and can load multiple vms inside the VM again.

But similar option is not found in the aws ec2 instance.

Any suggestion on how to enable it if any such option.

I'm trying to create an application users for my Aurora RDS using Terraform. I can only find links for creating a MySQL user or a Postgresql user but not a generic one for Aurora.

I can see the last successful build artifact in jenkins once I open the job page.

artifact

How, to get previous build artifacts?