Questions[kubectl](server)

Thank you for reading and taking your time to review this problem.

I have a problem using my Kubernetes cluster. It is running Flatcar Linux, made by Kinvolk, recently acquired by Microsoft. I have setup the cluster using their Lokomotive (lokoctl) tool.

I have 4 nodes in total.

• socrates001 (master)
• socrates002 (node)
• socrates003 (node)
• socrates004 (node)

Today, around 2 PM, my master node restarted because of the auto update service provided by Lokomotive (the cluster management tool made by Kinvolk).

My master node came back up, however the k8s did not.

The output of docker container ls ran on socrates001 is the following:

CONTAINER ID   IMAGE                             COMMAND                  CREATED         STATUS         PORTS     NAMES
e33995c69e10   quay.io/kinvolk/kubelet:v1.21.4   "/usr/local/bin/kube…"   7 minutes ago   Up 7 minutes             kubelet
b6093a1f343a   quay.io/coreos/etcd:v3.4.16       "/usr/local/bin/etcd"    7 minutes ago   Up 7 minutes             etcd

This indicates that Kubelet and Etcd are running. Kubelet however is giving me a lot of errors, and honestly, I would not know where to start digging...

When I run journalctl -u kubelet, it gives me the following output, I've put it in a Pastebin because it's too big. Warning, it's a big one. https://pastebin.com/A9Lmf0tc

Things I've already tried:

• Rebooting the master node
• Restarting kubelet
• Restarting etcd
• Manually trying to start up the kube api, however this immediately gets terminated (by kubelet I think)
• Forced swap off with sudo swapoff -a, however during provisioning of the cluster, I'm quite sure that Lokomotive already turns it off on Flatcar Linux.

I have no clue why this is happening at all, so all comments and answers are welcome! I'm a student with quite a bit of time, so you should get a reply quickly.

Thanks in advance!

Edit: it looks like there is a bug in the Kubelet Checkpointer. I have filed an issue with Lokomotive here: https://github.com/kinvolk/lokomotive/issues/1576

I'm trying to debug a distroless POD by trying kubectl debug which uses ephemeral containers that are disabled by default in aws eks. I'm using aws eks 1.21

✗  kubectl debug -it opentelemetry-collector-agent-6hqvf --image=busybox --target=some-app

error: ephemeral containers are disabled for this cluster (error from server: "the server could not find the requested resource")

So how to enable them for EKS?

I have tried using kustomize to load properties file as a configmap.

For that, I created a sample set as in github link.

With base files:

#kustomize build base
apiVersion: v1
data:
  config: |-
    dbport=1234
    dcname=sfsdf
    dbssl=false
    locktime=300
    domainuser=
kind: ConfigMap
metadata:
  labels:
    owner: sara
  name: database-configmap
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
    owner: sara
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
      owner: sara
  template:
    metadata:
      labels:
        app: nginx
        owner: sara
    spec:
      containers:
      - image: nginx
        name: nginx

With external file:

#kustomize build file
apiVersion: v1
data:
  config: "dbport=156767\r\ndcname=dfsd\r\ndbssl=false\r\nlocktime=300\r\ndomainuser=somedts"
kind: ConfigMap
metadata:
  labels:
    env: dev
    owner: sara
  name: dev-database-configmap
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
    env: dev
    owner: sara
  name: dev-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
      env: dev
      owner: sara
  template:
    metadata:
      labels:
        app: nginx
        env: dev
        owner: sara
    spec:
      containers:
      - image: nginx
        name: nginx

If you observe the configmap | is removed and also replaced by \r\n as a single string. How to fix this alignment?

In have scaled my pods to 20 in my cluster and when I see the next day the few of the scaled pods were recreated.

When I say POD recreated, it is getting deleted and created freshly and the timestamp of the recreated pod and the scaled pods vary.

I was unable to find the reasons for the recreate of the PODs.

I could not find which POD went for a recreate as the POD is deleted and gone away. There are no logs in the journalctl regarding which POD got recreated. Is there any way I can debug further to find the reason for the POD recreate. or What might be the reason for the PODs getting deleted.

Note: I have readiness and liveness probes defined, but these probes would act on container and would not lead to POD recreate is my understanding.

I have a Kubernetes cluster which is running in a private cloud. I want to run some commands from another VM but I receive this:

[root@runner-tmp ~]# kubectl get pods --kubeconfig local-cluster.yaml
error: tls: failed to find any PEM data in certificate input

My local-cluster.yaml:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://x.x.x.x:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    namespace: FSM
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

Do you have any idea where should I specify this PEM certificate and how can I generate it?