Questions[k3s](server)

In a k3s cluster (with multiple control-plane nodes) and with Rancher Longhorn installed, I am observing the following warning for pods that have a pvc with the default longhorn storage class (output of kubectl get events, wrapped for better readability):

LAST SEEN   TYPE      REASON        OBJECT                      
113s        Warning   FailedMount   pod/grafana-6756f6587b-rv2xj

   MESSAGE
   MountVolume.SetUp failed for volume "pvc-7b6d12e3-132d-4af1-99c0-920ac5af0687" :
     rpc error:
       code = Aborted
       desc = no Pending workload pods for volume pvc-7b6d12e3-132d-4af1-99c0-920ac5af0687
              to be mounted: map[Running:[grafana-6756f6587b-rv2xj]]

What does this recurring warning mean and what action is required to fix it?

At least from what I can see in the pod's container, the volume has been mounted. I already tried restarting the pod and also recreating the pod, but the warning still persists. Also the longhorn dashboard does not show any problems.

See below for the pvc and deployment resource definitions:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  storageClassName: longhorn
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
  labels:
    app: grafana
spec:
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      securityContext:
        fsGroup: 472
        supplementalGroups:
          - 0
      containers:
        - name: grafana
          image: grafana/grafana:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http-grafana
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /robots.txt
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 2
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3000
            timeoutSeconds: 1
          resources:
            requests:
              cpu: 250m
              memory: 750Mi
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafana-pv
      volumes:
        - name: grafana-pv
          persistentVolumeClaim:
            claimName: grafana-pvc

I've set up a K3S Kubernetes Environment in my private Home-Lab on Raspberry PIs in order to teach myself some Kubernetes (Noob-Alert), using NGINX as Ingress Controller and I'm kind of stuck at passing the real IP of requests to the target Pods, in my case a Nextcloud instance. The Version of K3S is v1.22.5+k3s1.

The K3S was set up using Docker as container runtime and with the --no-deploy traefik option.

After that I deployed the NGINX Ingress Controller using

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.0/deploy/static/provider/baremetal/deploy.yaml

Then, after deploying the Nextcloud pods, I deployed the Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-ingress
  annotations:
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
  tls:
  - hosts:
    - my.own-dns.org
    secretName: very-secret-ssl-secret
  ingressClassName: nginx
  rules:
  - host: my.own-dns.org
    http:
      paths:
        - path: /somepath
          pathType: Prefix
          backend:
            service:
              name: someservice-service
              port:
                number: 8081
        - path: /
          pathType: Prefix
          backend:
            service:
              name: nextcloud-service
              port:
                number: 80

In the deployment of the IngressController, I added the following entries in the ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  allow-snippet-annotations: 'true'
  compute-full-forwarded-for: 'true'
  use-forwarded-headers: 'true'
  enable-real-ip: 'true'
  proxy-add-original-uri-header: 'true'
  forwarded-for-header: 'X-Forwarded-For'

and changed the ServiceType from the http service to LoadBalancer, so my Service of the IngressController looks like this:

apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    helm.sh/chart: ingress-nginx-4.0.10
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.1.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: LoadBalancer
  ipFamilyPolicy: SingleStack
  ipFamilies:
    - IPv4
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
      appProtocol: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
      appProtocol: https
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller

So far so good, accessing the Nextcloud Instance from the Internet working great, including redirection to https, etc. But the Nextcloud Audit Log is only getting an internal Cluster IP as Source IP (surprisingly no IP from any Service I am running inside the Cluster), not the real from the outer world.

So what am I missing? I tried setting use-proxy-protocol to true, but this results in a ERR_CONNECTION_RESET.

I have a K3s cluster with system pods (i.e. kube-system namespace) and my application pods:

kube-system   pod/calico-node-xxxx                          
kube-system   pod/calico-kube-controllers-xxxxxx   
kube-system   pod/metrics-server-xxxxx
kube-system   pod/local-path-provisioner-xxxxx
kube-system   pod/coredns-xxxxx
app-system    pod/my-app-xxxx
db-system     pod/my-db-xxxxx

I am looking for a shell/kubectl command (for automation script) that can delete my application namespace (want to modify kubectl delete namespace app-system db-system) without mentioning the app namespaces name in the command (as in future if more app namespace would come in cluster, I have to edit this script every time).

That means I want to delete all namespace in cluster except kube-system

something like - kubectl delete namespace -v kube-system (I know -v is not a valid parameter here, just showing how in grep -v is used to except the following words. Similar thing looking for kubectl delete ns... )

I have a k3s (v1.21.1+k3s1) cluster with following pods:

kube-system   pod/calico-node-xxxx                          
kube-system   pod/calico-kube-controllers-xxxxxx   
kube-system   pod/metrics-server-xxxxx
kube-system   pod/local-path-provisioner-xxxxx
kube-system   pod/coredns-xxxxx

How can I restart (stop and start the pods again) the pods either with command (kubectl maybe) or any script?

I am aware of the consequences and issues with running a single-node cluster. However, I'm still curious if it's possible. I plan on setting everything up myself.

In other words, can I run the control plane and a worker node on the same physical machine.