Home / server / Questions / 1165464
Accepted
Seididieci
Asked: 2024-09-19 21:35:45 +0800 CST

kubernetes metrics-server gets 403 from all nodes

  • 772

I've created a kubernetes cluster using this guide for fedora: https://docs.fedoraproject.org/en-US/quick-docs/using-kubernetes/

I set up 1 control-pane and 1 work node (2 libvirt VMs) and I am able to deploy applications ingresses run things and so on...

The only thing I miss is getting some metrics (kubectl top nodes/kubectl top pods), so I tried to install the metrics-server using the official yaml file (https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml).

Then i edited the deployment to fix the x509 certificate errors adding --kubelet-insecure-tls to the args list (this is not a production cluster so i left default self-signed certs).

My problem there is that metrics server does not work: in the log it shows errors scraping nodes because it gets 403 (forbidden) when asking for resources:"

I0919 08:32:37.415029       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key"
I0919 08:32:37.415090       1 secure_serving.go:213] Serving securely on [::]:10250
I0919 08:32:37.415119       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
E0919 08:32:37.417167       1 scraper.go:149] "Failed to scrape node" err="request failed, status: \"403 Forbidden\"" node="f40-work1"
E0919 08:32:37.421847       1 scraper.go:149] "Failed to scrape node" err="request failed, status: \"403 Forbidden\"" node="f40main"

I'm stuck there and cannot figure out how to solve this, google can't help me...

Node versions:

NAME        STATUS   ROLES                                            AGE     VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                           KERNEL-VERSION            CONTAINER-RUNTIME
f40-work1   Ready    metrics-server-aggregated-reader                 2d19h   v1.29.8   192.168.124.128   <none>        Fedora Linux 40 (Server Edition)   6.10.10-200.fc40.x86_64   cri-o://1.28.2
f40main     Ready    control-plane,metrics-server-aggregated-reader   2d20h   v1.29.8   192.168.124.146   <none>        Fedora Linux 40 (Server Edition)   6.10.9-200.fc40.x86_64    cri-o://1.28.2

I labeled both nodes with Role metrics-server-aggregate-reader trying to fix errors but no luck...

kubernetes

1 Answers

  1. Best Answer

    So far there's only one UGLIEST hack that has worked for me on 1.24+ k8s cluster versions (and yes, I've read and followed tonnes of issues and tried various approaches, including direct questions to metrics-server authors, none worked for me (however, some has seem to work for some people)):

    Disclaimer: DO NOT FOLLOW

    change --authorization-mode=Webhook to --authorization-mode=AlwaysAllow in kubelet startup environment (redhat-based instances have it defined in unit-file, I prefer to use drop-in config inclusion in /etc/systemd/system/kubelet/service.d) and restart kubelet.

    This hack effectively disables the authorization on kubelet service port 10250, so anyone having network connectivity to it can do about all nasty things he can imagine with it.

    Any more elegant working solution is welcome below.

    • 1