Home / user-954414

RedBluff's questions

Martin Hope
RedBluff
Asked: 2022-03-01 23:13:07 +0800 CST
  • 0

I'm struggling with Microsoft Teams/Prometheus integration on K8s cluster. I used helm to start all components. I have correctly working Prometheus and Alertmanager. It seems that all works fine. Prometheus communicate with Alertmanager. Then prometheus-msteams receives POST alert from Alert Manager and it should send it to a Microsoft Teams Channel but it's not.

2022/03/01 06:49:38 [DEBUG] POST https://xxx.webhook.office.com/webhookb2/xxx-xxx-xxx/IncomingWebhook/xxx
2022/03/01 06:50:08 [ERR] POST https://xxx.webhook.office.com/webhookb2/xxx-xxx-xxx/IncomingWebhook/xxx request failed: Post https://xxx.webhook.office.com/webhookb2/xxx-xxx-xxx/IncomingWebhook/xxx: dial tcp 42.12.12.542:443: i/o timeout

30s and timeout. I thought that it may be a proxy issue. So I added extraEnv parameter to config map, restarted pod but nothing changed. So my configuration looks like:

apiVersion: v1
data:
  connectors.yaml: |
    connectors:
      - alertmanager-warning: https://xxx.webhook.office.com/webhookb2/xxx-xxx-xxx/IncomingWebhook/xxx
      - alertmanager-critical: https://xxx.webhook.office.com/webhookb2/xxx-xxx-xxx/IncomingWebhook/xxx

    extraEnvs:
      HTTPS_PROXY: http://my-proxy.com:911
kind: ConfigMap
metadata:

I also loged into container to check if /etc/config/connectors.yaml is OK. I'm afraid that this extraEnvs doesn't work somehow. From K8s worker node I tried manually (with curl) post some test json to MS Chanel and without proxy it hanged. When I exported HTTPS_PROXY var the message was sucessfully created in MS Teams Chanel.

 export HTTPS_PROXY=http://my-proxy.com:911
 curl -X POST -d @test.json https://xxx.webhook.office.com/webhookb2/xxx-xxx-xxx/IncomingWebhook/xxx

Do you have any idea what can cause the problem? Is this HTTPS_PROXY env should be listed when I type printvenv in prometheus-msteams container?

prometheus docker kubernetes alertmanager
Martin Hope
RedBluff
Asked: 2022-02-11 04:01:40 +0800 CST
  • 0

I'm new in K8s and I'm facing a problem with certificate. 1.13 version is used. One of the worker nodes is in NotReady status. I check logs and it turned out that:

Part of the existing bootstrap client certificate is expired
Using bootstrap kubeconfig to generate TLS client cert, key and kubeconfig file
Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem"
Failed to connect to apiserver: the server has asked for the client to provide credentials

I checked and indeed kubelet-client-current.pem is pointing to expired certificate. My question is how to renew this kubelet-client-<current_date>.pem file.

EDIT:
I read that for this current cert: The initial validity period is one year. When the certificate is about to expire, it is automatically renewed and the validity period is extended by one year. Is there any way to create it manually? Or how to force kubeadm to do it?

I also deleted /var/lib/kubelet/pki and after kubelet restart it was recreated but only with kubelet-client.key.tmp, kubelet.crt, kubelet.key.

kubernetes