Home / server / Questions

Questions[rancher-2](server)

Martin Hope
Taxel
Asked: 2022-03-31 00:42:05 +0800 CST
  • 1

I have set up Postgres in Kubernetes (k3s administered with Rancher) which seems to be working fine but in order to take a look at the databases I tried to add a pgadmin4 pod.

I first tried adding the image dpage/pgadmin4 without any additional configuration (except for the a node port from port 80 and the PGADMIN_DEFAULT_EMAIL and PGADMIN_DEFAULT_PASSWORD env var), got the error below and tried a few things since then:

  • set PGADMIN_LISTEN_ADDRESS env var to 0.0.0.0 in case IPv6 is not supported
  • add a volume mount (from longhorn) for /var/lib/pgadmin

The pod is failing to start with the following log messages:

NOTE: Configuring authentication for SERVER mode.
sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?
[2022-03-30 08:27:31 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2022-03-30 08:27:31 +0000] [1] [ERROR] Retrying in 1 second.
[2022-03-30 08:27:32 +0000] [1] [ERROR] Retrying in 1 second.
[2022-03-30 08:27:33 +0000] [1] [ERROR] Retrying in 1 second.
[2022-03-30 08:27:34 +0000] [1] [ERROR] Retrying in 1 second.
[2022-03-30 08:27:35 +0000] [1] [ERROR] Retrying in 1 second.
[2022-03-30 08:27:36 +0000] [1] [ERROR] Can't connect to ('0.0.0.0', 80)

There seem to be two things wrong:

  • the sudo error, which results from pgadmin4 starting with userid 5050 but I don't know how to solve this
  • gunicorn not being able to listen on port 80

I am not very familiar with Kubernetes so I just don't know which options I have to fix this.

Thanks in advance for any help.

postgresql kubernetes k3s rancher-2
Martin Hope
crowne
Asked: 2022-03-11 22:01:33 +0800 CST
  • 0

I asked this here yesterday.

Can anyone advise how to apply the default ssl certificate to ingress-nginx on rke2?

I followed the doc at https://docs.rke2.io/networking/#nginx-ingress-controller
I imported the TLS certificate into a secret
I created this file on each node (6) in the cluster

# /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-config.yaml
---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      extraArgs:
        default-ssl-certificate: "kube-system/domain.net"

I selected redeploy via the rancher console on DaemonSet rke2-ingress-nginx-controller

I have an ingress defined and working but I keep getting insecure warning, certificate issued to 'Kubernetes Ingress Controller Fake Certificate'.

curl also reports the error: SSL certificate problem: unable to get local issuer certificate

rancher-2 rancher
Martin Hope
Oyabi
Asked: 2021-05-05 11:06:30 +0800 CST
  • 1

For a project, I must use big containers (500Mb to 60Gb).

I don't have a precise measurement but when I run containers larger than 3-5Gb with gitlab-runner, I get an error in rancher: CreateContainerError: context deadline exceeded

Our kubernetes cluster is build with rke with rancher as web ui and located in our datacenter.

The bug only appears with gitlab-runner, if I launch docker run ... on the kubernetes node, everything works fine.

Maybe there is a timeout somewhere?

Have any of you ever had this problem?

Thank you.

docker kubernetes gitlab helm rancher-2
Martin Hope
michafn
Asked: 2021-04-12 02:55:08 +0800 CST
  • 4

we are using an nginx (1.15.8.1) as ingress controller on our k8s cluster (v.1.17), managed by rancher (2.5.7).

This worked pretty fine so far, but now we set up a custom API pod that can be externally accessed via ingress.

Now, doing some load testing of the API by firing request every second at the API randomly returns "502 Bad Gateway" every couple of requests. However not periodically.

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>openresty/1.15.8.1</center>
</body>
</html>

The respective log on ingress controller:

2021/04/11 11:02:48 [error] 25430#25430: *55805583 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: xxx, server: xxx, request: "POST /api/v1/xxx HTTP/2.0", upstream: "http://xxx/api/v1/xxx", host: "xxx"

We ran the very same container on a non K8s environment as docker container, where we never experienced this issue before, so at the moment I would assume this is not a problem of the container/API implementation.

My thoughts were:

1- the service definition that abstracts the API pod cannot route the traffic to a pod and therefore replies with a 502

-> I changed the ingress to directly go to the workdload for testing to understand if this is the issue. I get the same errors then, so the service definition seems not to be the problem

2- timeout in ingress resource (b/c the post replies with latency) might cause the routing to fail

-> i set the timeouts to

    nginx.ingress.kubernetes.io/proxy-connect-timeout: 1d
    nginx.ingress.kubernetes.io/proxy-read-timeout: 1d
    nginx.ingress.kubernetes.io/proxy-send-timeout: 1d

but ran in the same error again. so this seem not to be the problem either

3- error in API container (most unlikely) which sends back errors

-> checking the container logs reveals that the missing request (which returned the 502 or anything similar which is then evaluated on ingress level as 502) was never received by the API container. Furthermore this doesn't make sense at all from a traffic routing perspective.

4- As a final step i scaled up the pod to two containers and everything works perfectly well.

Summary: with the results above i sum up that 1- service definition cannot be the problem (test 1) 2- ingress timeouts cannot be the problem 3- the api itself cannot be the problem

Thinking about test 4 it points in a direction that the container availability in the pods could be a problem in terms that the API drops requests (resulting in a 502) when the container is in a specific state (this would explain why it works with 2 containers scaling) OR the traffic routing times out somewhere on the way.

Currently i don't have any further ideas where to go from here. Any hint is appreciated.

kubernetes nginx-ingress rancher-2 bad-request
Martin Hope
user562397
Asked: 2020-03-03 10:16:05 +0800 CST
  • 2

I intend to deploy a k8s + Rancher cluster on my local network, but my environment has several VLANs, with pfsense acting as a firewal and router between such VLANs.

My cluster resides in XCP-NG as a hypervisor and I will inform the VLANs that it should pass on to the cluster nodes.

I intend to have some services in different VLANs, because I have VLAN for development, DMZ, production, management, etc., in that I would like to know if I have to take a different approach during the deployment of K8s + Rancher due to my environment?

To deploy a cluster that works with pods on multiple VLANs, must the cluster nodes have multiple NICs, each on a VLAN that I intend to use?

For example, if my cluster has 6 nodes, 3 master and 3 workers, must they be in the same VLAN or are they in different VLANs and having communication between them is enough?

If I want to deploy a pod on the development VLAN, and my cluster resides on the management VLAN, would that be possible?

Thanks in advance for your help.

networking kubernetes pfsense rancher-2