I've got a small k3s cluster in my home hosting a few websites and local applications. For the most part, I've been able to wrangle it to host a variety of services, but the LetsEncrypt functionality has never worked very well for me. When it has worked, I've been unsure as to why, and afraid to tinker with it lest it break again.
At the moment, there are two sites on the cluster with TLS support working fine... for years even, but now that I'm adding a third site, I'm running into the same errors I was getting in the past and I have no idea why. I'm hoping someone here can explain what I've missed.
The error appears to be a complaint that when the magic URL is requested (how do I determine the URL being attempted?), they get HTML rather than the expected response, though it's not clear why it's doing that, or even what service is constructing the response.
I0820 17:28:23.398812 1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="REDACTED.tld" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-9qp5z" "related_resource_namespace"="my-namespace" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="tls-REDACTED" "resource_namespace"="my-namespace" "resource_version"="v1" "type"="HTTP-01"
E0820 17:28:23.585197 1 sync.go:190] cert-manager/challenges "msg"="propagation check failed" "error"="did not get expected response when querying endpoint, expected \"REDACTED.REDACTED\" but got: <!DOCTYPE html PUBLIC \"-... (truncated)" "dnsName"="REDACTED.tld" "resource_kind"="Challenge" "resource_name"="tls-REDACTED" "resource_namespace"="my-namespace" "resource_version"="v1" "type"="HTTP-01"
I0820 17:28:33.585741 1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="REDACTED.tld" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-5gnhm" "related_resource_namespace"="my-namespace" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="tls-REDACTED" "resource_namespace"="my-namespace" "resource_version"="v1" "type"="HTTP-01"
I0820 17:28:33.585898 1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="REDACTED.tld" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-9qp5z" "related_resource_namespace"="my-namespace" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="tls-REDACTED" "resource_namespace"="my-namespace" "resource_version"="v1" "type"="HTTP-01"
E0820 17:28:33.600402 1 sync.go:190] cert-manager/challenges "msg"="propagation check failed" "error"="did not get expected response when querying endpoint, expected \"REDACTED.REDACTED\" but got: <!DOCTYPE html PUBLIC \"-... (truncated)" "dnsName"="REDACTED.tld" "resource_kind"="Challenge" "resource_name"="tls-REDACTED" "resource_namespace"="my-namespace" "resource_version"="v1" "type"="HTTP-01"
The entire application consists of a Python web app, a Python worker, Redis, and Nginx, which adds up to a lot of YAML. I'm including the markup I assume is relevant for this case:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace: my-namespace
spec:
replicas: 1
revisionHistoryLimit: 0
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:alpine
imagePullPolicy: IfNotPresent
resources:
requests:
memory: 32Mi
cpu: 125m
limits:
memory: 32Mi
cpu: 125m
ports:
- containerPort: 80
volumeMounts:
- name: public
mountPath: "/usr/share/nginx/html"
volumes:
- name: public
persistentVolumeClaim:
claimName: public
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: web
namespace: my-namespace
annotations:
kubernetes.io/ingress.class: traefik
cert-manager.io/cluster-issuer: letsencrypt
acme.cert-manager.io/http01-edit-in-place: 'true'
spec:
rules:
- host: REDACTED.tld
http:
paths:
- path: /static
pathType: Prefix
backend:
service:
name: nginx
port:
number: 8000
- path: /media
pathType: Prefix
backend:
service:
name: nginx
port:
number: 8000
- path: /
pathType: Prefix
backend:
service:
name: web
port:
number: 8000
tls:
- secretName: tls
hosts:
- REDACTED.tld
---
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: my-namespace
spec:
selector:
app: nginx
ports:
- port: 8000
targetPort: 80
name: tcp
---
apiVersion: v1
kind: Service
metadata:
name: web
namespace: my-namespace
spec:
selector:
app: web
ports:
- port: 8000
targetPort: 8000
name: gunicorn
I'm quite comfortable working with "bare metal" Linux administration, but Kubernetes still feels like too much "magic" to me. The above was built through following the K3s Rocks tutorials and as mentioned above, somehow works for two other domains — though they didn't at first and then somehow started working.
I'd appreciate any suggestions for this problem, as well as any recommendations for how I could/should do the above better.
