Questions[cert-manager](server)

I have a self-hosted Kubernetes cluster with an Nginx Ingress. Cert-manager is also running on the cluster, with which I try to get valid SSL certificates using Letsencrypt. It all works and I get a valid certificate for example.com, www.example.com or app1.example.com, but not for a general wildcard *.example.com. If I try in any way to enter a wildcard in my ingress under sec.tls.hosts, no certificate is generated for me. I get the output for

kubectl get certificate

NAME              READY   SECRET            AGE
tls-test-cert     False   tls-electi-cert   20h

kubectl get CertificateRequest

NAME                    APPROVED   DENIED   READY   ISSUER                REQUESTOR                                         AGE
tls-test-cert-8jw75     True                False   letsencrypt-staging   system:serviceaccount:cert-manager:cert-manager   18m

kubectl describe CertificateRequest

[...]
Status:
  Conditions:
    Last Transition Time:  2022-02-27T13:54:38Z
    Message:               Certificate request has been approved by cert-manager.io
    Reason:                cert-manager.io
    Status:                True
    Type:                  Approved
    Last Transition Time:  2022-02-27T13:54:38Z
    Message:               Waiting on certificate issuance from order gateway/tls-test-cert-8jw75-1425588341: "pending"
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:
  Type    Reason           Age   From          Message
  ----    ------           ----  ----          -------
  Normal  cert-manager.io  18m   cert-manager  Certificate request has been approved by cert-manager.io
  Normal  OrderCreated     18m   cert-manager  Created Order resource gateway/tls-test-cert-8jw75-1425588341
  Normal  OrderPending     18m   cert-manager  Waiting on certificate issuance from order gateway/tls-test-cert-8jw75-1425588341: ""

My Nginx Ingress: (I swapped my domain to example.com for this post)

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-management
  namespace: gateway
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: "letsencrypt-staging"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  ingressClassName: nginx
  tls:
  - secretName: tls-test-cert
    hosts:
      - example.com
      - '*.example.com'
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: test-gateway
                port:
                  number: 80
    - host: '*.example.com'
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: test-gateway
                port:
                  number: 80

Issuer: (I've redacted my email here)

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-staging
  namespace: cert-manager
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: *******
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
      - http01:
          ingress:
            class: nginx

My reverse proxy (test-gateway) definitely works and forwards all subdomains to my website. Thanks in advance for any ideas as to what might be causing this.

I'm having issues with my rapberry pi kubernetes implementation

Problem:

I have cert-manager letsencrypt ACME challenge waiting due to a 401 error code on bare metal kubernetes install.

Setup

Platform: Raspberry Pi 4

OS: Ubuntu Server 20.04.3 LTS 64 bit

Ingress: Nginx

Loadbalancer: Metallb

Networking: Calico

I installed metallb and nginx via helm using:

helm install metallb metallb/metallb --namespace kube-system\
    --set configInline.address-pools[0].name=default\
    --set configInline.address-pools[0].protocol=layer2\
    --set configInline.address-pools[0].addresses[0]=<ip-range>

and

helm install ingress-nginx ingress-nginx/ingress-nginx --namespace kube-system

My letsencrypt looks like this:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    email: <email redacted>
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx

My nginx ingress set up looks like this:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: "nextcloud" # Same namespace as the deployment
  name: "nextcloud-ingress" # Name of the ingress (see kubectl get ingress -A)
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    cert-manager.io/cluster-issuer: "letsencrypt-prod" # Encrypt using the ClusterIssuer deployed while setting up Cert-Manager
    nginx.ingress.kubernetes.io/proxy-body-size:  "125m" # Increase the size of the maximum allowed size of the client request body
spec:
  tls:
  - hosts:
    - "nextcloud.<domain redacted>" # Host to access nextcloud
    secretName: "nextcloud-prod-tls" # Name of the certificate (see kubectl get certificate -A)
  rules:
  - host: "nextcloud.<domain redacted>" # Host to access nextcloud
    http:
      paths:
        - path: /  # We will access NextCloud via the URL https://nextcloud.<domain.com>/
          pathType: Prefix
          backend:
            service: 
              name: "nextcloud-server" # Mapping to the service (see kubectl get services -n nextcloud)
              port: 
                number: 80 # Mapping to the port (see kubectl get services -n nextcloud)
---

Debugging

When I look at the ingress controller logs (different namespace) I see:

Service "nextcloud/cm-acme-http-solver-9tccf" does not have any active Endpoint.

But the endpoint appears to exist when I do kubectl get endpoints -A

My certificate exists as:

kubectl get certificate -n nextcloud
NAME                 READY   SECRET               AGE
nextcloud-prod-tls   False   nextcloud-prod-tls   3h58m

Following the recommended debug steps from cert manager I tracked the issue to the challenges whereby I get:

Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: wrong status code '401', expected '200'
  State:       pending
Events:        <none>

I'm kinda stuck I've been googling my heart out but there doesn't seem to be a lot on this. I'm guessing I've stuffed up on the set up but I've mainly been following the documentation on the relevant pages. Any pointers would be greatly appreciated :). If you need any additional info let me know this is currently quite long so I tried to include what I thought were problem points.

Recently, we changed uninstalled our cert manager and reinstalled it into a new namespace.

There were no issues with the uninstall / reinstall of the cert manager. When we tried to create a new certificate issuer, the cert manager is unable to handle the request as it has a reference to the previous web hook service.

kubectl apply -f c:\temp\clusterissuereus.yaml -n cert-manager

Error from server (InternalError): error when creating "c:\\temp\\clusterissuereus.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-1617928457-webhook.cert-manager.svc:443/mutate?timeout=10s: service "cert-manager-1617928457-webhook" not found

cert-manager-1617928457-webhook is the previous webhook pod / service.

This is similar issue as described on the Cert Manager Github site.

We did try to install cert manager via helm with the --set installCRDs=true param but it did not work.

Any suggestions on removing the previous webhook reference ?

I have the following ingress manifest file:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: fsm
  name: fsm
  labels:
    app: fsm
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    cert-manager.io/issuer: "letsencrypt-staging"
spec:
    tls:
    - hosts: 
      - k8s-cluster.int
      secretName: quickstart-example-tls
    rules:    
    - host: k8s-cluster.int
      http:
        paths:
          - path: /fsm(/|$)(.*)
            backend:
              serviceName: fsm
              servicePort: 8081

I am working with VMware with Vsphere. I don't have a domain like www.google.com, just a DNS name which is k8s-cluster and the domain .int (inside my company). When I am trying to generate the certificate I receive this error: "msg"="error waiting for authorization" "error"="acme: authorization error for k8s-cluster.int: 400 urn:ietf:params:acme:error:dns: DNS problem: NXDOMAIN looking up A for k8s-cluster.int - check that a DNS record exists for this domain" "dnsName"="k8s-cluster.int" "resource_kind"="Challenge" "resource_name"="quickstart-example-tls-w7vj9-4141989927-3312743172" "resource_namespace"="fsm" "resource_version"="v1" "type"="HTTP-01"

Can this problem appear because k8s-cluster.int is inside a intranet? If I curl k8s-cluster.int

<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.19.1</center>
</body>
</html>

So, I think that the DNS works.

I'm trying to uninstall and reinstall cert-manager on our Kubernetes clusters. Their uninstall docs mention:

Before continuing, ensure that all cert-manager resources that have been created by users have been deleted. You can check for any existing resources with the following command:

$ kubectl get Issuers,ClusterIssuers,Certificates,CertificateRequests,Orders,Challenges --all-namespaces

The command outputs hundreds of resources, spread across two dozen namespaces.

How can I efficiently delete them all, without deleting anything else in the namespaces?

The kubectl delete command requires specifying a namespace when deleting all of a certain type, like this: kubectl delete certificates -n example-ns, so that's no good here.

Deleting each one by name in a loop while specifying -A also doesn't work, since I need to specify the namespace:

$ kubectl delete -A order.certmanager.k8s.io/fcfa95477bc0149dbc16c99c54faa82e-cert-1862418815
error: a resource cannot be retrieved by name across all namespaces

What's the correct cli magic here?