Home / server / Questions / 1062565
In Process
Cricket
Asked: 2021-05-05 11:23:20 +0800 CST

HTTPS 308 permanent redirection k8s ingress nginx

  • 772

I have created an EKS cluster following the examples from AWS EKS, I have deployed the nginx ingress controller on top from kubernetes/nginx, Created an ingress resource which points to back end k8s services, However, when I am trying to query via curl -kv https://dev01.cricket.com/demo/hello it returns Permanent Redirect 308

Here is my service-nlb.yaml

    --- (service-nlb.yaml)
kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: default
  annotations:
    **service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http**
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
    service.beta.kubernetes.io/aws-load-balancer-internal: "0.0.0.0/0"
    **service.beta.kubernetes.io/aws-load-balancer-ssl-cert:  "arn:aws:acm:us-west-2:xxxxx:certificate/x-037xxxd-4a1e-x-x"**
    service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-TLS-1-2-Ext-2018-06
    **service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https**
spec:
  ports:
    - name: https
      port: 443
      targetPort: 80

ingress.yaml

```apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  labels:
    app: jg
    env: dev
  name: jg-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
    **ingress.kubernetes.io/ssl-redirect: "false"**
spec:
  rules:
    - host: dev01.cricket.com
      http:
        paths:
          - backend:
              serviceName: jg-service
              servicePort: 8080
            path: /demo
    - host: dev02.cricket.com
      http:
        paths:
          - backend:
              serviceName: sad-service
              servicePort: 8080
            path: /sad
  tls:
   - hosts:
        - dev01.cricket.com
     secretName: my-tls-secret```

when I am doing curl -kv dev01.cricket.com it returns 308 permanent redirect

curl -kv https://dev01.cricket.com/demo/hello
*   Trying 10.41.168.92...
* TCP_NODELAY set
* Connected to dev01.cricket.com (10.41.168.92) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=dev01.cricket.com; OU=management:idms.group.140787; O=Cricket Inc.; ST=California; C=US
*  start date: Apr 19 08:37:06 2021 GMT
*  expire date: May 19 08:37:05 2023 GMT
*  issuer: CN=Cricket Corporate Server CA 1; OU=Certification Authority; O=Cricket Inc.; C=US
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7ff9eb80d200)
> GET /demo/hello HTTP/2
> Host: dev01.cricket.com
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 308
< server: Cricket
< date: Tue, 04 May 2021 16:38:38 GMT
< content-type: text/html
< content-length: 171
< location: https://dev01.cricket.com/demo/hello
< strict-transport-security: max-age=31536000; includeSubdomains
< x-frame-options: SAMEORIGIN
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
<
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.17.8</center>
</body>
</html>
* Connection #0 to host dev01.cricket.com left intact
* Closing connection 0

I am not sure why the TLS termination is not happening and the permanent redirect error is happening if I do the same curl on http it returns 301

curl -kv http://dev01.cricket.com/demo/hello
*   Trying 10.41.162.164...
* TCP_NODELAY set
* Connected to dev01.cricket.com (10.41.162.164) port 80 (#0)
> GET /demo/hello HTTP/1.1
> Host: dev01.cricket.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Server: Cricket
< Date: Tue, 04 May 2021 19:17:26 GMT
< Content-Type: text/html
< Content-Length: 162
< Connection: keep-alive
< Location: https://dev01.cricket.com/demo/hello
<
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>Cricket</center>
</body>
</html>
* Connection #0 to host dev01.cricket.com left intact
* Closing connection 0

backend-service.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jgdeployment
  labels:
    app: jg
spec:
  replicas: 3
  selector:
    matchLabels:
      app: jg
  template:
    metadata:
      labels:
        app: jg
    spec:
      containers:
      - name: jg
        image: docker.cricket.com/jga:1.0.0
        ports:
        - containerPort: 8080
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /demo/hello
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /demo/hello
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: jg-service
  namespace: "default"
spec:
  type: ClusterIP
  ports:
    - port: 8080
      targetPort: 8080
  selector:
    app: jg

Not sure what I am doing missing to get 308 error on https.

kubernetes nginx-ingress nlb

3 Answers

  1. This is the answer from user Ariseaz from the github issue mentioned in the answer from Azeez Adeniji.

    Using helm ingress-nginx chart on EKS
    Edit configmap ingress-nginx-controller
    kubectl edit configmap ingress-nginx-controller -n ingress-nginx
    Add
    data: server-snippet: | listen 8000; if ( $server_port = 80 ) { return 308 https://$host$request_uri; } ssl-redirect: "false"

    Edit service/ingress-nginx-controller by adding
    meta.helm.sh/release-namespace: ingress-nginx service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60" service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <acm arn> service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https service.beta.kubernetes.io/aws-load-balancer-type: nlb

    Setup your port in the ingress controller to look like what I have below:
    NB: special port is what you are going to add to the ingress containerPort
    ports:

    • name: http
      port: 80
      protocol: TCP
      targetPort: 80
    • name: https
      port: 443
      protocol: TCP
      targetPort: special

    Now Edit ingress controller deployment containerPort
    kubectl edit deployment.apps/ingress-nginx-controller -n ingress-nginx
    Add:

    • containerPort: 8000
      name: special
      protocol: TCP
    • 1

  2. I am not sure why the TLS termination is not happening and the permanent redirect error is happening

    Given your Service spec:

    spec:
  ports:
    - name: https
      port: 443
      targetPort: 80

    causes the following to take place:

    User -- [Internet] -HTTPS-> :443 [NLB] -HTTPS-> :443 [iptables] -HTTP-> :80 [nginx]

    Thus, because nginx does not realize you have TLS in front of it, and inside the cluster that traffic is HTTP, the nginx believes it is being helpful by redirecting the request to its 443 port and thus secures the request

    I haven't done TLS termination with an NLB but in theory the idea is the same: you'd want to use TLS from the NLB to the (presumably self-signed) certificate on nginx's 443 port, because -- all things being equal -- the AWS LB will not verify the validity of the nginx certificate. Thus, while you do actually have a MITM, there will not be any cleartext protocol communication from the user's browser to nginx, making it happy and not 308 anymore

    I do see the annotation you highlighted on your Ingress, but I suspect strongly that the mechanism you're experiencing is happening before it gets to the specific Ingress resource, perhaps due to a ConfigMap setting or other. It may be possible to tune nginx as you thought you were asking, but I believe the "pseudo-end-to-end TLS" is so simple to actually fix that it wouldn't behoove you to find work-arounds

    • 0

  3. Read my comment on this thread: https://github.com/kubernetes/ingress-nginx/issues/2724

    Comment by Ariseaz

    • 0