Home / server / Questions / 1045556
In Process
tommyduarte
Asked: 2020-12-10 04:44:24 +0800 CST

Jupyter notebook on kubernetes not being able to connect to outside docker service

  • 772

Im running a kubernetes (kubeflow + k8s) pod with a jupyter notebook and a docker service outside of the kubernetes server, im currently trying to connect to a sql service but it keeps getting ConnectionResetError, both firewall and docker are exposing the port needed but the k8s keeps not being able to connect, what could be the problem? (tell me if you need more details).

Thanks.

docker kubernetes istio

1 Answers

  1. As mentioned in comments

    From the istio perspective to make this work you would have to add ServiceEntry so istio injected pods could talk with external database.

    ServiceEntry enables adding additional entries into Istio’s internal service registry, so that auto-discovered services in the mesh can access/route to these manually specified services. A service entry describes the properties of a service (DNS name, VIPs, ports, protocols, endpoints). These services could be external to the mesh (e.g., web APIs) or mesh-internal services that are not part of the platform’s service registry (e.g., a set of VMs talking to services in Kubernetes). In addition, the endpoints of a service entry can also be dynamically selected by using the workloadSelector field. These endpoints can be VM workloads declared using the WorkloadEntry object or Kubernetes pods. The ability to select both pods and VMs under a single service allows for migration of services from VMs to Kubernetes without having to change the existing DNS names associated with the services.

    There is an example in istio documentation.

    Note that You may find MySQL can’t connect after installing Istio. This is because of PERMISSIVE mode, which does not work with MySQL. You may see error messages such as ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0.

    There have two options to solve the problem.

    1.Disable Mutual TLS.

    Choose this option if you don’t want Istio mutual TLS. You achieve this by disabling mutual TLS on the MySQL service explicitly.

    $ kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: mysql-nomtls-peerauthn
spec:
  selector:
    matchLabels:
      app: <YOUR-MYSQL-SERVICE>     # The label of *your* K8s Service
  mtls:
    mode: DISABLE
EOF

    2.Enable mutual TLS in STRICT mode.

    If you want mutual TLS protection for MySQL, enable mutual TLS using a destination rule and an authentication policy.

    $ kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: mysql-mtls-peerauthn
spec:
  selector:
    matchLabels:
      app: <YOUR-MYSQL-SERVICE>     # The label of *your* K8s Service
  mtls:
    mode: STRICT
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: mysql-mtls-dr
spec:
  host: YOUR-MYSQL-SERVICE     # The name of *your* K8s Service
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
EOF
    • 0