Dzmitry Lazerka's questions

I'm trying to set up GKE with Standalone NEG (avoiding Ingress, and using Terraform for Load Balancer setup instead). Everything works fine, but so far I've been using Firewall Rules from another Ingress.

But to create a proper Firewall Rule, I need GKE_NODE_NETWORK_TAGS. But I cannot set them when creating an Autopilot cluster. I cannot list nodes as listed in the docs as well, as Autopilot nodes are not visible for gcloud compute instances describe.

How to create the Firewall Rule properly for an Autopilot cluster?

PS: Docs on Standalone NEGs: https://cloud.google.com/kubernetes-engine/docs/how-to/standalone-neg#attaching_a_load_balancer_to_your_standalone_negs

I tried to use Google Managed Certificate (not through k8s) in Ingress.

If Ingress is in default namespace, everything works fine using ingress.gcp.kubernetes.io/pre-shared-cert: my-cert-name annotation.

However, if Ingress is in a namespace, it looks for a certificate named my-namespace/my-cert-name. But it's impossible to create a certificate with / in its name.

Using GKE k8s ManagedCertificate everything works fine. How to make it work with a non-k8s ManagedCertificate?

UPDATE: we use Terraform to manage SSL certificates, using google_compute_managed_ssl_certificate resource. We used GKE with Ingress, and tried to use that certificate with it. If Ingress is in default namespace -- everything works fine. If Ingress is in some other namespace -- it's impossible to use that certificate, because Ingress looks for certificate named namespacename/certname instead of certname.

I set up a Spark 2.3.1 cluster on kubernetes, however, I have trouble sending a sample SparkPi job to it:

The submit script I'm using:

bin/spark-submit \
    --master k8s://https://10.0.15.7:7077 \
    --deploy-mode cluster \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.kubernetes.container.image=gcr.io/my-project/spark:spark-test \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.executor.instances=3 \
    local:///opt/spark/examples/jars/spark-examples_2.11-2.3.1.jar 1000

but get

Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failed to start websocket
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onFailure(WatchConnectionManager.java:194)
...
Caused by: java.net.SocketTimeoutException: timeout
    at okio.Okio$4.newTimeoutException(Okio.java:230)
    ... 
Caused by: java.net.SocketException: Socket closed
    at java.net.SocketInputStream.read(SocketInputStream.java:204)
...

My guess is that it cannot access k8s apiserver. But I don't understand why. I use --conf serviceAccountName and RBAC is fully configured as in the above url.

All spark pods use the spark serviceAccountName, defined as

apiVersion: v1
kind: ServiceAccount
metadata:
  name: spark
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: spark-role
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: edit
subjects:
- kind: ServiceAccount
  name: spark
  namespace: default

I set up a Spark 2.3.1 cluster, however, I have trouble sending a sample SparkPi job to it:

Running Spark using the REST application submission protocol.
2018-09-06 13:45:53 INFO  RestSubmissionClient:54 - Submitting a request to launch an application in spark://10.0.15.7:7077.
2018-09-06 13:46:04 WARN  RestSubmissionClient:66 - Unable to connect to server spark://10.0.15.7:7077.
Warning: Master endpoint spark://10.0.15.7:7077 was not a REST server. Falling back to legacy submission gateway instead.
2018-09-06 13:46:04 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-09-06 13:46:10 ERROR ClientEndpoint:70 - Exception from cluster was: java.nio.file.NoSuchFileException: /opt/spark/examples/jars/spark-examples_2.11-2.3.1.jar
java.nio.file.NoSuchFileException: /opt/spark/examples/jars/spark-examples_2.11-2.3.1.jar
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:526)
    at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253)
    at java.nio.file.Files.copy(Files.java:1274)
    at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$copyRecursive(Utils.scala:632)
    at org.apache.spark.util.Utils$.copyFile(Utils.scala:603)
    at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:688)
    at org.apache.spark.util.Utils$.fetchFile(Utils.scala:485)
    at org.apache.spark.deploy.worker.DriverRunner.downloadUserJar(DriverRunner.scala:155)
    at org.apache.spark.deploy.worker.DriverRunner.prepareAndRunDriver(DriverRunner.scala:173)
    at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:92)

The submit script I'm using:

bin/spark-submit \
    --master spark://10.0.15.7:7077 \
    --deploy-mode cluster \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.kubernetes.container.image=gcr.io/my-project/spark:spark-test \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.executor.instances=3 \
    /opt/spark/examples/jars/spark-examples_2.11-2.3.1.jar 1000

I built the gcr.io/my-project/spark:spark-test image using default

bin/docker-image-tool.sh -r gcr.io/my-project -t spark-test build push

so /opt/spark/examples/jars/spark-examples_2.11-2.3.1.jar is in the container. Confirmed it:

docker run --rm -it --entrypoint "/bin/ls" gcr.io/my-project/spark:spark-test -l /opt/spark/examples/jars/

My Spark cluster runs on Kubernetes. I'm using spark:// scheme, not new k8s://https:// scheme, so it should work just as regular Spark cluster. IP and ports are visible, including Spark Web UI.

I don't understand where it tries to look for the jar file.

I tried to prepend path with local://, as in spark examples: https://spark.apache.org/docs/2.3.1/running-on-kubernetes.html but it doesn't work with spark:// master url, it throws No FileSystem for scheme: local exception.

RBAC is configured according to the URL above, all pods use spark service account.

I'm out of ideas.

I cannot create Dataproc Spark cluster for some reason. I run:

gcloud dataproc \
    --region us-west1 clusters create my-test1 \
    --project some_project \
    --scopes 'https://www.googleapis.com/auth/cloud-platform' \
    --bucket my-bucket \
    --zone us-west1-b \
    --single-node \
    --master-machine-type n1-highmem-16 \
    --master-boot-disk-size 500 \
    --image-version 1.3 \
    --initialization-actions \
'gs://dataproc-initialization-actions/connectors/connectors.sh',\
'gs://dataproc-initialization-actions/zeppelin/zeppelin.sh'\
    --metadata 'gcs-connector-version=1.9.4'

but get

+ VM_CONNECTORS_DIR=/usr/lib/hadoop/lib
+ declare -A MIN_CONNECTOR_VERSIONS
+ MIN_CONNECTOR_VERSIONS=(["bigquery"]="0.11.0" ["gcs"]="1.7.0")
++ /usr/share/google/get_metadata_value attributes/bigquery-connector-version
++ true
+ BIGQUERY_CONNECTOR_VERSION=
++ /usr/share/google/get_metadata_value attributes/gcs-connector-version
+ GCS_CONNECTOR_VERSION=1.9.4
+ [[ -z '' ]]
+ [[ -z 1.9.4 ]]
+ [[ -z '' ]]
+ [[ 1.9.4 = \1\.\7\.\0 ]]
+ [[ '' = \0\.\1\1\.\0 ]]
+ update_connector bigquery ''
+ local name=bigquery
+ local version=
+ [[ -n '' ]]
+ update_connector gcs 1.9.4
+ local name=gcs
+ local version=1.9.4
+ [[ -n 1.9.4 ]]
+ validate_version gcs 1.9.4
+ local name=gcs
+ local version=1.9.4
+ local min_valid_version=1.7.0
++ min_version 1.7.0 1.9.4
++ echo -e '1.7.0\n1.9.4'
++ sort -r -t. -n -k1,1 -k2,2 -k3,3
++ tail -n1
+ [[ 1.7.0 != \1\.\7\.\0 ]]
+ rm -f /usr/lib/hadoop/lib/gcs-connector-1.9.0-hadoop2.jar
++ gsutil ls 'gs://hadoop-lib/gcs/gcs-connector-*1.9.4*.jar'
++ grep hadoop2
AccessDeniedException: 403 7**********[email protected] does not have storage.objects.list access to hadoop-lib.
+ local path=
++ echo ''
++ wc -w
+ local path_count=0
+ [[ 0 != 1 ]]
+ echo -e 'ERROR: Only one gcs connector path should be listed for 1.9.4 version, but listed 0 paths:\n'
ERROR: Only one gcs connector path should be listed for 1.9.4 version, but listed 0 paths:

I checked that the service user has "Storage Object Viewer" role assigned, along with "Editor".

Do you have any idea how to create a Spark Cluster with GCS connector on Google Cloud Dataproc?