Questions[apache-spark](server)

I set up a Spark 2.3.1 cluster on kubernetes, however, I have trouble sending a sample SparkPi job to it:

The submit script I'm using:

bin/spark-submit \
    --master k8s://https://10.0.15.7:7077 \
    --deploy-mode cluster \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.kubernetes.container.image=gcr.io/my-project/spark:spark-test \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.executor.instances=3 \
    local:///opt/spark/examples/jars/spark-examples_2.11-2.3.1.jar 1000

but get

Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failed to start websocket
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onFailure(WatchConnectionManager.java:194)
...
Caused by: java.net.SocketTimeoutException: timeout
    at okio.Okio$4.newTimeoutException(Okio.java:230)
    ... 
Caused by: java.net.SocketException: Socket closed
    at java.net.SocketInputStream.read(SocketInputStream.java:204)
...

My guess is that it cannot access k8s apiserver. But I don't understand why. I use --conf serviceAccountName and RBAC is fully configured as in the above url.

All spark pods use the spark serviceAccountName, defined as

apiVersion: v1
kind: ServiceAccount
metadata:
  name: spark
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: spark-role
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: edit
subjects:
- kind: ServiceAccount
  name: spark
  namespace: default

I've been trying to get to the bottom of a memory issues for some time now and I simply can't fathom out what the problem is. Any help is greatly appreciated.

The error is:

[![OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000005662c0000, 10632822784, 0) failed; error='Cannot allocate memory' (errno=12)
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 10632822784 bytes for committing reserved memory.]

I have a very small spark job that I'm running on a cluster. Of the various permutations that I've run these are my findings: (new clusters in each case, all identical in configuration)

CLI ONLY - One cluster I start and run all of the steps via the cli, each step results in a slight increase in memory that persists, Ganglia shows the cluster cache memory increasing with every step, it drops again after completion but not to the base level. Eventually resulting in there not being enough memory to allocate for a new JVM to run any additional steps. using htop on this cluster shows spark history servers the main memory intensive process - could the history server be retaining too much information?

CONSOLE ONLY - This cluster was created in very much the same way as the others, the difference is that I add the steps via the console. (I came to try this as I was simply out of ideas) This cluster has only run one step thus far, htop showing oozie as being the highest memory consuming process.

Others - All other clusters ran and failed in the same manner, an interesting case is where a new cluster was started, one step ran, it completed but gradually consumed memory to the point where the exception occurred again. For all of these other clusters, hadoop was always the task at the top of the process tree for memory consumption.

Any help or suggestions with respect to how to solve would be fantastic, thank you in advance.

I've attached a few images that may help explain the above.

I have written a program example.jar which uses a spark context. How can I run this on a cluster which uses Slurm? This is related to https://stackoverflow.com/questions/29308202/running-spark-on-top-of-slurm but the answers are not very detailed and not on serverfault.

I'm looking for a solution to improve my Spark cluster performances, I have read from http://spark.apache.org/docs/latest/hardware-provisioning.html:

We recommend having 4-8 disks per node

, I have tried both with one and two disks but I have seen that with 2 disks the execution time is doubled. Any explanations about this?

This is my configuration: 1 machine with 140 GB RAM 2 disks and 32 CPU (I know that is an unusual configuration) and on this I have a standalone Spark cluster with 1 Worker.

The nodes in my Spark cluster have two network interfaces each, one public and one private. Using the SPARK_MASTER_IP environment variable, I can configure Spark to listen on port 7077 on one or the other ip address.

For example:

netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 10.x.x.x:7077           :::*                    LISTEN      43403/java

Is there any way to configure Spark to listen on port 7077 on both the public and private interface at the same time?