how to pass access token to connect to kubernetes cluster from spark-submit

Mike Source

I am trying to run spark job natively in the kubernetes cluster with the following command:

bin/spark-submit \ --deploy-mode cluster \ --class org.apache.spark.examples.SparkPi \ --master k8s://https://XXXX \ --kubernetes-namespace sidartha-spark-cluster \ --conf spark.executor.instances=2 \ --conf \ --conf spark.kubernetes.driver.docker.image=kubespark/spark-driver:v2.1.0-kubernetes-0.1.0-rc1 \ --conf spark.kubernetes.executor.docker.image=kubespark/spark-executor:v2.1.0-kubernetes-0.1.0-rc1 \ examples/jars/spark-examples_2.11-2.1.0-k8s-0.1.0-SNAPSHOT.jar 1000

Our kubernets cluster is secured with the access token. I could not find any docs or example where we can include the access token in spark-submit parameters.



comments powered by Disqus