We are launching all spark jobs as kubernetics(k8es) containers inside a k8es cluster. We also create a service on each job and do port forwarding for the spark UI (container's 4040 is mapped to SvcPort, say, 31123). The same set of nodes is also hosting a Yarn cluster. Container's entry command is a call to spark-submit to Yarn in client mode.
Now, the spark container may get launched on any of the name-nodes - master or standby. There is a VIP assigned to the active name-node.
When spark container gets launched on active name-node, spark UI and all its tabs are easily accessible from anywhere using VIP:SvcPort However, when spark container gets launched on standby name node, spark UI is NOT accessible and eventually request fails with 500 error (as the VIP:SvcPort gets redirected to ActiveNamenode:8080 which is not accessible as driver is now running on standby name node.
I have tried to reason and try out multiple service configurations but nothing seems to work this scenario. This config is what I have tried using which UI is accessible when container is on master node but UI is NOT accessible when container is on standby node.
apiVersion: v1 kind: Service metadata: name: spark-svc labels: deployer: sample_app app: spark-app spec: selector: app: spark-app ports: - port: 31123 protocol: TCP targetPort: 4040 name: ui externalIPs: - 192.168.122.22
May anyone pls suggest proper k8es service config for this scenario. We do not want to restrict the spark containers to master name-node as let k8es address the job load within k8es cluster.
Any help is much appreciated.apache-sparkkubernetes
there is a spark ui proxy, that is connected to the spark-master and proxies requests for other nodes. See more details here: https://github.com/kubernetes/examples/tree/master/staging/spark spark-ui-proxy-controller.yaml and spark-ui-proxy-service.yaml
Is this helpful?