Accessing UI for spark running as kubernetics container on standby name-node

Mohit Gupta Source

We are launching all spark jobs as kubernetics(k8es) containers inside a k8es cluster. We also create a service on each job and do port forwarding for the spark UI (container's 4040 is mapped to SvcPort, say, 31123). The same set of nodes is also hosting a Yarn cluster. Container's entry command is a call to spark-submit to Yarn in client mode.

Now, the spark container may get launched on any of the name-nodes - master or standby. There is a VIP assigned to the active name-node.

When spark container gets launched on active name-node, spark UI and all its tabs are easily accessible from anywhere using VIP:SvcPort However, when spark container gets launched on standby name node, spark UI is NOT accessible and eventually request fails with 500 error (as the VIP:SvcPort gets redirected to ActiveNamenode:8080 which is not accessible as driver is now running on standby name node.

I have tried to reason and try out multiple service configurations but nothing seems to work this scenario. This config is what I have tried using which UI is accessible when container is on master node but UI is NOT accessible when container is on standby node.

apiVersion: v1
kind: Service
   name: spark-svc
     deployer: sample_app
     app: spark-app
    app: spark-app
  - port: 31123
    protocol: TCP
    targetPort: 4040
    name: ui

May anyone pls suggest proper k8es service config for this scenario. We do not want to restrict the spark containers to master name-node as let k8es address the job load within k8es cluster.

Any help is much appreciated.



answered 11 months ago user3232739 #1

there is a spark ui proxy, that is connected to the spark-master and proxies requests for other nodes. See more details here: spark-ui-proxy-controller.yaml and spark-ui-proxy-service.yaml

Is this helpful?

comments powered by Disqus