What are the requirements to recognize a Spark app?


We recognize spark based on command line arguments. It needs to contain one of the following:

 

org.apache.spark.deploy.worker.DriverWrapper

when spark app is deployed to spark standalone server.

 

 

org.apache.spark.deploy.yarn.ApplicationMaster

when deployed to hadoop yarn.

 

 

org.apache.spark.deploy.SparkSubmit

when deployed in client mode.

 

In order to confirm that the app is really a spark app, you can check it from the command line arguments ( spark history server ) :


org.apache.spark.deploy.history.HistoryServer

Have more questions? Submit a request

Comments