Spark History server Cluster configuration and use (troubleshoot problems that are not displayed after performing spark tasks) __spark

In the conf file of your spark path, the CP copy Spark-defaults.conf.template is spark-defaults.conf

and add the following file

spark.eventLog.enabled true
Spark.eventLog.dir hdfs://master:9000/history
Spark.eventLog.compress true

Distribute configuration to other child nodes I'm using rsync.

rsync sparkconf Path/spark-defaults.conf child node ip:sparkconf path/spark-defaults.conf

Start Spark History Server, execute on the master node

Spark Path/sbin/start-history-server.sh Hdfs://master:9000/history

。。。 The path that follows must be the same as set in Conf, if you do not take this parameter, the default value is local/tmp/spark-events