Application notes for Spark Web UI

Each Sparkcontext runtime runs a Web UI with a default port of 4040.

Interpretation: Each Spark application opens its own Web UI with a default port number of 4040. Access address: http://:4040. If more than one Spark application is running on the same node, the ports of these apps will be incremented, eg:4040,4041,4042.

Spark application is only valid during application run time, application run is complete, you cannot view the information by application the Web UI. Persistent application Web UI-related information requires the Spark log persistence feature to be turned on. Set spark.eventLog.enabled to True.

Interpretation: By default, if Spark application is executed, we will not be able to look through the Web UI for information about spark application execution, which is undoubtedly not good for troubleshooting and program tuning, so the Spark Events log feature Need to be turned on.

Spark Log service, which can be used to view spark application logs for various cluster modes.

Interpretation: not only to turn on the Spark Events log feature, but also to start the Spark log service. After startup, the log service Web UI can be accessed through a URL.