Spark on yarn runs to create JAR package conflict

1.1 Problem Description

When the Spark streaming program resolves protobuf serialized data,--jars to add a dependent Protobuf-java-3.0.0.jar package, using the local mode program is normal, and using yarn mode will report errors that are not found for the method, as follows:

1.2 Workaround

Analysis of the local mode can run, yarn mode can not be run because the user submitted Protobuf-java-3.0.0.jar and spark_home/ The Protobuf-java-2.5.0.jar conflict in Lib, the default is to load the jar under Spark_home/lib instead of your program jar, so "nosuchmethoderror" will occur.

WORKAROUND: Submit the parameter add--conf spark.executor.extraclasspath= $user _jars

Note that the Protobuf-java-3.0.0.jar package is to be distributed to all nodes as shown

Summary of the relevant parameters for this type of problem:

Property name	Default Value	meaning
Spark.driver.extraClassPath	(none)	Additional classpath entities attached to the classpath of the driver. The main purpose of this setting is the backward compatibility issue between spark and older versions. Users generally do not have to set this option
Spark.executor.extraClassPath	(none)	Additional classpath entities attached to the classpath of the executors. The main purpose of this setting is the backward compatibility issue between spark and older versions. Users generally do not have to set this option
Spark.driver.userClassPathFirst	False	Experimental) when a class is loaded in driver, the user adds a jar with a higher priority than the Spark's own jar. This property can reduce the conflict between spark dependencies and user dependencies. It is now an experimental feature.
Spark.executor.userClassPathFirst	False	Experimental) Whether the user adds a jar with a higher priority than Spark's own jar when the class is loaded in executor. This property can reduce the conflict between spark dependencies and user dependencies. It is now an experimental feature.

Spark on yarn runs to create JAR package conflict