Spark Question [a]:java.lang.classnotfoundexception:org.apache.spark.launcher.main
In development, you need to use crontab to dispatch shell scripts to perform spark tasks. The manual execution of the script is not a problem, but it is executed in crontab and the following error is reported:
Exception in thread ' main ' Java.lang.noclassdeffounderror:org/apache/spark/launcher/main
caused by: Java.lang.ClassNotFoundException:org.apache.spark.launcher.Main at
Java.net.urlclassloader$1.run ( urlclassloader.java:217) at
java.security.AccessController.doPrivileged (Native method)
at Java.net.URLClassLoader.findClass (urlclassloader.java:205) at
Java.lang.ClassLoader.loadClass ( classloader.java:323) at
sun.misc.launcher$appclassloader.loadclass (launcher.java:294)
at Java.lang.ClassLoader.loadClass (classloader.java:268)
It was crontab that the current environment variable could not be found during the run, and there were 2 ways to resolve it:
1: In the crontab command, add the source command, or add the source command to the shell script:
# Add A * * source ~/.bash_profile;bash/home/test/test-etl/sbin/crontab_etl.sh to the crontab command line
# Add
#!/bin/sh source ~/.bash_profile at the front of the shell script
2: Through the which spark-submit solution, I was solved by the 2nd way:
Spark_home= ' which spark-submit '
$SPARK _home--class com.test.etl.job.scala.job.ETLJob