Spark question [a]:java.lang.classnotfoundexception:org.apache.spark.launcher.main

Source: Internet
Author: User
Spark Question [a]:java.lang.classnotfoundexception:org.apache.spark.launcher.main

In development, you need to use crontab to dispatch shell scripts to perform spark tasks. The manual execution of the script is not a problem, but it is executed in crontab and the following error is reported:

Exception in thread ' main ' Java.lang.noclassdeffounderror:org/apache/spark/launcher/main
caused by: Java.lang.ClassNotFoundException:org.apache.spark.launcher.Main at
        Java.net.urlclassloader$1.run ( urlclassloader.java:217) at
        java.security.AccessController.doPrivileged (Native method)
        at Java.net.URLClassLoader.findClass (urlclassloader.java:205) at
        Java.lang.ClassLoader.loadClass ( classloader.java:323) at
        sun.misc.launcher$appclassloader.loadclass (launcher.java:294)
        at Java.lang.ClassLoader.loadClass (classloader.java:268)

It was crontab that the current environment variable could not be found during the run, and there were 2 ways to resolve it:
1: In the crontab command, add the source command, or add the source command to the shell script:

# Add A * * source ~/.bash_profile;bash/home/test/test-etl/sbin/crontab_etl.sh to the crontab command line

# Add
#!/bin/sh source ~/.bash_profile at the front of the shell script

2: Through the which spark-submit solution, I was solved by the 2nd way:

Spark_home= ' which spark-submit '
$SPARK _home--class com.test.etl.job.scala.job.ETLJob

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.