Submit a job with Hadoop Eclipse plugin and add multiple third-party jars

From: http://heipark.iteye.com/blog/1171923

Through "Conf.set" ("Tmpjars", jars); "You can set up a third-party jar that has been just adding a jar, running OK, and I'm going to add multiple jars today and find that MapReduce is not found at runtime. Class (ClassNotFoundException), the trace code found that the jar file did upload to HDFs, so there was no solution, and then the jar was uploaded to HDFs, Then using the Distributedcache.addfiletoclasspath () method does not work either. Depressed half-day, later saw Job.xml in a strange setting, Mapred.job.classpath.files value for "/user/heipark/lib/commons-lang-2.3.jar; /user/heipark/lib/guava-r08.jar ", you can see that this delimiter is a semicolon (my OS is Windows), the Linux system and the Hadoop system are generally separated by commas and colons, and then I continue to dig, The Discovery Distributedcache.addarchivetoclasspath () method (Tmpjars also used this method) uses "System.getproperty (" Path.separator "), so the inspiration flashes, Modify this value to be the colon of the Linux system, I am a cha, actually succeeded, engaged me for 4 hours, Eclipse finally can add several third party jar package. Encapsulates the method, adding the jar package directly to the main method.

Call:

Addtmpjar ("D:/java/new_java_workspace/scm/lib/guava-r08.jar", conf);

Method definition:

/**
* Add third-party jar packages for MapReduce
*
* @param jarpath
* Example: D:/java/new_java_workspace/scm/lib/guava-r08.jar
* @param conf
* @throws IOException
*/
public static void Addtmpjar (String jarpath, Configuration conf) throws IOException {
System.setproperty ("Path.separator", ":");
FileSystem fs = filesystem.getlocal (conf);
String Newjarpath = new Path (jarpath). makequalified (FS). ToString ();
String tmpjars = Conf.get ("Tmpjars");
if (Tmpjars = = NULL | | tmpjars.length () = = 0) {
Conf.set ("Tmpjars", Newjarpath);
} else {
Conf.set ("Tmpjars", Tmpjars + "," + Newjarpath);
}
}