Running Hadoop on Eclipse HelloWorld

What's the use of Hadoop? I'm not going to tell you about the first time you ran Hadoop on eclipse.

Hadoop installation I'm not going to say a lot of online tutorials-I use Hadoop on the company's Linux directly.

Eclipse uses Mars,jdk1.8,win7,hadoop as the Hadoop 1.2.1 on the company's Linux, and the associated JDK is 1.7.

First download the plugin hadoop-eclipse-plugin-1.2.1 for Hadoop affiliate eclipse. It's best to download the same version, or it's easy to go wrong (I'm typical). Put the plugin into the Eclipse installation directory under the plugins (hadoop-eclipse-plugin-2.2.0 in the Dropins directory is OK, but 1.2.1 is not), restart Eclipse, on the eclipse page of this stuff ~

　　　　　　　　　　

Next Configure Hadoop, click on the elephant on the graph and this appears under Eclipse. Click on the blue elephant on the right to go to the Hadoop configuration page.

　　　　　

Detailed information about the configuration is shown on the diagram. Linux commands This is not said here, even Linux recommended with SECURECRT.

Click Finish and wait for completion. Upon completion, on the left side of Eclipse, DFS location appears below Project Explorer. Success will load Hadoop files, equivalent to the results of the Hadoop fs-ls command. If there is a connection failure, check whether the address or port is wrong, can be in the CMD Telnet port to see if it can be connected, Telnet 192.168.0.201 49000. If prompted "Your computer a software block access", then may be hadoop-eclipse plug-in version of the wrong ~

All ok so you can write HelloWorld.

Package Com.qldhlbs.hadoop.helloworld;import Java.io.ioexception;import Java.util.stringtokenizer;import Org.apache.hadoop.conf.configuration;import Org.apache.hadoop.fs.path;import org.apache.hadoop.io.IntWritable; Import Org.apache.hadoop.io.longwritable;import Org.apache.hadoop.io.text;import Org.apache.hadoop.mapreduce.Job; Import Org.apache.hadoop.mapreduce.mapper;import Org.apache.hadoop.mapreduce.reducer;import Org.apache.hadoop.mapreduce.lib.input.fileinputformat;import Org.apache.hadoop.mapreduce.lib.input.textinputformat;import Org.apache.hadoop.mapreduce.lib.output.fileoutputformat;import Org.apache.hadoop.mapreduce.lib.output.textoutputformat;public class WordCount {public static class Wordcountmap extendsmapper<longwritable, text, text, intwritable> {private final intwritable one = new intwritable (1);p rivate Te XT Word = new Text ();p ublic void map (longwritable key, Text value, Context context) throws IOException, Interruptedexceptio n {String line = value.tostring (); StrinGtokenizer token = new StringTokenizer (line), while (Token.hasmoretokens ()) {Word.set (Token.nexttoken ()); Context.write (Word, one);}}} public static class Wordcountreduce Extendsreducer<text, intwritable, text, intwritable> {public void reduce (text k EY, iterable<intwritable> values,context Context) throws IOException, interruptedexception {int sum = 0;for (IntWri Table val:values) {sum + = Val.get ();} Context.write (Key, New intwritable (sum));}} public static void Main (string[] args) throws Exception {configuration conf = new Configuration (); Job Job = new Job (conf); Job.setjarbyclass (Wordcount.class); Job.setjobname ("WordCount"); Job.setoutputkeyclass ( Text.class); Job.setoutputvalueclass (Intwritable.class); Job.setmapperclass (Wordcountmap.class); Job.setreducerclass (Wordcountreduce.class); Job.setinputformatclass (Textinputformat.class); Job.setoutputformatclass (Textoutputformat.class); Fileinputformat.addinputpath (Job, New Path (Args[0])); Fileoutputformat.setoutputpath (Job, new Path (args[1]); Job.waitforcompletion (True);}} 

The idea to import the package. The Hadoop package and the Jackson package in the downloaded Hadoop files are included, several commons package also to import, or will be error, where Commons-cli-1.3.1.jar is a tool to handle the command. For example, the main method input string[] need to parse. You can pre-define the rules for parameters, and then you can invoke the CLI to parse them. The above code will be error-less than this package.

　　　　　　　　　　　　　　　　　　　　　　

Next, create a file in Hadoop. Command Hadoop fs-mkdir/user/qldhlbs/input to create a folder. These Hadoop commands are similar to Linux's/user/qldhlbs/input, which are created folder directories. Right-click Dfs location refresh or re-connect on Eclipse and you will see the Qldhlbs/input directory in the user directory. To put the file to count in input, you can use the simple method, right-click on the creation of input, select Upload file to DFS. If you do not have an accident, you will be reminded that you have no permission. Command Hadoop fs-chmod 777/user/qldhlbs/input. Where 777 is the read and write execution permission (because it corresponds to 4,2,1),/user/qldhlbs/input is the directory that gives permission. You can upload the file after the Eclipse Refresh-upload successfully and then the input directory can see the file you uploaded.

Next right click Run configurations, configure the parameters. Hdfs://192.168.0.201:49000/user/qldhlbs/input Hdfs://192.168.0.201:49000/user/qldhlbs/output, it will automatically help you build the output directory, If you create an error, the file already exists. Click Apply,run. Parameter address don't write wrong, originally I wrote wrong to find a half-day error did not find ~zzz. Here will be prompted qldhlbs This directory does not have permissions, and the same as before the county on the line. If you use 1.2.1 This version of the words will also prompt a tmp/hadoop-administrator/xxxx do not have permission error, change the Hadoop-core-1.2.1.jar file source code on the line, decompile this jar, The Checkreturnvalue method in the Fileutil class under the FS package will comment out the contents.

　　　　　　　　　　

RePack the changed file into a jar and put it in the project, run it. A successful run will show two files below the output.

To here Hadoop's HelloWorld finished ~ did not carefully look at the time, out of some should not make a small mistake, wasting a lot of time ~

Running Hadoop on Eclipse HelloWorld