Running, configuring Hadoop in Eclipse

Copyright: [Email protected] Prohibited reprint!

1. Installing plugins

To prepare the program:

eclipse-3.3.2 (this version of the plugin can only be used with this version of Eclipse)
Hadoop-0.20.2-eclipse-plugin.jar (under the Hadoop-0.20.2/contrib/eclipse-plugin directory)

Copy the Hadoop-0.20.2-eclipse-plugin.jar to the Eclipse/plugins directory and restart Eclipse.

2. Open the MapReduce view

Window---Open perspective, other select Map/reduce, the icon is a blue elephant.

3. Add a MapReduce environment

At the lower end of eclipse, there will be a tab next to the console, called "Map/reduce Locations", right-click in the blanks below, select "New Hadoop location ...",:

In the dialog box that pops up, fill in the following:

Location name(take a name)
map/reduce Master(the IP and port of the Job tracker, according to the mapred.job.tracker configured in Mapred-site.xml)
DFS Master(Name node IP and port, based on fs.default.name configured in Core-site.xml)

4. Use Eclipse to modify HDFS content

After the previous step, the left "Project Explorer" should appear in the configuration of HDFs, right-click, you can create new folders, delete folders, upload files, download files, delete files and other operations.

Note: The changes cannot be displayed immediately after each operation in eclipse and must be refreshed.

5. Create a MapReduce Project 5.1 Configure the Hadoop path

Window---Preferences Select "Hadoop map/reduce" and click "Browse ..." to select the path to the Hadoop folder.
This step is independent of the operating environment, but it is possible to automatically import all the jar packages from the Hadoop root and the Lib directory when the new project is created.

5.2 Creating a Project

Project, File, New, select Map/reduce Project, and then enter the project name to create the project. The plugin automatically imports all the jar packages from the Hadoop root and the Lib directory.

5.3 Creating mapper or Reducer

Mapper, File, New, creates the Mapper, automatically inherits the Mapreducebase from the mapred package and implements the Mapper interface.
Note: This plugin automatically inherits the old version of the mapred package and the interface, the new version of the Mapper have to write their own.

Reducer.

6. Run WordCount Program 6.1 import WordCount in eclipseWordCount6.2 Configuring Run Parameters

Run as--Open run Dialog ... Select the WordCount program to configure the run parameters in arguments:/MAPREDUCE/WORDCOUNT/INPUT/MAPREDUCE/WORDCOUNT/OUTPUT/1

Represents the input directory and output directory under HDFs, where there are several text files in the input directory, and the output directory must not exist.

6.3 Run

Run as-run on Hadoop Select the previously configured MapReduce runtime environment and click "Finish".

The console outputs the relevant running information.

6.4 View run results in the output directory/MAPREDUCE/WORDCOUNT/OUTPUT/1, you can see the output file of the WordCount program. In addition, you can see a logs folder, there will be running logs