Executing, configuring Hadoop in Eclipse

All rights reserved: [email protected] Prohibited reprint!

1. Installing plugins

To prepare the program:

eclipse-3.3.2 (This version number of plugins can only use this version number of Eclipse)
Hadoop-0.20.2-eclipse-plugin.jar (under the Hadoop-0.20.2/contrib/eclipse-plugin folder)

Copy the Hadoop-0.20.2-eclipse-plugin.jar to the Eclipse/plugins folder and restart Eclipse.

2. Open the MapReduce view

Window---Open perspective, other select Map/reduce, the icon is a blue elephant.

3. Join a MapReduce environment

At the lower end of eclipse, there will be a tab next to the console, called "Map/reduce Locations", right-click in the blanks below and select "New Hadoop location ..." and see:

In the popup dialog box, fill in the following example:

Location name(take a name)
map/reduce Master(IP and port for Job Tracker, based on Mapred.job.tracker configured in Mapred-site.xml)
DFS Master(Name node IP and port, based on fs.default.name configured in Core-site.xml)

4. Use eclipse to make changes to HDFS content

After the previous step, the left side "Project Explorer" should appear in the configuration of HDFs, right-click, to be able to create new directories, delete directories, upload files, download files, delete files and other operations.

Note: You will not be able to display changes immediately after each operation in eclipse, you must refresh it.

5. Create a MapReduceproject5.1 configuration Hadoop path

Window---Preferences Select "Hadoop map/reduce" and click "Browse ..." to select the path to the Hadoop directory.
This step is independent of the execution environment, and it is only possible to actively import all the jar packages under the Hadoop root folder and the Lib folder itself when you create a new project.

5.2 Creating Project

Project, File, New, select Map/reduce Project, and then enter the project name to create the project. The plugin will take the initiative to import all the jar packages under the Hadoop root folder and the Lib folder.

5.3 Creating mapper or Reducer

File, New, Mapper create Mapper, and take the initiative to inherit the mapreducebase inside the mapred package and implement the Mapper interface.
Note: This plugin self-inheriting is the mapred package in the old version of the class and interface, the new version of the Mapper have to write their own.

Reducer.

6. Execute WordCount Program 6.1 import WordCount in eclipseWordCount6.2 Configuring the execution of the parameters

Run as--Open run Dialog ... Select the WordCount program to configure the execution parameters in arguments:/MAPREDUCE/WORDCOUNT/INPUT/MAPREDUCE/WORDCOUNT/OUTPUT/1

Indicates the input folder and output folder under HDFs, where there are several text files in the input folder, and the output folder must not exist.

6.3 Execution

Run as--run on Hadoop Select the previously configured MapReduce execution environment and click "Finish" to execute.

The console outputs the relevant execution information.

6.4 View execution results in the output folder/MAPREDUCE/WORDCOUNT/OUTPUT/1, you can see the output file of the WordCount program. In addition, you can see a logs folder, there will be executed log