Write the first mapreduce program in eclipse.

My development environment is described as follows:

Operating System: Ubuntu 10.10 is installed with wubi in windows.
Hadoop version: hadoop-0.20.2.tar.gz
Eclipse version: eclipse-jee-helios-SR1-linux-gtk.tar.gz

For the convenience of learning, this example is developed in the "pseudo distributed mode" hadoop installation mode.

Step 1: Start the hadoop daemon.
If you have read my 1st articlesArticleFull history of hadoop learning-hadoop getting startedWe should be clear about how to start the hadoop daemon in "pseudo-distributed mode.

Step 2: Install hadoop-plugin in eclipse.

1. Copy the hadoop installation directory/contrib/eclipse-plugin/hadoop-0.20.2-eclipse-plugin.jar to the eclipse installation directory/plugins.

2. Restart eclipse and configure hadoop installation directory.
If the plugin is successfully installed, open window --> preferens and you will find the hadoop MAP/reduce option. In this option, you need to configure hadoop installation directory. After the configuration is complete, exit.

3. Configure MAP/reduce locations.
Open Map/reduce locations in window --> show view.
Create a new hadoop location in MAP/reduce locations. In this view, right-click --> New hadoop location. In the pop-up dialog box, you need to configure the location name, such as myubuntu, as well as map/reduce master and DFS master. The host and port here are the addresses and ports you configured in the mapred-site.xml and core-site.xml respectively. For example:

MAP/reduce master

Java Code

1. Localhost
2. 9001

DFS master

Java code

1. Localhost
2. 9000

 

Exit after configuration. Click DFS locations --> myubuntu. If the folder is displayed (2), the configuration is correct. If "no connection" is displayed, check your configuration.
 

Step 3: Create a project.
File --> New --> Other --> MAP/reduce project
Project names can be retrieved as needed, such as hadoop-test.
Copy the hadoop installation directory/src/example/org/Apache/hadoop/example/wordcount. Java to the project you just created.

Step 4: Upload the simulated data folder.
To runProgram, We need an Input Folder and an output folder. Output Folder, which is automatically generated after the program runs successfully. We need to input a folder for the program.

1. Create the input folder in the current directory (such as the hadoop installation directory), and create two files file01 and file02 under the folder. The content of these two files is as follows:

File01

Java code

1. Hello World bye world

File02

Java code

1. Hello hadoop goodbye hadoop

2. Upload the folder input to the distributed file system.

In the hadoop daemon terminal that has started CD to the hadoop installation directory, run the following command:

Java code

1. Bin/hadoop FS-put input input01

This command uploads the Input Folder to the hadoop file system and adds an input01 folder to the system. You can run the following command to view the folder:

Java code

1. Bin/hadoop FS-ls

Step 5: run the project.

1. In the newly created hadoop-test project, click wordcount. Java, right-click --> Run as --> RUN deployments
2. In the pop-up run deployments dialog box, click Java application, right-click --> new, and a new application named wordcount will be created.
3. Configure the running parameters, click arguments, and enter "the Input Folder you want to pass to the program and the folder you want the program to save the computing result" in program arguments, for example:

Java code

1. HDFS:// Localhost: 9000/user/panhuizhi/input01 HDFS: // localhost: 9000/user/panhuizhi/output01

Here input01 is the folder you just uploaded. You can enter the folder address as needed.

4. Click Run to run the program.

Click Run to run the program. After a period of time, the running is completed. After the running is completed, run the following command on the terminal:

Java code

1. Bin/hadoop FS-ls

Check whether the folder output01 is generated.

Run the following command to view the generated file content:

Java code

1. Bin/hadoop FS-cat output01 /*

If the following figure is displayed, congratulations! You have successfully run the first mapreduce program in eclipse.

Java code

1. bye 1
2. goodbye 1
3. hadoop 2
4. Hello 2
5. world 2