Single-machine pseudo-distributed deployment of Hadoop under Windows (3)

The integration of MyEclipse with Hadoop is described below.

The MyEclipse version I used was 8.5.

1. Installing the Hadoop development plugin

Hadoop-eclipse-pligin-1.2.1.jar is no longer available under the hadoop1.2.1 version of the installation package contrib/directory;

Instead, the source code files are provided, and we need to recompile them to the jar package file, which is convenient for you to download from here:

Hadoop-eclipse-plugin-1.2.1.jar.pdf

Due to the type limitation of the blog upload file, the PDF suffix is added, renamed after download, and ". pdf" is removed.

The file is then copied to the/dropins directory under the MyEclipse root directory.

2. Set MyEclipse

Start MyEclipse, open perspective;

"Window", "Open Perspective", "other ...", "Map/reduce", "OK"

Then open view:

"Window", "Show View", "Other ...", "MapReduce Tools", "Map/reduce Locations", "OK"

Then, add the Hadoop location and right-click in the space below Map/reduce locations:

(I have built one here, so there will be one, the first build, should be blank)

Then choose Edit Hadoop location

Location name: I filled in: first-hadoop
Map/reduce Master in this box
Host: The cluster machine where the Jobtracker is located, write localhost here
Hort: This is Jobtracker's port, which is written in 9001.
These two parameters are the IP and port inside Mapred.job.tracker inside the mapred-site.xml.

DFS Master in this box
Host: The cluster machine where the Namenode is located, write localhost here
Port: It's Namenode's port, which writes 9000.
These two parameters are the IP and port inside Fs.default.name inside the core-site.xml.

(Use M/R master host, this check box if selected, the default and Map/reduce master in this box, the same as the host, if not selected, you can define the input, here Jobtracker and Namenode on a machine, so is the same , tick the box)

User name: This is the username that connects to Hadoop, and mine is LDM.
Then click the Finish button, at which point there is more than one record in this view.

Restart MyEclipse and re-edit the connection record that you just created, and now we edit Advance Parameters tab page

Most of the properties here have been automatically filled in, in fact, the Core-defaulte.xml, Hdfs-defaulte.xml, mapred-defaulte.xml inside some of the configuration properties show out.

Because there are changes in the site family configuration file when installing Hadoop, it is also necessary to make the same settings here.

The main concerns are the following properties:
Fs.defualt.name: This has been set in the General tab page
Mapred.job.tracker: This is also set on the General tab page
Dfs.replication: The default here is 3, because I set it to 1 in Hdfs-site.xml, so this is set to 1.
Then click Finish and then connect (start the SSHD service first, start the Hadoop process), connect the flag

3. WordCount instances

new Map/reduce Project:

"File", "New", "Project ...", "Map/reduce", "Map/reduce project"

"Project Name:wordcount", "Configure Hadoop install directory ...", "Hadoop installation directory:c:\cygwin64\ Home\ldm\hadoop "

"Apply", "OK", "Next", "Allow output folders for source folders", "Finish"

new WordCount class :

The code in the instance Wordcount.java class in the Hadoop installation package is then copied into.

Next, create a new folder, input, create a new two files input/file1,input/file2;

File contents are Hello World Bye world and Hello Hadoop Goodbye Hadoop

Then run the command in the Cygwin terminal: Hadoop fs-put input input

The folder input is uploaded to the Distributed File System , and the path in the command is based on its own path.

To configure the run parameters:

① in the new Project WordCount, click Wordcount.java, right--->run as-->run configurations
② in the pop-up Run Configurations dialog box, click Java Application, right-click-->new, then create a new application named WordCount
③ Configure the run parameters, click Arguments, enter "The input folder you want to pass to the program and the folder where you require the program to save the calculation results", as:

then click Run;

If shown below, the first MapReduce program has been successfully run under MyEclipse.

1. Bye 1
2. Goodbye 1
3. Hadoop 2
4. Hello 2
5. World 2