Hadoop-eclipse Development environment Setup and error:failure to login error.

For Hadoop developers, programming through the Java API is the first step into map-reduce distributed development. Since eclipse itself does not provide support for the MapReduce programming model, there are some simple steps to implement.

1. Install Hadoop.

Hadoop in this article is a pseudo-distribution pattern deployed on a virtual machine. The relevant software environment is as follows:

Jdk:sun jdk1.6.0_30

hadoop:hadoop-0.20.203.0

Linux:ubuntu 11.10

Eclipse:eclipse Indigo 3.7.1

The specific configuration of the Hadoop pseudo-distributed environment is not mentioned here. In terms of setting, the Fs.default.name in Core-site.xml is hdfs://localhost:9000

2. Setting up the environment

Enter in the terminal

$ sudo vim/etc/profile

Modify the running environment and add the following at the end of the file:

Export hadoop_home=/home/wangyucao/hadoop-0.20.203.0 (here is the installation directory for HADOOP)
Export path= $PATH: #HADOOP_HOME/bin

3. Install Eclipse

Search for Eclipse in Ubuntu Software Center for installation, or download the Eclipse archive from your website. Here you choose to download through the official website to get:

Eclipse-jee-indigo-sr1-linux-gtk.tar.gz

Unzip the file and place it in the/usr directory:

$ TAR-ZXVF eclipse-jee-indigo-sr1-linux-gtk.tar.gz

$ sudo mv eclipse/usr/

Complete the installation steps.

4. Installing the Hadoop-eclipse Plugin

The Hadoop-eclipse-plugin plugin is included in the release version of Hadoop. When you develop a Hadoop application, you need to install the Eclipse plug-in first.

Seeing tutorials on the web, it is generally said that the Hadoop-eclipse-plugin-0.20.203.0.jar in Hadoop_home/lib is copied directly to the plugins directory under the Eclipse installation directory. However, in my practice attempt, found that the hadoop-0.20.203.0 version of the package if copied directly to the Eclipse plug-in directory, the connection to the DFS error, the message is: "Error:failure to Login", the popup Error box content is " An internal error occurred during: "Connecting to DFS Hadoop". Org/apache/commons/configuration/configuration ". After looking at Eclipse's log, the discovery was caused by a missing jar package. After further finding the data, it was found that the Hadoop-eclipse-plugin-0.20.203.0.jar was copied directly, and the jar package was missing from the Lib directory in the package.

After the online data collection, the correct installation method is given here:

The first thing to do is to modify the Hadoop-eclipse-plugin-0.20.203.0.jar. Open the package with the Archive Manager and find only Commons-cli-1.2.jar and Hadoop-core.jar two packages. The Hadoop_home/lib directory Commons-configuration-1.6.jar, Commons-httpclient-3.0.1.jar, Commons-lang-2.4.jar, 5 packages such as Jackson-core-asl-1.0.1.jar and Jackson-mapper-asl-1.0.1.jar are copied to the Hadoop-eclipse-plugin-0.20.203.0.jar Lib directory, as shown below:

Then, modify the MANIFEST.MF in the package Meta-inf directory to change the classpath to something:

bundle-classpath:classes/,lib/hadoop-core.jar,lib/commons-cli-1.2.jar,lib/commons-httpclient-3.0.1.jar,lib/ jackson-core-asl-1.0.1.jar,lib/jackson-mapper-asl-1.0.1.jar,lib/commons-configuration-1.6.jar,lib/ Commons-lang-2.4.jar

The following figure:

This completes the modification of the Hadoop-eclipse-plugin-0.20.203.0.jar.

Finally, copy the Hadoop-eclipse-plugin-0.20.203.0.jar to the plugins directory of Eclipse:

$ CD ~/hadoop-0.20.203.0/lib

$ sudo cp hadoop-eclipse-plugin-0.20.203.0.jar/usr/eclipse/plugins/

5. Configure the plug-in in Eclipse.

First, open Eclipse and set the workspace.

Select Windows---open perspective, other ..., select the Map/reduce with the elephant icon.

At this point, the Map/reduce development environment is opened. Can see

To the lower right corner of a map/reduce locations box. The following figure:

Then, set the environment parameters for Hadoop. Select the Map/reduce Locations tab, click on the icon on the far right of the label, the elephant icon on the right side of the gear icon, open the parameter Settings page, and fill in the parameters as shown below:

Location Name: Set the name of the parameter here, you can fill in any

Map/reduce Master (Here is the Map/reduce address for the Hadoop cluster, which should be the same as the Mapred.job.tracker settings in Mapred-site.xml)

Host:localhost

port:9001

DFS Master (here is the master server address for Hadoop, which should be the same as the Fs.default.name setting in Core-site.xml)

Host:localhost

port:9000

When the settings are complete, click Finish to apply the setting.

In this case, you can see the DFS directory in the leftmost Project Explorer, as shown in the following figure.

Here, the Hadoop plugin setup process is complete.