Hadoop development Environment Builds-ECLIPSE plug-in configuration

Hadoop development is divided into two components: the build of Hadoop clusters, the configuration of the Eclipse development environment. Several of the above articles have documented my Hadoop cluster setup in detail, A simple Hadoop-1.2.1 cluster consisting of a master and two slave was successfully built, and the Hadoop development environment was built by installing the Hadoop plugin in Eclipse, leaving the Hadoop application in one step. Hadoop developers have developed Hadoop Eclipse plug-ins that embed eclipse in the Hadoop development environment, enabling a graphical development environment that reduces the difficulty of writing and debugging Hadoop applications. After installing the Hadoop plugin in Eclipse and configuring the information, if the user creates a Hadoop program, the plugin automatically imports the jar file of the Hadoop programming interface so that the user can write, debug, Run Hadoop programs, including standalone and distributed programs, where you can view your program's real-time status, error messages, and run results, as well as view, manage HDFs, and files.

The structure of my Hadoop development environment is shown in the following figure. One of the CentOS6.5 hosts has an eclipse with the Hadoop plugin configured to write the source code for the Hadoop application, and the other three CentOS6.5 hosts the Hadoop cluster. Set up the four hosts a local area network, the subnet mask is 255.255.255.0, the IP address is allocated as shown in the figure. It is important to note that the users who are installed on the Eclipse host are consistent with the users of the Hadoop cluster and avoid permissions issues during application operation. During the Hadoop cluster build process, the user is Sunnie, and the host on the application development is also logged on as a user Sunnie.

Start configuring the Eclipse development environment below. The first step is to install Eclipse on the CentOS6.5, and my CentOS comes with eclipse because I chose to install the software when I installed the system. Installing Eclipse on CentOS is also very simple and can be self-baidu ~ with Eclipse, the next step is to install and set up Hadoop plug-ins, refer to the Eclipse-based Hadoop application development environment configuration steps are as follows:

1, Install the Hadoop Source: Before installing eclipse download hadoop-1.2.1.tar.gz extract to the/home/sunnie/documnes/directory

2. Plugin access: My Hadoop cluster version is Hadoop-1.2.1, so install the Hadoop-1.2.1 plugin in eclipse. Hadoop developers release the Hadoop contains the source of the package, including the relevant Eclipse plug-in Source, 0.20.x later version does not contain the Eclipse plug-in jar package, because the version of Eclipse is different, so you need to compile the source code to generate the corresponding plug-in. Eclipse Plugin Compilation Method reference Hadoop 1.2.1 Eclipse Plugin compile here I stole a lazy, directly from this blog link to download the compiled plug-in, after decompression for Hadoop-eclipse-plugin-1.2.1.jar

3, plug-in installation: first to turn off eclipse, if it is running. Su-Switch to the root user, copy the above obtained Hadoop-eclipse-plugin-1.2.1.jar to the/usr/share/eclipse/dropins/directory, switch back to the user Sunnie of the Hadoop cluster, It's so simple ~

4.Configuration of Hadoop installation directory: start the Hadoop cluster first and then start Eclipse. Open Window-->preferens, if the installation is successful, you will find the Hadoop map/reduce option, in this option you need to configure Hadoop installation directory, which is the Hadoop source code installation directory in step 1, My source code is located after the/home/sunnie/documents/hadoop-1.2.1 configuration is complete and exits.

5,map/reduce locations configuration: Select Window->open perspective->other in Eclipse, select the map/reduce of the elephant icon, you can switch to map/ Reduce working directory. Right-click "Map/reduce Locations" under the Eclipse work window and select New Hadoop location, configure location name, such as Hadoop or master, and Map/reduce Master and DFS Master, the host and port are the addresses and ports configured in the Mapred-site.xml, core-site.xml files in the Hadoop cluster setup process, and click Finish to exit when the configuration is complete. My configuration is shown in the following figure:

In the case where the Hadoop cluster is running normally, click the Elephant icon under DFS locations in the left window of Eclipse, if the display folder (3) illustrates the configuration success, as shown in the following figure:

The Map/reduce project is created below in Eclipse. File-->new-->other-->map/reduce project, project name takes Try1, click Finish. You can see that eclipse automatically loads the development Hadoop application into the associated jar package into the new project