Configure Eclipse in Ubuntu to compile and develop Hadoop (MapReduce) source code

This article is not intended for HDFS or MapReduce configuration, but for Hadoop development. The premise for development is to configure the development environment, that is, to obtain the source code and first to build smoothly. This article records the process of configuring eclipse to compile Hadoop source code on Linux (Ubuntu10.10. Which version of the source code should be used to develop Hadoop? One option is to track the latest source code synchronously, And the other option is to select

This article is not intended for HDFS or MapReduce configuration, but for Hadoop development. The premise for development is to configure the development environment, that is, to obtain the source code and first compile smoothly (build). This article records) configure eclipse to compile Hadoop source code.

Which version of the source code should be used to develop Hadoop? One option is to track the latest source code synchronously, And the other option is to select a newer stable release version. To improve Hadoop itself, select the former (for example, the Hadoop core developer with write permission on the SVN source). If other products are developed based on Hadoop, select the latter, the product must be updated to make it compatible only when a new stable version of Hadoop is released. In any situation, the source code should be synchronized from the official SVN,Not recommendedDownload the package file of the release version and import the source code to eclipse. Because the release version is not for recompilation, some problems may occur during compilation.

1. Install the Java Development Tool (JDK)

Eclipse and Hadoop are both developed using Java. Therefore, we need JDK first and execute the following command line:

$ Sudo apt-get install openjdk-6-jdk

2. Install Eclipse

Recommended Development Environment Using eclipse, directly download the latest version of Eclipse IDE for Java Developers from the http://www.eclipse.org/downloads, local decompression can be used.

3. Install Subclipse (SVN plug-in of Eclipse)

Obtain source code from Hadoop. We use the software installation function provided by eclipse to automatically download and install the plug-in through the address provided by Subclipse.

• First visit the http://subclipse.tigris.org, find the Subclipse version that is appropriate for the Eclipse version on the Download and Install page, the latest is 1.6, copy its Eclipse update site URL (that is, the http://subclipse.tigris.org/update_1.6.x;
• Then open the eclipse menu Help> Install New Software, click the Add button, fill in the copied installation address (that is, the http://subclipse.tigris.org/update_1.6.x), you can name the address subclipse; www.linuxidc.com click OK back to the Install window, select Subclipse and then press Next to install it gradually.
• In addition, you also need to install the JavaHL required by SVN and execute the following command line:
  $ Sudo apt-get install libsvn-java 

4. Download Hadoop (MapReduce) source code

First, the Hadoop source code consists of three parts: common, hdfs (Distributed File System), and mapred (MapReduce). There is a dependency between them: common is the public component of the later two, mapred is based on hdfs. however, these dependencies have been well handled without developers' concern,Three projects can be separately downloaded and compiled. This article takes MapReduce as an example.

The source code SVN address is at: http://svn.apache.org/repos/asf/hadoop.

Can be accessed directly from a browser, under whichThe directory structure may change.However, you can find the desired part based on common sense. Currently, the "common" Directory (which has different meanings from the common component mentioned in the previous section) contains the code we usually need. Go to the common directory:

• If you select the latest version, you can go directly to the trunck directory, such as MapReduce Source Code address: http://svn.apache.org/repos/asf/hadoop/common/trunk/mapreduce
• If you select the release version, you need to go to the tags directory browsing, find the desired version, such as 0.21.0 MapReduce Source Code address: http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.21.0/mapreduce
  

After confirming the SVN address, the local eclipse requires the following operations:

• In eclipse, click File> Import, select SVN> check items from SVN, and click Next to go to the Next step;
• Select "Create a new resource library location", click the Next button, enter the address of the corresponding part, and continue to wait for the directory structure to be listed. Select the root directory;
• Next, select "check out as a new project and use the new Project Wizard for configuration". Do not select "check out as a project in the workspace" for www.linuxidc.com because the project type cannot be selected; click Finish to enter the New Project wizard;
• Select Java Project as the Project type, and select the Project name. JRE should select 1.6. Click Next to enter the Java Setting window;
• Set Default output folder... /bin changed... /build/eclipse-classes (or other) to prevent eclipse from automatically clearing the manually written files in the bin folder. Click Finish to wait for the download to complete.

At this time, there will be a lot of files marked with a Red Cross number, which is a normal phenomenon. Later we will use ant to compile the supporting eclipse project file to eliminate these errors.

5. Set to compile using Ant Builder

Taking the mapred section as an example, set the project created in the previous step as MapReduce.

• Right-click the Project (or select Properties from the Project menu), find the Builders tab, deselect the Java Builder check box, and click the New button to select Ant Builder to enter the Edit Configuration window;
• It is initially the Main tab. You can set the Name mapcece_builder (or other) in the Name column. In the Buildfile column, select the build. xml file under the MapReduce project through the Browse Workspace button;
• Switch to the Targets tab and click the Set Targets button in the Manual Build column. In addition to the default compile option, check compile-test-system and eclipse (automatically generate eclipse project files ); if the search is inconvenient, select Sort targets (in alphabetical order) and Hide internal targets.

After confirmation, the project starts compilation (For A Long Time). You can see that the Console has various outputs, and several files will be automatically downloaded during this period.

Right-click the project and select Refresh (or press F5). You can see that the organization structure of the project has changed and the old Red Cross has disappeared.

[Note] Because the Project size is large, after the first compilation, You Can deselect Build Automatically in the Project menu.