Hadoop learning notes-3. Hadoop source code eclipse compilation tutorial

1. Download Hadoop source code
Source code of each Hadoop Member: Just pull it out. Note that only the contents in the trunk directory on SVN are checked-out, for example:
Http://svn.apache.org/repos/asf/hadoop/common/trunk,
Instead of http://svn.apache.org/repos/asf/hadoop/common,
The reason is that the http://svn.apache.org/repos/asf/hadoop/common directory contains a lot of non-source code files, very large, resulting in a long check-out time.

2. Prepare the compiling environment

2.1. Hadoop code version
The Hadoop used in this tutorial is the source code downloaded on the morning of, Beijing time, And the hadoop-0.19.x version of the difference may be large.
2.2. Networking
Compiling Hadoop depends on many third-party libraries, but the compilation tool Ant automatically downloads the missing libraries from the Internet. Therefore, you must ensure that the machine can access the Internet.
2.3. java
Compile Hadoop with JDK1.6 or above at: http://java.sun.com/javase/downloads/index.jsp.
After installation, set the JAVA_HOME environment variable.
2.4. Ant and Cygwin
Ant needs to use Ant tool to compile Hadoop, and Ant needs to use some tools provided by Cygwin, such as sed, can download Ant from: http://ant.apache.org/ivy/download.cgi, from http://www.cygwin.cn/download Cygwin (Cygwin installation, for more information, see "tutorial on installing Hadoop on Windows ). After installation, add the bin directories of Ant and Cygwin to the environment variable PATH, as shown in:

Note: When Cygwin is installed, we recommend that you install SVN because it will be downloaded through SVN during Ant compilation.
Some files, but this is not necessary. If the download is unsuccessful, no errors are found, and the compilation can still be successful.

2.5. Eclipse

Eclipse can be downloaded from http://www.eclipse.org/downloads.

 

3. Compile Hadoop

Here, we take compiling the Hadoop family member common as an example. The compilation method for other Hadoop members is similar.
3.1. Compile common members
Step 1) Right-click the Elipse Package view and choose New> Java Project, as shown in:

Step 2) Select the source code directory and set the Project name.

In the displayed dialog box, click the Browse button, select the common source code directory, and set Projectname to common. After the project is imported, go to the Eclipse main interface and you can see that the common has been imported in, but you can see that the common has a Red Cross, because Elipse uses Java Builder by default, instead of Ant Builder, so the next step is to use Ant Builder.
Step 3) set Builder to Ant: Right-click common-> Properties-> Builders:

On the interface shown in, click the New button and select Ant Builder in the pop-up dialog box. After confirmation, the following dialog box is displayed:

Click the Browse File System button and select build under the common source code directory. xml file, and set Name to common_Builder (Name can be changed to another one, but we recommend that you use common_Builder because it is a real Name). The operation result is as follows:

In addition to the settings shown, you also need to set Targets as shown in. We recommend that you set it to the Manual Build compilation method instead of the Auto Build compilation method. In Auto Build mode, any modification will trigger compilation. In Manual Build mode, you only need to click the compile button or menu compile when necessary.

Each Hadoop member needs to be compiled into a jar, so make a change as shown in:

After confirmation, return to the Edit Configuration dialog box as shown in:

After completing the preceding steps, return to the Builder Main Dialog Box, move the Java Builder in the dialog box down, and remove the preceding check box.
Go to the main Eclipse interface. Because Manual Build was selected, Manual compilation is required. After compilation is SUCCESSFUL, you can see the words "build successful.

NOTE: If Build Automatically in the menu shown in is checked, the Build sub-menu may not appear in the context menu of common.
Ant automatically downloads the dependent libraries from the Internet during compilation. After the compilation of common is successful, you can find the file hadoop-core-0.21.0-dev.jar generated after compilation in the build directory.

3.2. compile other Hadoop members
Hdfs, mapreduce, and hive are compiled in the same way as common.

 

4. FAQ

4.1. Networking
Make sure that you can access the internet. Ant needs to automatically download many third-party dependent libraries. If you cannot connect to the internet, compilation will be much more complicated.
4.2. Compile hive
Hive compilation is more complex. By default, it compiles multiple versions of hive, creates and modifies the ivy. xml file under the shims directory, and removes unnecessary version compilation.
4.3. compile and generate the file location
Build \ hadoop-core-0.21.0-dev.jar generated after common compilation;
Build \ hadoop-hdfs-0.21.0-dev.jar generated after hdfs compilation;
Build \ hadoop-mapred-0.21.0-dev.jar generated after mapreduce compilation;
Build \ service \ hive_service.jar is generated after hive compilation. Note that it is not directly placed in the build directory;
Build \ hbase-0.21.0-dev.jar generated after hbase compilation;
Sometimes an error occurs during compilation. You can skip this step and compile others, Refresh, and then compile.