Hadoop source code compilation

Reprinted from: http://jbm3072.iteye.com/blog/1113827

Hadoop is a framework of distributed storage and computing. When we use hadoop on a daily basis, we will find that hadoop cannot fully meet our needs. We may need to modify the hadoop source code and re-compile and package it.

The following describes in detail how to import from SVN and compile hadoop in eclipse.

 

Because the project uses the hadoop version 0.20.2, So we based on hadoop-0.20.2 for secondary development.

 

(1) first, check out the source code from SVN. SVN address:

 

URL code

1. Http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.2/

 

 

(2) Prepare the development and compilation environments during the download process. Install at least the following software in Windows:

Java code

1. JDK 6 -- you need to set the PATH environment variable.
2. Ant
3. -- After downloading and unzipping, add the ant bin directory to the environment variable of path.
4. Cygwin
5. -- Install cygwin refer to the http://ebiquity.umbc.edu/Tutorials/Hadoop/03%20-%20Prerequistes.html for installation and configuration. Select as many installation packages as possible in cygwin to facilitate future development.

 

 

 

 

(3) After checking out from SVN, use the command line to enter the home directory downloaded from hadoop and execute the command:

 

Java code

1. $ Ant

 

Now, ant starts to download dependencies and compile files. I made a compilation error during compilation. It was found that there was a problem with the package-info.java generated by $ hadoop_home/src/saveversion. Sh, causing the compilation to fail. Modify saveversion. sh:

 

Java code

1. Unset Lang
2. Unset lc_ctype
3. Version = $1
4. User = 'whoam' # change it to a fixed value, for example, jbm3072.
5. Date = 'date'
6. If [-D. Git]; then
7. Revision = 'git log-1 -- pretty = format: "% H "'
8. Hostname = 'hostname'
9. Branch = 'git branch | sed-n-e's/^ * // P''
10. Url = "Git: // $ Hostname/$ CWD on Branch $ branch"
11. Else
12. Revision = 'svn info | sed-n-e's/last changed rev: \ (. * \)/\ 1/P''
13. Url = 'svn info | sed-n-e's/url: \ (. * \)/\ 1/P''
14. Fi
15. Mkdir-P build/src/org/Apache/hadoop
16. Cat <EOF | \
17. Sed-e "s/version/$ version/"-e "s/user/$ user/"-e "s/date/$ date /"\
18. -E "s | URL | $ URL |"-e "s/Rev/$ revision /"\
19. > Build/src/org/Apache/hadoop/package-info.java
20. /*
21. * Generated by src/saveversion. Sh
22. */
23. @ Hadoopversionannotation (version = "version", revision = "Rev ",
24. User = "user", date = "date", url = "url ")
25. Package org. Apache. hadoop;
26. EOF

 

After the modification, the compilation is successful.

 

(4) Copy eclipse-files to the project directory

Run the following command:

Java code

1. Ant eclipse-Files

You can copy eclipse-files to the project directory.

(5) import the project

 

Open eclipse, select import in file, select general-> exsiting projects into workspace, click Next, and select the hadoop source code directory. Now you can identify hadoop as an Eclipse project, click Finish. After a while, an Eclipse project without errors will be OK.

 

(6) Now you can modify the hadoop source code based on Eclipse.