From zero teaches you how to get hadoop2.4 source code and use Eclipse to associate hadoop2.4 source code

From zero teaches you how to get hadoop2.4 source code and use Eclipse to associate hadoop2.4 source code
Http://www.aboutyun.com/thread-8211-1-1.html
(Source: About Cloud development)

Problem Guide:
1. How to get all the code for Hadoop via the official website src package

3.maven?






If we want to develop, researching the source code is very helpful to us. Do not understand the principle is like a black box, encounter problems, we can not touch the idea. So here you are.
I. How to obtain the source code
two. How to associate source code

How to get the source code

1. Download the maven package for Hadoop

(1) Download the website
Here we first download maven package hadoop-2.4.0-src.tar.gz from the official website.
Official website

For those who do not know how to download the official website, you can view: Beginner's Guide: Introduction to Hadoop website and how to download Hadoop (2.4) Versions and view Hadoop API introduction

(2) screen download
can also be downloaded from the network disk:
Http://pan.baidu.com/s/1kToPuGB

2. There are two ways to get the source code through MAVEN
, one way through the command line, and one through eclipse. This is mainly about the way of the command

Get the source code by command:
1. Unpacking the package

 


The following problem was encountered while unpacking the package. But don't worry, let's go down

1: Unable to create file: D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\ Hadoop-yarn-server-applicationhistoryservice\target\classes\org\apache\hadoop\yarn\server\ Applicationhistoryservice\applicationhistoryclientservice$applicationhsclientprotocolhandler.class:
The total length of the path and filename cannot exceed 260 characters
The system could not find the path specified. D:\hadoop2\hadoop-2.4.0-src.zip
2: Unable to create file: D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\ Hadoop-yarn-server-applicationhistoryservice\target\classes\org\apache\hadoop\yarn\server\        Applicationhistoryservice\timeline\leveldbtimelinestore$lockmap$countingreentrantlock.class: The system cannot find the path specified. D:\hadoop2\hadoop-2.4.0-src.zip
3: Unable to create file: D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\ Hadoop-yarn-server-applicationhistoryservice\target\test-classes\org\apache\hadoop\yarn\server\        Applicationhistoryservice\webapp\testahswebapp$mockapplicationhistorymanagerimpl.class: The system cannot find the path specified. D:\hadoop2\hadoop-2.4.0-src.zip
4: Unable to create file: D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\ Hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\monitor\ Capacity\testproportionalcapacitypreemptionpolicy$ispreemptionrequestfor.class:
The total length of the path and filename cannot exceed 260 characters
The system could not find the path specified. D:\hadoop2\hadoop-2.4.0-src.zip
5: Unable to create file: D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\ Hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\        Testfsrmstatestore$testfsrmstatestoretester$testfilesystemrmstore.class: The system cannot find the path specified. D:\hadoop2\hadoop-2.4.0-src.zip
6: Unable to create file: D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\ Hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\ Testzkrmstatestore$testzkrmstatestoretester$testzkrmstatestoreinternal.class:
The total length of the path and filename cannot exceed 260 characters
The system could not find the path specified. D:\hadoop2\hadoop-2.4.0-src.zip
7: Unable to create file: D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\ Hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\ Testzkrmstatestorezkclientconnections$testzkclient$testforwardingwatcher.class:
The total length of the path and filename cannot exceed 260 characters
The system could not find the path specified. D:\hadoop2\hadoop-2.4.0-src.zip
8: Unable to create file: D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\ Hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\ Testzkrmstatestorezkclientconnections$testzkclient$testzkrmstatestore.class:
The total length of the path and filename cannot exceed 260 characters
The system could not find the path specified. D:\hadoop2\hadoop-2.4.0-src.zip
9: Unable to create file: D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\ Hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\rmapp\ Attempt\testrmappattempttransitions$testapplicationattempteventdispatcher.class:
The total length of the path and filename cannot exceed 260 characters
The system could not find the path specified. D:\hadoop2\hadoop-2.4.0-src.zip

2. Get the source code through MAVEN

It is necessary to note that when using MAVEN, you need to install JDK,PROTOC first, if not installed, you can refer to win7 How to install maven, install Protoc

(1) Enter Hadoop-2.4.0-src\hadoop-maven-plugins, run mvn install

1. D:\HADOOP2\HADOOP-2.4.0-SRC\HADOOP-MAVEN-PLUGINS>MVN Install

Copy Code

Displays the following information

4. [INFO] Scanning for projects ...
6. [WARNING]
8. [WARNING] Some problems were encountered while building the effective model for
10. org.apache.hadoop:hadoop-maven-plugins:maven-plugin:2.4.0
12. [WARNING] ' Build.plugins.plugin. (Groupid:artifactid) ' must is unique but found D
14. uplicate Declaration of Plugin Org.apache.maven.plugins:maven-enforcer-plugin @
16. org.apache.hadoop:hadoop-project:2.4.0, D:\hadoop2\hadoop-2.4.0-src\hadoop-proje
18. Ct\pom.xml, line 1015, column 15
20. [WARNING]
22. [WARNING] It is highly recommended to fix these problems because they threaten t
24. He stability of your build.
26. [WARNING]
28. [WARNING] For this reason, the future Maven versions might no longer support Buildin
30. G such malformed projects.
32. [WARNING]
34. [INFO]
36. [INFO] Using the builder Org.apache.maven.lifecycle.internal.builder.singlethrea
38. Ded. Singlethreadedbuilder with a thread count of 1
40. [INFO]
42. [INFO]------------------------------------------------------------------------
44. [INFO] Building Apache Hadoop Maven Plugins 2.4.0
46. [INFO]------------------------------------------------------------------------
48. [INFO]
50. [INFO]---maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-maven-plugins
52. ---
54. [INFO] Executing tasks
58. Main
60. [INFO] Executed Tasks
62. [INFO]
64. [INFO]---maven-plugin-plugin:3.0:descriptor (default-descriptor) @ hadoop-mave
66. N-plugins---
68. [INFO] Using ' UTF-8 ' encoding to read Mojo metadata.
70. [INFO] Applying Mojo extractor for language:java-annotations
72. [INFO] Mojo Extractor for language:java-annotations found 2 Mojo descriptors.
74. [INFO] Applying Mojo extractor for Language:java
76. [INFO] Mojo extractor for Language:java found 0 Mojo descriptors.
78. [INFO] Applying Mojo extractor for LANGUAGE:BSH
80. [INFO] Mojo extractor for LANGUAGE:BSH found 0 Mojo descriptors.
82. [INFO]
84. [INFO]---maven-resources-plugin:2.2:resources (default-resources) @ Hadoop-mav
86. En-plugins---
88. [INFO] Using default encoding to copy filtered resources.
90. [INFO]
92. [INFO]---maven-compiler-plugin:2.5.1:compile (default-compile) @ hadoop-maven-
94. Plugins---
96. [INFO] Nothing to Compile-all classes is up to date
98. [INFO]
100. [INFO]---maven-plugin-plugin:3.0:descriptor (mojo-descriptor) @ hadoop-maven-p
102. Lugins---
104. [INFO] Using ' UTF-8 ' encoding to read Mojo metadata.
106. [INFO] Applying Mojo extractor for language:java-annotations
108. [INFO] Mojo Extractor for language:java-annotations found 2 Mojo descriptors.
110. [INFO] Applying Mojo extractor for Language:java
112. [INFO] Mojo extractor for Language:java found 0 Mojo descriptors.
114. [INFO] Applying Mojo extractor for LANGUAGE:BSH
116. [INFO] Mojo extractor for LANGUAGE:BSH found 0 Mojo descriptors.
118. [INFO]
120. [INFO]---maven-resources-plugin:2.2:testresources (default-testresources) @ ha
122. Doop-maven-plugins---
124. [INFO] Using default encoding to copy filtered resources.
126. [INFO]
128. [INFO]---maven-compiler-plugin:2.5.1:testcompile (default-testcompile) @ Hadoo
130. P-maven-plugins---
132. [INFO] No sources to compile
134. [INFO]
136. [INFO]---maven-surefire-plugin:2.16:test (default-test) @ hadoop-maven-plugins
138. ---
140. [INFO] No tests to run.
142. [INFO]
144. [INFO]---maven-jar-plugin:2.3.1:jar (default-jar) @ hadoop-maven-plugins---
146. [INFO] Building Jar:d:\hadoop2\hadoop-2.4.0-src\hadoop-maven-plugins\target\had
148. Oop-maven-plugins-2.4.0.jar
150. [INFO]
152. [INFO]---maven-plugin-plugin:3.0:addpluginartifactmetadata (default-addplugina
154. rtifactmetadata) @ hadoop-maven-plugins---
156. [INFO]
158. [INFO]---maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hadoop-
160. Maven-plugins---
162. [INFO]
164. [INFO]---maven-install-plugin:2.3.1:install (default-install) @ hadoop-maven-p
166. Lugins---
168. [INFO] Installing D:\hadoop2\hadoop-2.4.0-src\hadoop-maven-plugins\target\hadoop
170. -maven-plugins-2.4.0.jar to C:\Users\hyj\.m2\repository\org\apache\hadoop\hadoop
172. -maven-plugins\2.4.0\hadoop-maven-plugins-2.4.0.jar
174. [INFO] Installing D:\hadoop2\hadoop-2.4.0-src\hadoop-maven-plugins\pom.xml to C:
176. \users\hyj\.m2\repository\org\apache\hadoop\hadoop-maven-plugins\2.4.0\hadoop-ma
178. Ven-plugins-2.4.0.pom
180. [INFO]------------------------------------------------------------------------
182. [INFO] BUILD SUCCESS
184. [INFO]------------------------------------------------------------------------
186. [INFO] Total time:4.891 S
188. [INFO] Finished at:2014-06-23t14:47:33+08:00
190. [INFO] Final memory:21m/347m
192. [INFO]------------------------------------------------------------------------

Copy Code

Sections are as follows:








(2) operation

1. MVN eclipse:eclipse-dskiptests

Copy Code

Now, notice that we're in hadoop_home, and I'm D:\hadoop2\hadoop-2.4.0-src here.

Some of the information below

2. [INFO]
4. [INFO]------------------------------------------------------------------------
6. [INFO] Reactor Summary:
8. [INFO]
10. [INFO] Apache Hadoop Main ......... ............... SUCCESS [0.684 S]
12. [INFO] Apache Hadoop Project POM ......... ......... SUCCESS [0.720 S]
14. [INFO] Apache Hadoop Annotations ......... .......... SUCCESS [0.276 S]
16. [INFO] Apache Hadoop Project Dist POM ......... ..... SUCCESS [0.179 S]
18. [INFO] Apache Hadoop assemblies ......... .......... SUCCESS [0.121 S]
20. [INFO] Apache Hadoop Maven Plugins ........ ......... SUCCESS [1.680 S]
22. [INFO] Apache Hadoop minikdc ......... ............. SUCCESS [1.802 S]
24. [INFO] Apache Hadoop Auth ......... ............... SUCCESS [1.024 S]
26. [INFO] Apache Hadoop Auth Examples ........ ......... SUCCESS [0.160 S]
28. [INFO] Apache Hadoop Common ......... ............. SUCCESS [1.061 S]
30. [INFO] Apache Hadoop NFS ......... ................ SUCCESS [0.489 S]
32. [INFO] Apache Hadoop Common Project ......... ....... SUCCESS [0.056 S]
34. [INFO] Apache Hadoop HDFS ......... ............... SUCCESS [2.770 S]
36. [INFO] Apache Hadoop Httpfs ......... ............. SUCCESS [0.965 S]
38. [INFO] Apache Hadoop HDFS bookkeeper Journal .... ..... SUCCESS [0.629 S]
40. [INFO] Apache Hadoop Hdfs-nfs ......... ............ SUCCESS [0.284 S]
42. [INFO] Apache Hadoop HDFS Project ......... ......... SUCCESS [0.061 S]
44. [INFO] Hadoop-yarn ..... ....... ................... SUCCESS [0.052 S]
46. [INFO] Hadoop-yarn-api .......... ................. SUCCESS [0.842 S]
48. [INFO] Hadoop-yarn-common ......... ............... SUCCESS [0.322 S]
50. [INFO] hadoop-yarn-server ......... ............... SUCCESS [0.065 S]
52. [INFO] Hadoop-yarn-server-common ......... ......... SUCCESS [0.972 S]
54. [INFO] Hadoop-yarn-server-nodemanager ......... ..... SUCCESS [0.580 S]
56. [INFO] hadoop-yarn-server-web-proxy ......... ....... SUCCESS [0.379 S]
58. [INFO] Hadoop-yarn-server-applicationhistoryservice ... SUCCESS [0.281 S]
60. [INFO] Hadoop-yarn-server-resourcemanager ....... ..... SUCCESS [0.378 S]
62. [INFO] hadoop-yarn-server-tests ......... .......... SUCCESS [0.534 S]
64. [INFO] hadoop-yarn-client ......... ............... SUCCESS [0.307 S]
66. [INFO] hadoop-yarn-applications ......... .......... SUCCESS [0.050 S]
68. [INFO] Hadoop-yarn-applications-distributedshell ..... SUCCESS [0.202 S]
70. [INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [0.194 S]
72. [INFO] hadoop-yarn-site .......... ................ SUCCESS [0.057 S]
74. [INFO] hadoop-yarn-project ......... .............. SUCCESS [0.066 S]
76. [INFO] hadoop-mapreduce-client ......... ........... SUCCESS [0.091 S]
78. [INFO] Hadoop-mapreduce-client-core ......... ....... SUCCESS [1.321 S]
80. [INFO] Hadoop-mapreduce-client-common ......... ..... SUCCESS [0.786 S]
82. [INFO] hadoop-mapreduce-client-shuffle ......... ..... SUCCESS [0.456 S]
84. [INFO] Hadoop-mapreduce-client-app ........ ......... SUCCESS [0.508 S]
86. [INFO] hadoop-mapreduce-client-hs ......... ......... SUCCESS [0.834 S]
88. [INFO] hadoop-mapreduce-client-jobclient ....... ..... SUCCESS [0.541 S]
90. [INFO] hadoop-mapreduce-client-hs-plugins ....... ..... SUCCESS [0.284 S]
92. [INFO] Apache Hadoop MapReduce Examples ........ ..... SUCCESS [0.851 S]
94. [INFO] hadoop-mapreduce .......... ................ SUCCESS [0.099 S]
96. [INFO] Apache Hadoop MapReduce streaming ....... ..... SUCCESS [0.742 S]
98. [INFO] Apache Hadoop distributed Copy ......... ..... SUCCESS [0.335 S]
100. [INFO] Apache Hadoop Archives ......... ............ SUCCESS [0.397 S]
102. [INFO] Apache Hadoop rumen ......... .............. SUCCESS [0.371 S]
104. [INFO] Apache Hadoop gridmix ......... ............. SUCCESS [0.230 S]
106. [INFO] Apache Hadoop Data Join ......... ........... SUCCESS [0.184 S]
108. [INFO] Apache Hadoop Extras ......... ............. SUCCESS [0.217 S]
110. [INFO] Apache Hadoop Pipes ......... .............. SUCCESS [0.048 S]
112. [INFO] Apache Hadoop OpenStack support ....... ....... SUCCESS [0.244 S]
114. [INFO] Apache Hadoop Client ......... ............. SUCCESS [0.590 S]
116. [INFO] Apache Hadoop mini-cluster ......... ......... SUCCESS [0.230 S]
118. [INFO] Apache Hadoop Scheduler Load Simulator ...... SUCCESS [0.650 S]
120. [INFO] Apache Hadoop Tools Dist ......... .......... SUCCESS [0.334 S]
122. [INFO] Apache Hadoop Tools ......... .............. SUCCESS [0.042 S]
124. [INFO] Apache Hadoop distribution ......... ......... SUCCESS [0.144 S]
126. [INFO]------------------------------------------------------------------------
128. [INFO] BUILD SUCCESS
130. [INFO]------------------------------------------------------------------------
132. [INFO] Total time:31.234 S
134. [INFO] Finished at:2014-06-23t14:55:08+08:00
136. [INFO] Final memory:84m/759m
138. [INFO]------------------------------------------------------------------------

Copy Code

At this time, we have to download the source code down. At this point, we will see a noticeable increase in the file size.


<ignore_js_op>



3. Associated Eclipse Source

Join our program below
<ignore_js_op>hadoop2.2mapreduce example. rar (1.14 MB, download number: 7, Price: 1 cloud currency)
As shown, they were packaged
<ignore_js_op>

These two files, maxtemperature.zip for the MapReduce example, Mockito-core-1.8.5.jar the package referenced by the MapReduce example
（here need to explain is, MapReduce is 2.2, but does not affect the associated source code, just give you how to associate the source code）
After we unzip, import eclipse
(For importing projects unfamiliar, refer to the 0 basics to teach you how to import Eclipse projects)

<ignore_js_op>


After we import, see a lot of red line, these are actually no reference package, below we begin to solve these grammatical problems.
First, resolve the import jar package
(1) Introduction of Mockito-core-1.8.5.jar

(2) hadoop2.4 the jar files in the compiled package, the locations of these files are as follows:

Hadoop_home in the Share\hadoop folder, specifically I this position D:\hadoop2\hadoop-2.4.0\share\hadoop
Find the jar package inside, for example: the jar package in the Lib file, and the following jar packages are added to the BuildPath.
If you do not know how to add these jar packages for the reference package, refer to the Hadoop development approach summary and instructions.
（Note that we are here to introduce a compiled package that compiles the download hadoop--642.4.0.tar.gz
Link: http://pan.baidu.com/s/1c0vPjG0 Password: xj6l)

More package downloads can be found in the Hadoop family, Strom, Spark, Linux, Flume and other jar packages, installation package summary download

Second, the relevant source code
1. After we import the jar package, there is no error, as shown in



2. Source code not found

When we want to see how a class or function is implemented, the source file is not found through open call Hierarchy.








3.Attach Source



The above three places, we add in order, we select the package, click OK, OK, our work is finished.

Note: for Hadoop-2.2.0-src.zip is we above through maven download source code, and then compress the file, remember must be compressed file Zip form


4. Check the source code after verifying the association

We perform the above operation again, via the open call Hierarchy




See what follows






Then we double-click the main class, The Scarlet Letter section, and we see the following:

>

Problem:
Careful classmate, we have a problem in this, because we see a. class file instead of a. java file. Then he will not be the same as what we see. java files that.
In fact, is the same, interested students, can be verified.


Next article:
How to view and read hadoop2.4 source code through eclipse