From zero teaches you how to get hadoop2.4 source code and use Eclipse to associate hadoop2.4 source code

Problem Guide:
1. How to get all the code for Hadoop via the official website src package

3.maven?






If we want to develop, researching the source code is very helpful to us. Do not understand the principle is like a black box, encounter problems, we can not touch the idea. So here you are.
I. How to obtain the source code
two. How to associate source code

How to get the source code

1. Download the maven package for Hadoop

(1) Download the website
Here we first download maven package hadoop-2.4.0-src.tar.gz from the official website.
Official website

For those who do not know how to download the official website, you can view: Beginner's Guide: Introduction to Hadoop website and how to download Hadoop (2.4) Versions and view Hadoop API introduction

(2) screen download
can also be downloaded from the network disk:
Http://pan.baidu.com/s/1kToPuGB

2. There are two ways to get the source code through MAVEN
, one way through the command line, and one through eclipse. This is mainly about the way of the command

Get the source code by command:
1. Unpacking the package

<ignore_js_op> &NBSP


The following problem was encountered while unpacking the package. But don't worry, let's go down

1: Unable to create file: D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\ Hadoop-yarn-server-applicationhistoryservice\target\classes\org\apache\hadoop\yarn\server\ Applicationhistoryservice\applicationhistoryclientservice$applicationhsclientprotocolhandler.class: 
The total length of the path and filename cannot exceed 260 characters 
The system could not find the path specified. D:\hadoop2\hadoop-2.4.0-src.zip 
2: Unable to create file: D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\ Hadoop-yarn-server-applicationhistoryservice\target\classes\org\apache\hadoop\yarn\server\        Applicationhistoryservice\timeline\leveldbtimelinestore$lockmap$countingreentrantlock.class: The system cannot find the path specified. D:\hadoop2\hadoop-2.4.0-src.zip 
3: Unable to create file: D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\ Hadoop-yarn-server-applicationhistoryservice\target\test-classes\org\apache\hadoop\yarn\server\        Applicationhistoryservice\webapp\testahswebapp$mockapplicationhistorymanagerimpl.class: The system cannot find the path specified. D:\hadoop2\hadoop-2.4.0-src.zip 
4: Unable to create file: D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\ Hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\monitor\ Capacity\testproportionalcapacitypreemptionpolicy$ispreemptionrequestfor.class: 
The total length of the path and filename cannot exceed 260 characters 
The system could not find the path specified. D:\hadoop2\hadoop-2.4.0-src.zip 
5: Unable to create file: D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\ Hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\        Testfsrmstatestore$testfsrmstatestoretester$testfilesystemrmstore.class: The system cannot find the path specified. D:\hadoop2\hadoop-2.4.0-src.zip 
6: Unable to create file: D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\ Hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\ Testzkrmstatestore$testzkrmstatestoretester$testzkrmstatestoreinternal.class: 
The total length of the path and filename cannot exceed 260 characters 
The system could not find the path specified. D:\hadoop2\hadoop-2.4.0-src.zip 
7: Unable to create file: D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\ Hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\ Testzkrmstatestorezkclientconnections$testzkclient$testforwardingwatcher.class: 
The total length of the path and filename cannot exceed 260 characters 
The system could not find the path specified. D:\hadoop2\hadoop-2.4.0-src.zip 
8: Unable to create file: D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\ Hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\ Testzkrmstatestorezkclientconnections$testzkclient$testzkrmstatestore.class: 
The total length of the path and filename cannot exceed 260 characters 
The system could not find the path specified. D:\hadoop2\hadoop-2.4.0-src.zip 
9: Unable to create file: D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\ Hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\rmapp\ Attempt\testrmappattempttransitions$testapplicationattempteventdispatcher.class: 
The total length of the path and filename cannot exceed 260 characters 
The system could not find the path specified. D:\hadoop2\hadoop-2.4.0-src.zip

2. Get the source code through MAVEN

It is necessary to note that when using MAVEN, you need to install JDK,PROTOC first, if not installed, you can refer to win7 How to install maven, install Protoc

(1) Enter Hadoop-2.4.0-src\hadoop-maven-plugins, run mvn install

1. D:\HADOOP2\HADOOP-2.4.0-SRC\HADOOP-MAVEN-PLUGINS>MVN Install

Copy Code

Displays the following information

1. [INFO] Scanning for projects ...

2. [WARNING]

3. [WARNING] Some problems were encountered while building the effective model for

4. org.apache.hadoop:hadoop-maven-plugins:maven-plugin:2.4.0

5. [WARNING] ' Build.plugins.plugin. (Groupid:artifactid) ' must is unique but found D

6. uplicate Declaration of Plugin Org.apache.maven.plugins:maven-enforcer-plugin @

7. org.apache.hadoop:hadoop-project:2.4.0, D:\hadoop2\hadoop-2.4.0-src\hadoop-proje

8. Ct\pom.xml, line 1015, column 15

9. [WARNING]

10. [WARNING] It is highly recommended to fix these problems because they threaten t

11. He stability of your build.

12. [WARNING]

13. [WARNING] For this reason, the future Maven versions might no longer support Buildin

14. G such malformed projects.

15. [WARNING]

16. [INFO]

17. [INFO] Using the builder Org.apache.maven.lifecycle.internal.builder.singlethrea

18. Ded. Singlethreadedbuilder with a thread count of 1

19. [INFO]

20. [INFO]------------------------------------------------------------------------

21. [INFO] Building Apache Hadoop Maven Plugins 2.4.0

22. [INFO]------------------------------------------------------------------------

23. [INFO]

24. [INFO]---maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-maven-plugins

25. ---

26. [INFO] Executing tasks

27. Main

28. [INFO] Executed Tasks

29. [INFO]

30. [INFO]---maven-plugin-plugin:3.0:descriptor (default-descriptor) @ hadoop-mave

31. N-plugins---

32. [INFO] Using ' UTF-8 ' encoding to read Mojo metadata.

33. [INFO] Applying Mojo extractor for language:java-annotations

34. [INFO] Mojo Extractor for language:java-annotations found 2 Mojo descriptors.

35. [INFO] Applying Mojo extractor for Language:java

36. [INFO] Mojo extractor for Language:java found 0 Mojo descriptors.

37. [INFO] Applying Mojo extractor for LANGUAGE:BSH

38. [INFO] Mojo extractor for LANGUAGE:BSH found 0 Mojo descriptors.

39. [INFO]

40. [INFO]---maven-resources-plugin:2.2:resources (default-resources) @ Hadoop-mav

41. En-plugins---

42. [INFO] Using default encoding to copy filtered resources.

43. [INFO]

44. [INFO]---maven-compiler-plugin:2.5.1:compile (default-compile) @ hadoop-maven-

45. Plugins---

46. [INFO] Nothing to Compile-all classes is up to date

47. [INFO]

48. [INFO]---maven-plugin-plugin:3.0:descriptor (mojo-descriptor) @ hadoop-maven-p

49. Lugins---

50. [INFO] Using ' UTF-8 ' encoding to read Mojo metadata.

51. [INFO] Applying Mojo extractor for language:java-annotations

52. [INFO] Mojo Extractor for language:java-annotations found 2 Mojo descriptors.

53. [INFO] Applying Mojo extractor for Language:java

54. [INFO] Mojo extractor for Language:java found 0 Mojo descriptors.

55. [INFO] Applying Mojo extractor for LANGUAGE:BSH

56. [INFO] Mojo extractor for LANGUAGE:BSH found 0 Mojo descriptors.

57. [INFO]

58. [INFO]---maven-resources-plugin:2.2:testresources (default-testresources) @ ha

59. Doop-maven-plugins---

60. [INFO] Using default encoding to copy filtered resources.

61. [INFO]

62. [INFO]---maven-compiler-plugin:2.5.1:testcompile (default-testcompile) @ Hadoo

63. P-maven-plugins---

64. [INFO] No sources to compile

65. [INFO]

66. [INFO]---maven-surefire-plugin:2.16:test (default-test) @ hadoop-maven-plugins

67. ---

68. [INFO] No tests to run.

69. [INFO]

70. [INFO]---maven-jar-plugin:2.3.1:jar (default-jar) @ hadoop-maven-plugins---

71. [INFO] Building Jar:d:\hadoop2\hadoop-2.4.0-src\hadoop-maven-plugins\target\had

72. Oop-maven-plugins-2.4.0.jar

73. [INFO]

74. [INFO]---maven-plugin-plugin:3.0:addpluginartifactmetadata (default-addplugina

75. rtifactmetadata) @ hadoop-maven-plugins---

76. [INFO]

77. [INFO]---maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hadoop-

78. Maven-plugins---

79. [INFO]

80. [INFO]---maven-install-plugin:2.3.1:install (default-install) @ hadoop-maven-p

81. Lugins---

82. [INFO] Installing D:\hadoop2\hadoop-2.4.0-src\hadoop-maven-plugins\target\hadoop

83. -maven-plugins-2.4.0.jar to C:\Users\hyj\.m2\repository\org\apache\hadoop\hadoop

84. -maven-plugins\2.4.0\hadoop-maven-plugins-2.4.0.jar

85. [INFO] Installing D:\hadoop2\hadoop-2.4.0-src\hadoop-maven-plugins\pom.xml to C:

86. \users\hyj\.m2\repository\org\apache\hadoop\hadoop-maven-plugins\2.4.0\hadoop-ma

87. Ven-plugins-2.4.0.pom

88. [INFO]------------------------------------------------------------------------

89. [INFO] BUILD SUCCESS

90. [INFO]------------------------------------------------------------------------

91. [INFO] Total time:4.891 S

92. [INFO] Finished at:2014-06-23t14:47:33+08:00

93. [INFO] Final memory:21m/347m

94. [INFO]------------------------------------------------------------------------

95. 
    

Copy Code

Sections are as follows:

<ignore_js_op>


<ignore_js_op>



(2) operation

1. MVN eclipse:eclipse-dskiptests

Copy Code

Now, notice that we're in hadoop_home, and I'm D:\hadoop2\hadoop-2.4.0-src here.

Some of the information below

2. [INFO]
4. [INFO]------------------------------------------------------------------------
6. [INFO] Reactor Summary:
8. [INFO]
10. [INFO] Apache Hadoop Main ......... ............... SUCCESS [0.684 S]
12. [INFO] Apache Hadoop Project POM ......... ......... SUCCESS [0.720 S]
14. [INFO] Apache Hadoop Annotations ......... .......... SUCCESS [0.276 S]
16. [INFO] Apache Hadoop Project Dist POM ......... ..... SUCCESS [0.179 S]
18. [INFO] Apache Hadoop assemblies ......... .......... SUCCESS [0.121 S]
20. [INFO] Apache Hadoop Maven Plugins ........ ......... SUCCESS [1.680 S]
22. [INFO] Apache Hadoop minikdc ......... ............. SUCCESS [1.802 S]
24. [INFO] Apache Hadoop Auth ......... ............... SUCCESS [1.024 S]
26. [INFO] Apache Hadoop Auth Examples ........ ......... SUCCESS [0.160 S]
28. [INFO] Apache Hadoop Common ......... ............. SUCCESS [1.061 S]
30. [INFO] Apache Hadoop NFS ......... ................ SUCCESS [0.489 S]
32. [INFO] Apache Hadoop Common Project ......... ....... SUCCESS [0.056 S]
34. [INFO] Apache Hadoop HDFS ......... ............... SUCCESS [2.770 S]
36. [INFO] Apache Hadoop Httpfs ......... ............. SUCCESS [0.965 S]
38. [INFO] Apache Hadoop HDFS bookkeeper Journal .... ..... SUCCESS [0.629 S]
40. [INFO] Apache Hadoop Hdfs-nfs ......... ............ SUCCESS [0.284 S]
42. [INFO] Apache Hadoop HDFS Project ......... ......... SUCCESS [0.061 S]
44. [INFO] Hadoop-yarn ..... ....... ................... SUCCESS [0.052 S]
46. [INFO] Hadoop-yarn-api .......... ................. SUCCESS [0.842 S]
48. [INFO] Hadoop-yarn-common ......... ............... SUCCESS [0.322 S]
50. [INFO] hadoop-yarn-server ......... ............... SUCCESS [0.065 S]
52. [INFO] Hadoop-yarn-server-common ......... ......... SUCCESS [0.972 S]
54. [INFO] Hadoop-yarn-server-nodemanager ......... ..... SUCCESS [0.580 S]
56. [INFO] hadoop-yarn-server-web-proxy ......... ....... SUCCESS [0.379 S]
58. [INFO] Hadoop-yarn-server-applicationhistoryservice ... SUCCESS [0.281 S]
60. [INFO] Hadoop-yarn-server-resourcemanager ....... ..... SUCCESS [0.378 S]
62. [INFO] hadoop-yarn-server-tests ......... .......... SUCCESS [0.534 S]
64. [INFO] hadoop-yarn-client ......... ............... SUCCESS [0.307 S]
66. [INFO] hadoop-yarn-applications ......... .......... SUCCESS [0.050 S]
68. [INFO] Hadoop-yarn-applications-distributedshell ..... SUCCESS [0.202 S]
70. [INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [0.194 S]
72. [INFO] hadoop-yarn-site .......... ................ SUCCESS [0.057 S]
74. [INFO] hadoop-yarn-project ......... .............. SUCCESS [0.066 S]
76. [INFO] hadoop-mapreduce-client ......... ........... SUCCESS [0.091 S]
78. [INFO] Hadoop-mapreduce-client-core ......... ....... SUCCESS [1.321 S]
80. [INFO] Hadoop-mapreduce-client-common ......... ..... SUCCESS [0.786 S]
82. [INFO] hadoop-mapreduce-client-shuffle ......... ..... SUCCESS [0.456 S]
84. [INFO] Hadoop-mapreduce-client-app ........ ......... SUCCESS [0.508 S]
86. [INFO] hadoop-mapreduce-client-hs ......... ......... SUCCESS [0.834 S]
88. [INFO] hadoop-mapreduce-client-jobclient ....... ..... SUCCESS [0.541 S]
90. [INFO] hadoop-mapreduce-client-hs-plugins ....... ..... SUCCESS [0.284 S]
92. [INFO] Apache Hadoop MapReduce Examples ........ ..... SUCCESS [0.851 S]
94. [INFO] hadoop-mapreduce .......... ................ SUCCESS [0.099 S]
96. [INFO] Apache Hadoop MapReduce streaming ....... ..... SUCCESS [0.742 S]
98. [INFO] Apache Hadoop distributed Copy ......... ..... SUCCESS [0.335 S]
100. [INFO] Apache Hadoop Archives ......... ............ SUCCESS [0.397 S]
102. [INFO] Apache Hadoop rumen ......... .............. SUCCESS [0.371 S]
104. [INFO] Apache Hadoop gridmix ......... ............. SUCCESS [0.230 S]
106. [INFO] Apache Hadoop Data Join ......... ........... SUCCESS [0.184 S]
108. [INFO] Apache Hadoop Extras ......... ............. SUCCESS [0.217 S]
110. [INFO] Apache Hadoop Pipes ......... .............. SUCCESS [0.048 S]
112. [INFO] Apache Hadoop OpenStack support ....... ....... SUCCESS [0.244 S]
114. [INFO] Apache Hadoop Client ......... ............. SUCCESS [0.590 S]
116. [INFO] Apache Hadoop mini-cluster ......... ......... SUCCESS [0.230 S]
118. [INFO] Apache Hadoop Scheduler Load Simulator ...... SUCCESS [0.650 S]
120. [INFO] Apache Hadoop Tools Dist ......... .......... SUCCESS [0.334 S]
122. [INFO] Apache Hadoop Tools ......... .............. SUCCESS [0.042 S]
124. [INFO] Apache Hadoop distribution ......... ......... SUCCESS [0.144 S]
126. [INFO]------------------------------------------------------------------------
128. [INFO] BUILD SUCCESS
130. [INFO]------------------------------------------------------------------------
132. [INFO] Total time:31.234 S
134. [INFO] Finished at:2014-06-23t14:55:08+08:00
136. [INFO] Final memory:84m/759m
138. [INFO]------------------------------------------------------------------------

Copy Code

At this time, we have to download the source code down. At this point, we will see a noticeable increase in the file size.


<ignore_js_op>



3. Associated Eclipse Source

Join our program below
<ignore_js_op>hadoop2.2mapreduce example. rar (1.14 MB, download number: 400, Price: 1 cloud currency)
As shown, they were packaged
<ignore_js_op>

These two files, maxtemperature.zip for the MapReduce example, Mockito-core-1.8.5.jar the package referenced by the MapReduce example
（here need to explain is, MapReduce is 2.2, but does not affect the associated source code, just give you how to associate the source code）
After we unzip, import eclipse
(For importing projects unfamiliar, refer to the 0 basics to teach you how to import Eclipse projects)

<ignore_js_op>


After we import, see a lot of red line, these are actually no reference package, below we begin to solve these grammatical problems.
First, resolve the import jar package
(1) Introduction of Mockito-core-1.8.5.jar

(2) hadoop2.4 the jar files in the compiled package, the locations of these files are as follows:

Hadoop_home in the Share\hadoop folder, specifically I this position D:\hadoop2\hadoop-2.4.0\share\hadoop
Find the jar package inside, for example: the jar package in the Lib file, and the following jar packages are added to the BuildPath.
If you do not know how to add these jar packages for the reference package, refer to the Hadoop development approach summary and instructions.
（Note that we are here to introduce a compiled package that compiles the download hadoop--642.4.0.tar.gz
Link: http://pan.baidu.com/s/1c0vPjG0 Password: xj6l)

More package downloads can be found in the Hadoop family, Strom, Spark, Linux, Flume and other jar packages, installation package summary download

<ignore_js_op>



<ignore_js_op>


Second, the relevant source code
1. After we import the jar package, there is no error, as shown in

<ignore_js_op>

2. Source code not found

When we want to see how a class or function is implemented, the source file is not found through open call Hierarchy.

<ignore_js_op>



<ignore_js_op>


3.Attach Source

<ignore_js_op>

Three of the above, we add in order, after we select the compressed package, clickOK, OK, our work is finished.

Note:For Hadoop-2.2.0-src.zip is the source we downloaded through MAVEN, and then compressed the file,Remember it must be compressed file Zip form


4. Check the source code after verifying the association

We perform the above operation again, via the open call Hierarchy




See what follows



<ignore_js_op>


Then we double-click the main class, The Scarlet Letter section, and we see the following:

<ignore_js_op>


Questions:
Careful classmate, we have a problem in this, because we see a. class file instead of a. java file. Then he will not be the same as what we see. java files that.
In fact, is the same, interested students, can be verified.

From zero teaches you how to get hadoop2.4 source code and use Eclipse to associate hadoop2.4 source code