Hadoop Yarn (II)-create an Eclipse project, hadoopyarn
HamaWhite (QQ: 530422429) original works. For more information, see http://write.blog.csdn.net/postedit/40615469.
1. The installation environment is described as follows:
System: Ubuntu14.04
Hadoop version: hadoop-2.5.0 (click to download)
Java version: openjdk-1.7.0_55
Eclipse version: Release 4.4.0 (click to download)
2. Extract the downloaded Hadoop source package hadoop-2.5.0-src.tar.gz to the working directory (this article is/home/baisong ),
Get the hadoop-2.5.0-src folder.
3. Install Maven with the following command:
$ Sudo apt-get install maven
4. Install Google's open-source serialization Framework Protocol Buffers.
1) download the protobuf-2.5.0.tar.gz (click to download) and decompress it to get the protobuf-2.5.0 folder.
2) enter the protobuf-2.5.0, enter the subordinate command:
$./Configure
The following error is reported:
Analysis: this is a common Ubuntu error. You can download all the frequently used libraries. The command is as follows:
$ Sudo apt-get install build-essential
Then enter the $./configure command.
3) execute the following three commands in sequence.
$ Make
$ Make check
$ Sudo make install
4) Check the Protocol version number and whether the installation is successful.
$ Protoc -- version
The following error is reported:
The cause is that the system cannot find the lib library. Modify the/etc/ld. so. conf file and add/usr/local/lib. As follows:
$ Sudo gedit/etc/ld. so. conf
Content: include/etc/ld. so. conf. d/*. conf/usr/local/lib
Run the following command to update the library file path. Then you can run the $ protoc -- version command to view the version number.
$ Sudo ldconfig
5. Compile the Hadoop source code to generate the Jar package, without compiling native code, test cases and generate documents, in the/home/baisong/adoop-2.5.0-src directory, enter the following command:
$ Mvn package-Pdist-DskipTests-Dtar
After compilation is successful, the output information is as follows:
6. Generate the Eclipse project file. The command is as follows:
$ Cd hadoop-maven-plugins
$ Mvn install
$ Hadoop-2.5.0-src cd
$ Mvn eclipse: eclipse-DskipTests
The output information after success is as follows:
7. Import the source code in Eclipse: "File"> "Import"> "Existing Projects into Workspace", select the hadoop-2.5.0-src, and then confirm.
As shown in, there are 59 errors after import, but they are generally divided into three categories. The solution is as follows:
Error 1: org. apache. hadoop. ipc. protobuf. × cannot be parsed.
The solution is as follows: execute the following commands in sequence.
$ Cd hadoop-2.5.0-src/hadoop-common-project/hadoop-common/src/test/proto // Note: Go to this directory
$ Protoc -- java_out = ../java *. proto
Finally, refresh all projects in Eclipse. At this point, there are only 6 errors left.
Error 2: AvroRecord cannot be resolved to a type TestAvroSerialization. java
The solution is as follows:
First, download the avro-tools-1.7.4.jar (click to download ). Store in the hadoop-2.5.0-src directory.
Run the following command:
$ Cd hadoop-2.5.0-src/hadoop-common-project/hadoop-common/src/test/avro
$ Java-jar ~ /Hadoop-2.5.0-src/avro-tools-1.7.4.jar compile schema avroRecord. avsc ../java
Finally, refresh the Eclipse project. Only three errors are left.
Error 3: Project 'hadoop-streaming' is missing required source... Build Path Problem
The solution is as follows:
Right-click the hadoop-streaming project and choose properties> Java Build Path on the left> Source> selected error item> Remove on the right, as shown in.
So far, the Hadoop project has been created in Eclipse.
Using eclipse to call hadoop in win7, the following problems occur when creating a mapReduce project:
It is estimated that the JRE version for starting Eclipse is too low. Try to change to a later version of JRE.
Hadoop eclipse cannot create folders
In the log folder of the hadoop installation directory, if you have not modified the Log directory
