Summary of hadoop running in eclipse in Linux

Reprinted please indicate the original author: http://www.cnblogs.com/beanmoon

Recently, I was studying big data and would inevitably use hadoop. So I started to read hadoop: the definite Guide 2 and met an instance when analyzing weather data, then try to run the command.

% Export hadoop_classpath = build/classes
% Hadoop maxtemperature input/ncdc/sample.txt output

Hadoop_classpath is used to specify the location of the compiled class file (I compiled it through eclipse, but I don't know how the author compiled it. Directly Using javac will report a lot of error information, of course, this problem is to be solved because no additional jar package is added ~~ Export _^). If you do not specify hadoop_classpath, even if you execute the hadoop command in the current folder, the following error occurs: cocould not find or load main class maxtemperature. You can also use "jar CMF manifest maxtemperature. jar *. class "command to generate a jar package, and then use" hadoop jar maxtemperature. jar input output "command to run the jar package (hadoop_classpath is not specified at this time). If the main-class is not specified through the manifest file during packaging, add the declaration of the main class during execution: "hadoop jar maxtemperature. jarMaxtemperatureInput Output ". The content of the manifest file is shown below:

Manifest-version: 1.0
Main-class: maxtemperature

If eclipse is used, install eclipse first. In Debian, the first installation is via aptitude install eclipse (the installation location is/usr/lib/eclipse ), the installed version is from the Galileo version. If it is too old, the Juno version is released. The installation process is as follows:

(1)extract eclipse-java-juno-sr2-linux-gtk.tar.gz to a directory. I decompress it to the main directory and obtain the eclipse directory.

(2) create a STARTUP script eclipse in the/usr/bin directory and execute the following command to create it:
Sudo gedit/usr/bin/eclipse

Add the following content to the file:
#! /Bin/sh
Export export illa_five_home = "/usr/lib/Mozilla /"
Export eclipse_home = "/home/wangxing/development/Eclipse"
$ Eclipse_home/eclipse $ *($ * What ??)

(3) modify the permission of the script to make it executable and execute the following command:
Sudo chmod A + x/usr/bin/eclipse

(4) add an icon on the applications menu

Sudo gedit/usr/share/applications/eclipse. Desktop

Add the following content to the file:
[Desktop entry]
Encoding = UTF-8
Name = eclipse Platform
Comment = Eclipse IDE
Exec = eclipse
Icon =/home/bean/Eclipse/icon. XPM
Terminal = false
Startupnotify = true
Type = Application
Categories = application; development;

It is good to start eclipse through the terminal, but it won't work through the mouse. the following error is reported:

A Java Runtime Environment (JRE) or Java Development Kit (JDK) must be available in order to run eclipse. No Java virtual machine was found after searching the following locations :...

The solution is as follows:

Enter your eclipse directory on the terminal and enter:

Mkdir JRE
CD JRE
Ln-s your JDK directory/Bin

Now we start to integrate hadoop into eclipse. The method found on the internet is as follows:

Http://blog.sina.com.cn/s/blog_62186b4601012acs.html

This is the solution to the Internet, but specific to me when the execution of the problem, first of all, my hadoop is the latest version of 1.0.4, it does not directly give you hadoop-eclipse-plugin-1.0.4.jar, it requires you to export it by yourself. The online method is as follows:

1. In eclipse, select file --> Import --> existing projects into workspace to import existing projects. Assume that the path is:

/Download/hadoop-1.0.1/src/contrib/eclipse-plugin, the default project name is mapreducetools

2. The default project name is mapreducetools, and then the lib directory is created in the project mapreducetools, and copy hadoop-core, commons-cli-1.2.jar, commons-lang-2.4.jar, commons-configuration-1.6.jar, jackson-mapper-asl-1.8.8.jar, jackson-core-asl-1.8.8.jar, commons-httpclient-3.0.1.jar, to this directory.

3. Modify ../build-contrib.xml (is this the last layer that is stored in the location of the project ??)

<Property name = "version" value = "1.0.1"/>
<Property name = "hadoop. Root" location = "/download/develop/hadoop-1.0.1"/>
<Property name = "Eclipse. Home" location = "/download/Eclipse"/>

4. Modify build. xml

<Target name = "jar" depends = "compile" Unless = "Skip. contrib">

<Mkdir dir = "$ {build. dir}/lib"/>
<Copy file = "$ {hadoop. root}/hadoop-core-$ {version }. jar "tofile =" $ {build. dir}/lib/hadoop-core.jar "verbose =" true "/>
<Copy file = "$ {hadoop. Root}/lib/commons-cli-1.2.jar" todir = "$ {build. dir}/lib" verbose = "true"/>
<Copy file = "$ {hadoop. Root}/lib/commons-lang-2.4.jar" todir = "$ {build. dir}/lib" verbose = "true"/>
<Copy file = "$ {hadoop. Root}/lib/commons-configuration-1.6.jar" todir = "$ {build. dir}/lib" verbose = "true"/>
<Copy file = "$ {hadoop. Root}/lib/jackson-mapper-asl-1.8.8.jar" todir = "$ {build. dir}/lib" verbose = "true"/>
<Copy file = "$ {hadoop. Root}/lib/jackson-core-asl-1.8.8.jar" todir = "$ {build. dir}/lib" verbose = "true"/>
<Copy file = "$ {hadoop. Root}/lib/commons-httpclient-3.0.1.jar" todir = "$ {build. dir}/lib" verbose = "true"/>
<Jar
Jarfile = "$ {build. dir}/hadoop-$ {name}-$ {version}. Jar"
Manifest = "$ {root}/META-INF/manifest. MF">
<Fileset dir = "$ {build. dir}" includes = "classes/lib/"/>
<Fileset dir = "$ {root}" includes = "Resources/plugin. xml"/>
</Jar>
</Target>

5. Right-click the project --> export, select JAR file for the type, select the plugin. xml file, classes directory, lib directory and resources directory, and select the exported jar directory to export the hadoop-eclipse-plugin-1.0.0.jar file.

6. Put the file under the eclipse/plugin directory and restart eclipse.

Finally, the jar package was exported step by step (because the above process was not well viewed, many errors were encountered in the middle and the jar package could not be exported, which wasted a lot of time ), it is always impossible for eclipse to identify hadoop MAP/reduce in Windows-> preference. Should it be wrong to export my jar package, so I downloaded such a jar package online and finally got my wish (during this period I also thought about using other early versions of hadoop, because they have built-in hadoop-eclipse-plugin -***. jar, I'm under the hadoop-0.22.0 version, but many of the classes and interfaces are deprecated for insecure reasons, so I have to re-use the hadoop-1.0.4 ).

Second, another problem occurs when configuring the hadoop location. Right-click the hadoop location and choose not to respond when creating or editing the hadoop location. If you try it several times, you will not try it. Instead, you can directly create a map/reduce project, in maxtemperature. in Java, right-click the run configuration and specify the parameters of the input file output file (the input and output files are placed in the root directory of the project) to start running, then a window pops up asking me to configure hadoop location ~~ It's really cheap ~~

The output content is the same as the original text ~~

Reference link:

Http://hi.baidu.com/ihsauqaxblbdmwq/item/9ddaf2fca32315b230c199a7
Http://blog.csdn.net/lmc_wy/article/details/6053580
Http://blog.sina.com.cn/s/blog_62186b4601012acs.html
Http://blog.sina.com.cn/s/blog_62186b4601011c6a.html
Http://www.oschina.net/code/snippet_176897_7976
Http://blog.sina.com.cn/s/blog_6daadb2f01015rb7.html