Hadoop Development Environment Building

Source: Internet
Author: User
Tags echo message

Before a little bit about the knowledge of big data, in the environment when I was very melancholy, because it did not fix, just this few days have time, so the previous did not fix it again to configure a bit, did not think incredibly successful, the process went a lot of detours, access to a large number of online information, Finally, I came to tidy up a bit.

Hadoop is divided into 3 types of installation modes, single-machine mode, pseudo-distribution mode and fully distributed mode. As we all know, the fully distributed mode is the real development of enterprises, will require more than one computer, there is no conditions, so do not explain this aspect. Pseudo-distribution mode is can be more than a few virtual machine folder can be, and then various configurations, here I mainly want to big data analysis, temporarily do not consider operations, so I only use a single-machine mode. This article describes running a pseudo-distributed configuration.


First, configure Hadoop

First go to github download Source: Https://github.com/winghc/hadoop2x-eclipse-plugin

Install the JDK in Linux and configure SSH password-free login, in fact, the first I did not configure password-free, and later found that password-free landing really convenient. Password-free login is on the console input: Ssh-keygen and then all the way to the return.

I unpacked Hadoop and placed it on/home/admin1/to download/hadoop-2.5.2. We mainly configure the file in Etc/hadoop.

1, Core-site.xml

<configuration>   <property>      <name>fs.default.name</name>      <value>hdfs ://localhost:9000</value>   </property></configuration>
2, Hdfs-site.xml
<configuration>  <property>     <name>dfs.replication</name>     <value>1< /value>  </property></configuration>

3, Mapred-site.xml.template

<configuration>   <property>      <name>mapred.job.tracker</name>      <value> Localhost:9001</value>   </property></configuration>

Also need to configure Java_home: in the hadoop-env.sh: (just change to your absolute path on it, do not use relative path).

Export JAVA_HOME=/USR/LIB/JVM/JAVA-1.7.0-OPENJDK-AMD64   
If you run an error, it should be that Java_home is not configured

Input with command line: sudo gedit/etc/profile

At the very end, add:


This should basically be able to run:

Starting mode: Used in/home/admin1/download/hadoop-2.5.2:
Bin/hadoop Namenode-format

sbin/start-all.sh

The Access address is:
http://localhost:50070/



Http://localhost:8088/cluster


If you want to close: sbin/stop-all.sh


Problem:

If Hadoop is an error, configure the Hadoop command:
Export hadoop_home=/home/admin1/Download/hadoop-2.5.2
Export path= $HADOOP _home/bin: $HADOOP _home/sbin: $PATH


If Datanode is not started
rm-rf/usr/hadoop/tmp/*
rm-rf/tmp/hadoop*
Do the above two steps before 1.


Second, configure Eclipse

Because I was installing eclipse directly in Linux, I used the 4.5 version, and then there were all sorts of weird problems. I first thought it was Hadoop-eclipse-plugin-2.5.2.jar, and then I went down. It should be the Ant and Hadoop2x-eclipse-plugin-master to recompile, the method of compiling is basically this:

1, first to download Hadoop2x-eclipse-plugin-master,https://github.com/winghc/hadoop2x-eclipse-plugin

2. Download ant:http://ant.apache.org/bindownload.cgi

3, respectively decompression out, if you are in win under the need to configure the environment variables, under Linux do not configure is actually can be run. Into the/hadoop2x-eclipse-plugin-master/src/contrib/eclipse-plugin you just unzipped.

4, under which directory, the following files are modified:

In Vxy: libraries.properties

Inside the version remember to and your direct version of the corresponding, if you do not find your version can go to your Hadoop file Share/hadoop in the search, in fact, there is a very simple way, you can directly copy my code in the past, and so on when compiling reports which jar package version is wrong, Then you can change it according to the error message.

# Licensed under the Apache License, Version 2.0 (the "License"); # you are not a use of this file except in compliance with The license.# obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## unless req Uired by applicable law or agreed to in writing, software# distributed under the License are distributed on a "as is" BA sis,# without warranties or CONDITIONS of any KIND, either express or implied.# see the License for the specific Langu Age governing permissions and# limitations under the License. #This Properties File lists the versions of the various artifacts used by Hadoop and components. #It drives Ivy and the Gen Eration of a Maven pom# this was the version of Hadoop we are generatinghadoop.version=2.5.2hadoop-gpl-compression.version= 0.1.0 #These is the versions of our dependencies (in alphabetical order) apacheant.version=1.7.0ant-task.version=2.0.10 a sm.version=3.2aspectj.version=1.6.5aspectj.version=1.6.11 checkstyle.version=4.2 Commons-cli.version=1.2commons-codec.version=1.4commons-collections.version=3.2.1commons-configuration.version= 1.6commons-daemon.version=1.0.13commons-httpclient.version=3.1commons-lang.version=2.6commons-logging.version= 1.1.3commons-logging-api.version=1.0.4commons-math.version=3.1.1commons-el.version= 1.0commons-fileupload.version=1.2commons-io.version=2.4commons-net.version=3.1core.version= 3.1.1coreplugin.version=1.3.2 hsqldb.version=1.8.0.10 ivy.version=2.1.0 jasper.version=5.5.12jackson.version= 1.9.13#not able to figureout the version of JSPs & Jsp-api version to get it resolved throught ivy# but still declared Here as we is going to has a local copy from the Lib folderjsp.version=2.1jsp-api.version=5.5.12jsp-api-2.1.version=6.1. 14jsp-2.1.version=6.1.14jets3t.version=0.6.1jetty.version=6.1.26jetty-util.version=6.1.26jersey-core.version= 1.9jersey-json.version=1.9jersey-server.version=1.9junit.version=4.11jdeb.version=0.8jdiff.version= 1.0.9json.version=1.0 kfs.version=0.1 log4j.version=1.2.17lucene-core.version=2.3.1 mockito-all.version=1.8.5jsch.version=0.1.42 oro.version=2.0.8 rats-lib.version=0.5.1 servlet.version=4.0.6servlet-api.version=2.5slf4j-api.version=1.7.5slf4j-log4j12.version =1.7.5 wagon-http.version=1.0-beta-2xmlenc.version=0.52xerces.version=1.4.4protobuf.version=2.5.0guava.version= 11.0.2netty.version=3.6.2.final

Build.xml

This is also the same, if your version of the wrong can be based on the above reason. XML file header remember to write shelf.

<?xml version= "1.0" encoding= "UTF-8" standalone= "no"?> <!--Licensed to the Apache software Fo  Undation (ASF) under one or more contributor license agreements.         See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License");  You are not a use of this file except in compliance with the License. Obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 unless required by applicable law or agreed to in writing, software distributed under the License are distributed on an "A"         S is ' BASIS, without warranties or CONDITIONS of any KIND, either express or implied.      See the License for the specific language governing permissions and limitations under the License. --<project default= "jar" Name= "ecliPse-plugin "> <import file=". /build-contrib.xml "/> <path id=" Eclipse-sdk-jars "> <fileset dir=" ${eclipse.home}/plugins            /"> <include name=" Org.eclipse.ui*.jar "/> <include name=" Org.eclipse.jdt*.jar "/>            <include name= "Org.eclipse.core*.jar"/> <include name= "Org.eclipse.equinox*.jar"/> <include name= "Org.eclipse.debug*.jar"/> <include name= "Org.eclipse.osgi*.jar"/> <i Nclude name= "Org.eclipse.swt*.jar"/> <include name= "Org.eclipse.jface*.jar"/> <inc Lude name= "Org.eclipse.team.cvs.ssh2*.jar"/> <include name= "Com.jcraft.jsch*.jar"/> </file set> </path> <path id= "Hadoop-sdk-jars" > <fileset dir= "${hadoop.home}/shar  E/hadoop/mapreduce "> <include name=" Hadoop*.jar "/> </fileset>         <fileset dir= "${hadoop.home}/share/hadoop/hdfs" > <include name= "Hadoop*.jar"/> &L t;/fileset> <fileset dir= "${hadoop.home}/share/hadoop/common" > <include name= "Hadoop*.jar "/> </fileset> </path> <!--Override classpath to include ECLI          PSE SDK Jars--<path id= "Classpath" > <pathelement location= "${build.classes}"/>          <!--pathelement location= "${hadoop.root}/build/classes"/--> <path refid= "Eclipse-sdk-jars"/> <path refid= "Hadoop-sdk-jars"/> </path> <!--Skip Building if Eclipse.home is unset. --<target name= "Check-contrib" unless= "Eclipse.home" > <property name= "skip.contrib" value= "y Es "/> <echo message=" eclipse.home unset:skipping Eclipse plugin "/> </target> & lt;! --<target Name= "CoMpile "depends=" init, Ivy-retrieve-common "unless=" Skip.contrib ">--> <!--here to remove depends=" init, ivy-retrieve- Common "-<target name=" compile "unless=" Skip.contrib "> <echo message=" contrib: ${name} "/&gt          ; <javac encoding= "${build.encoding}" srcdir= "${src.dir}" includes= "**/*.java" de Stdir= "${build.classes}" debug= "${javac.debug}" deprecation= "${javac.deprecation}" > <c  Lasspath refid= "Classpath"/> </javac> </target> <!--Override jar Target to Specify manifest--<target name= "jar" depends= "compile" unless= "Skip.contrib" > <mkdir dir= " ${build.dir}/lib "/> <copy todir=" ${build.dir}/lib/"verbose=" true "> <fileset dir=" ${h Adoop.home}/share/hadoop/mapreduce "> <include name=" Hadoop*.jar "/> </fileset&gt          ; </copy> <copy todir= "${build.dir}/lib/" verbose= "true" > <fileset dir= "${hadoop.home} /share/hadoop/common "> <include name=" Hadoop*.jar "/> </fileset> &lt ;/copy> <copy todir= "${build.dir}/lib/" verbose= "true" > <fileset dir= "${hadoop.home}/ Share/hadoop/hdfs "> <include name=" Hadoop*.jar "/> </fileset> </c opy> <copy todir= "${build.dir}/lib/" verbose= "true" > <fileset dir= "${hadoop.home}/sha Re/hadoop/yarn "> <include name=" Hadoop*.jar "/> </fileset> </copy > <copy todir= "${build.dir}/classes" verbose= "true" > <fileset dir= "${root}/src/                            Java "> <include name=" *.xml "/> </fileset> </copy> <copy File= "${hadoop.home}/share/hadoop/common/lib/protobuf-java-${protobuf.version}.jar" todir= "${build.dir}/lib" Verbose= "true"/> <copy file= "${hadoop.home}/share/hadoop/common/lib/log4j-${log4j.version}.jar" todir= "${b Uild.dir}/lib "verbose=" true "/> <copy file=" ${hadoop.home}/share/hadoop/common/lib/commons-cli-${commons-cl I.version}.jar "todir=" ${build.dir}/lib "verbose=" true "/> <copy file=" ${hadoop.home}/share/hadoop/common/l Ib/commons-configuration-${commons-configuration.version}.jar "todir=" ${build.dir}/lib "verbose=" true "/> < Copy file= "${hadoop.home}/share/hadoop/common/lib/commons-lang-${commons-lang.version}.jar" todir= "${build.dir}/ Lib "verbose=" true "/> <!--here to increase commons-collections dependency--<copy file=" ${hadoop.home}/share/h Adoop/common/lib/commons-collections-${commons-collections.version}.jar "todir=" ${build.dir}/lib "verbose=" true " /> <copy file= "${hadoop.home}/share/haDoop/common/lib/jackson-core-asl-${jackson.version}.jar "todir=" ${build.dir}/lib "verbose=" true "/> <copy F Ile= "${hadoop.home}/share/hadoop/common/lib/jackson-mapper-asl-${jackson.version}.jar" todir= "${build.dir}/lib" Verbose= "true"/> <copy file= "${hadoop.home}/share/hadoop/common/lib/slf4j-log4j12-${slf4j-log4j12.version} . Jar "todir=" ${build.dir}/lib "verbose=" true "/> <copy file=" ${hadoop.home}/share/hadoop/common/lib/slf4j-a Pi-${slf4j-api.version}.jar "todir=" ${build.dir}/lib "verbose=" true "/> <copy file=" ${hadoop.home}/share/ha Doop/common/lib/guava-${guava.version}.jar "todir=" ${build.dir}/lib "verbose=" true "/> <copy file=" ${hadoop . Home}/share/hadoop/common/lib/hadoop-auth-${hadoop.version}.jar "todir=" ${build.dir}/lib "verbose=" true "/> & Lt;copy file= "${hadoop.home}/share/hadoop/common/lib/commons-cli-${commons-cli.version}.jar" todir= "${build.dir} /lib "verbose=" true "/> <copy File= "${hadoop.home}/share/hadoop/common/lib/netty-${netty.version}.jar" todir= "${build.dir}/lib" verbose= "true" /> <jar jarfile= "${build.dir}/hadoop-${name}-${version}.jar" manifest= "${root}/ Meta-inf/manifest.        MF "> <manifest> <attribute name=" Bundle-classpath "value=" classes/, Lib/hadoop-mapreduce-client-core-${hadoop.version}.jar, lib/hadoop-mapreduce-client-common-${hadoop.version}.       Jar, Lib/hadoop-mapreduce-client-jobclient-${hadoop.version}.jar, Lib/hadoop-auth-${hadoop.version}.jar, Lib/hadoop-common-${hadoop.version}.jar, Lib/hadoop-hdfs-${hadoop.version}.jar, Lib/protobuf-java-${protobuf. Version}.jar, Lib/log4j-${log4j.version}.jar, Lib/commons-cli-${commons-cli.version}.jar,lib/commons-configura tion-${commons-configuration.version}.jar,lib/commons-httpclient-${commons-httpclient.version}.jar,lib/ Commons-lang-${commons-lang.version}.jar, lib/jackson-core-asl-${jackson.version},lib/jackson-mapper-asl-${jackson.version}.jar,lib/slf4j-log4j12-${ Slf4j-log4j12.version}.jar,lib/slf4j-api-${slf4j-api.version}.jar, Lib/guava-${guava.version}.jar, Lib/netty-${n Etty.version}.jar "/> </manifest> <fileset dir=" ${build.dir} "includes=" classes/lib/"/&gt            ; <!--fileset dir= "${build.dir}" includes= "*.xml"/--> <fileset dir= "${root}" includes= "Resources/plugin   . xml "/> </jar> </target> </project>

makeplus.sh

Ant jar-dversion=2.5.2-declipse.home=/home/admin1/public/eclipse     -dhadoop.home=/home/admin1/Download/hadoop-2.5.2

Description: Declipse is your installation directory for Eclipse, and Dhadoop is your Hadoop installation directory.

Then run./makeplus.sh can be compiled. Compile it and put it in Eclipse's plugins, then restart Eclipse.


Well, speaking of my inner breakdown here, the process desk complex has a wood, and finally I restarted eclipse unexpectedly did not succeed, and I found that so long was the cause of eclipse, my heart is almost 10,000 xxx, then I put 4.5 version of the deletion, replaced by 4.4 of the can. Of course there is a small episode, that is, the configuration can not right-click New Hadoop, so I entered the command line:

./eclipse-clean-consolelog-debug

And then restart it without a problem.

Now look at the Run configuration:


You can see a few folders I just created on the browser:



This is completely done, then you can happily start the development of Hadoop.

Summary: In learning a new thing, the first step is very difficult to step out of the difficult, and even let us very painful and irritable, but believe it, tired can rest, and then continue to complete, I in this environment when also took nearly 2 days, in the middle of a lot of detours, tried a lot of methods, I also tried to build multiple virtual machines to do a pseudo-distributed environment, but I finally found out that I did not want to do a pseudo-distributed environment. Online information is very rich, to be good at finding something useful for themselves, do not blindly follow the online code to run, to have their own ideas, believe it, the end will be successful! Not to do, only unexpected!




Hadoop Development Environment Building

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.