windows7+eclipse-jee-luna-sr1a-win32-x86_64+hadoop2.6 Environment Configuration

First, the Hadoop cluster environment configuration

See also: "Hadoop2.6 cluster environment Construction (HDFS Ha+yarn) original 4G memory can be capricious once."

WIN7 Environment:

Login Username: Hadoop, unified with the Linux environment users in the Hadoop cluster.

A. Modify the Hdfs-site.xml on the SY-0130 node plus the following, and synchronize the modifications to the other nodes.
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>

Designed to remove permission checks because I configured the Map/reduce connection to report the following error when I configured Eclipse to connect to the Hadoop server on my Windows machine. Org.apache.hadoop.security.AccessControlException:Permission denied.

Restart the Hadoop cluster.

Two. Windows Basic Environment Preparation

Windows7 (x64), JDK (64-bit), Ant,eclipse,hadoop

1.JDK Environment Configuration
Jdk-6u45-windows-x64.exe Configure the relevant JAVA_HOME environment variable after installation and configure the bin directory to path. Do not use JDK7 or JDK8 here, follow the instructions why.

: Http://pan.baidu.com/s/1kTn9mIj

2.eclipse Environment Configuration
Eclipse-jee-luna-sr1a-win32-x86_64.zip Extract to D:\hadoop2.6\eclipse-jee-luna-SR1a-win32-x86_64
: Http://pan.baidu.com/s/1dD5wv0P
3.ant Environment Configuration
Apache-ant-1.9.4-bin.zip extract to D:\hadoop2.6\apache-ant-1.9.4, configure the environment variable Ant_home, and configure the bin directory to path
: Http://mirror.bit.edu.cn/apache//ant/binaries/apache-ant-1.9.4-bin.zip

4. Download hadoop-2.6.0.tar.gz
Http://pan.baidu.com/s/1kTpBenX

5. Download hadoop-2.6.0-src.tar.gz
Http://pan.baidu.com/s/1o6yYzma

6. Download Hadoop2x-eclipse-plugin-master.zip
Https://github.com/winghc/hadoop2x-eclipse-plugin

I tested the version: http://pan.baidu.com/s/12KD3o

7. Download Hadoop-common-2.2.0-bin-master.zip
Https://github.com/srccodes/hadoop-common-2.2.0-bin

The version I downloaded when I tested: Http://pan.baidu.com/s/1o6HptNW

respectively, hadoop-2.6.0.tar.gz, hadoop-2.6.0-src.tar.gz, Hadoop2x-eclipse-plugin-master.zip, Hadoop-common-2.2.0-bin-master.zip

Download Unzip to D:\hadoop2.6

8. Modify the Win7 machine hosts file to include the following:

SY-0130 192.168.249.130
SY-0131192.168.249.131
SY-0132192.168.249.132
SY-0133192.168.249.133
SY-0134192.168.249.133

Third, compile Hadoop-eclipse-plugin-2.6.0.jar configuration

Operation under Win7

1. Add environment variable hadoop_home=d:\hadoop2.6\hadoop-2.6.0
Append environment variable path content:%hadoop_home%/bin

2. Modify the compilation package and dependent package version information
Modify D:\hadoop2.6\hadoop2x-eclipse-plugin-master\ivy\libraries.properties
hadoop.version=2.6.0 (When I test, the downloaded file, the setting is already 2.6.0)

3.ant compilation

Go to cmd command line:

D:\hadoop2.6\hadoop2x-eclipse-plugin-master\src\contrib\eclipse-plugin>

Ant Jar-dversion=2.6.0-declipse.home=d:\hadoop2.6\eclipse-jee-luna-sr1a-win32-x86_64\eclipse-dhadoop.home=d:\ hadoop2.6\hadoop-2.6.0

After compiling, Hadoop-eclipse-plugin-2.6.0.jar will be in D:\hadoop2.6\hadoop2x-eclipse-plugin-master\build\contrib\. Under the eclipse-plugin\ directory

iv. Configuration of the Eclipse environment
1. Copy the compiled Hadoop-eclipse-plugin-2.6.0.jar to the plugins directory of Eclipse and restart eclipse
2. Open the menu Window--preference--hadoop map/reduce to configure as shown in:

3. Display the Hadoop Connection Configuration window: Window--show view--other-mapreduce Tools, as shown in:

4. Configure the connection to Hadoop as shown in:

To see if the connection is successful, you can see the following information, which indicates a successful connection:

Vi. Creating a Map/reduce Project
1. Create a new project File--new--other--map/reduce project named MR1,
Then create the class Org.apache.hadoop.examples.WordCount, copy overrides from the HADOOP-2.5.2-SRC
(F:\hadoop\hadoop-2.5.2-src\hadoop-mapreduce-project\hadoop-mapreduce-examples\src\main\java\org\apache\hadoop \examples\wordcount.java)

2. Create a log4j.properties file
Create the Log4j.properties file in the SRC directory with the following content:
Log4j.rootlogger=debug,stdout,r
Log4j.appender.stdout=org.apache.log4j.consoleappender
Log4j.appender.stdout.layout=org.apache.log4j.patternlayout
log4j.appender.stdout.layout.conversionpattern=%5p-%m%n
Log4j.appender.r=org.apache.log4j.rollingfileappender
Log4j.appender.r.file=mapreduce_test.log
Log4j.appender.r.maxfilesize=1mb
Log4j.appender.r.maxbackupindex=1
Log4j.appender.r.layout=org.apache.log4j.patternlayout
log4j.appender.r.layout.conversionpattern=%p%t%c-%m%n
Log4j.logger.com.codefutures=debug

3. Settlement Java.lang.unsatisfiedlinkerror:org.apache.hadoop.io.nativeio.nativeio$windows.access0 (Ljava/lang/String;I) Exception issues
(Because your environment and I may not be consistent, you can have the following issues after the change)
Copy source files Org.apache.hadoop.io.nativeio.NativeIO to Project
Then position to 570 lines, directly modify to return true;
As shown in the following:

VII. running environment configuration under Windows(Restart the machine if not in effect)
Need Hadoop.dll,winutils.exe
I am directly copying the contents of the F:\hadoop\hadoop-common-2.2.0-bin-master\bin directory to overwrite F:\hadoop\hadoop-2.5.2\bin Viii. Running Project
In Eclipse, click Wordcount.java, right-click Run As->run configurations, configure the run parameters, i.e. input and output folders
Hdfs://ubuntu-v01:9000/user/hadoop/input Hdfs://ubuntu-v01:9000/user/hadoop/output
As shown in the following:

windows7+eclipse-jee-luna-sr1a-win32-x86_64+hadoop2.6 Environment Configuration