windows7+eclipse+hadoop2.5.2 Environment Configuration
I. Hadoop cluster environment configuration
Refer to my previous post (Ubuntu + hadoop2.5.2 Distributed Environment configuration http://www.cnblogs.com/huligong1234/p/4136331.html)
However, I also modified the following when I configured (because your environment and I may not be consistent, you can follow the relevant issues later to modify):
A. Modify hdfs-site.xml on the Master node (UBUNTU-V01) plus the following:
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
Designed to remove permission checks because I configured the Map/reduce connection to report the following error when I configured Eclipse to connect to the Hadoop server on my Windows machine. Org.apache.hadoop.security.AccessControlException:Permission denied:
B. Also modify hdfs-site.xml on the Master node (UBUNTU-V01) plus the following:
<property>
<name>dfs.web.ugi</name>
<value>jack,supergroup</value>
</property>
The reason is that at run time, the following error is reported WARN Org.apache.hadoop.security.ShellBasedUnixGroupsMapping:got exception trying to get groups for user Jack
Should be my Windows user name is Jack, no access rights
More permission configurations can be found in the official documentation:
HDFS Rights Management User Guide http://hadoop.apache.org/docs/r1.0.4/cn/hdfs_permissions_guide.html
Restart the Hadoop cluster after the configuration has been modified:
[email protected]: ~/data$./sbin/stop-dfs.sh
[email protected]: ~/data$./sbin/stop-yarn.sh
[email protected]: ~/data$./sbin/start-dfs.sh
[email protected]: ~/data$./sbin/start-yarn.sh
Two. Windows Basic Environment Preparation
Windows7 (x64), Jdk,ant,eclipse,hadoop
1.JDK Environment Configuration
Jdk-6u26-windows-i586.exe Configure the relevant JAVA_HOME environment variable after installation and configure the bin directory to path
2.eclipse Environment Configuration
Eclipse-standard-luna-sr1-win32.zip extracted to the D:\eclipse\ directory and named Eclipse-hadoop
: Http://developer.eclipsesource.com/technology/epp/luna/eclipse-standard-luna-SR1-win32.zip
3.ant Environment Configuration
Apache-ant-1.9.4-bin.zip unzip to the D:\apache\ directory, configure the environment variable Ant_home, and configure the bin directory to the path
: Http://mirror.bit.edu.cn/apache//ant/binaries/apache-ant-1.9.4-bin.zip
4. Download hadoop-2.5.2.tar.gz
Http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.5.2/hadoop-2.5.2.tar.gz
5. Download hadoop-2.5.2-src.tar.gz
Http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.5.2/hadoop-2.5.2-src.tar.gz
6. Download Hadoop2x-eclipse-plugin
Https://github.com/winghc/hadoop2x-eclipse-plugin
7. Download Hadoop-common-2.2.0-bin
Https://github.com/srccodes/hadoop-common-2.2.0-bin
Extract hadoop-2.5.2.tar.gz, hadoop-2.5.2-src.tar.gz, Hadoop2x-eclipse-plugin, hadoop-common-2.2.0-bin download to F:\hadoop\ respectively Directory under
8. Modify the local Hosts file to include the following: 192.168.1.112 UBUNTU-V01
third, compile Hadoop-eclipse-plugin-2.5.2.jar configuration
1. Add environment variable Hadoop_home=f:\hadoop\hadoop-2.5.2\
Append environment variable path content:%hadoop_home%/bin
2. Modify the compilation package and dependent package version information
Modify F:\hadoop\hadoop2x-eclipse-plugin-master\ivy\libraries.properties
hadoop.version=2.5.2
jackson.version=1.9.13
3.ant compilation
F:\hadoop\hadoop2x-eclipse-plugin-master\src\contrib\eclipse-plugin>
Ant jar-dversion=2.5.2-declipse.home=d:\eclipse\eclipse-hadoop\eclipse-dhadoop.home=f:\hadoop\hadoop-2.5.2
After compiling, the Hadoop-eclipse-plugin-2.5.2.jar will be in the F:\hadoop\hadoop2x-eclipse-plugin-master\build\contrib\eclipse-plugin directory.
iv. Configuration of the Eclipse environment
1. Copy the compiled Hadoop-eclipse-plugin-2.5.2.jar to the plugins directory of Eclipse and restart eclipse
2. Open the menu Window--preference--hadoop map/reduce to configure as shown in:
3. Display the Hadoop Connection Configuration window: Window--show view--other-mapreduce Tools, as shown in:
4. Configure the connection to Hadoop as shown in:
To see if the connection is successful, you can see the following information, which indicates a successful connection:
v. Hadoop cluster environment add test files(No configuration required if already available)
Create input directory on A.dfs
[email protected]: ~/data/hadoop-2.5.2$bin/hadoop fs-mkdir-p Input
B. Copy the README.txt from the Hadoop directory into DFS new input
[email protected]: ~/data/hadoop-2.5.2$bin/hadoop fs-copyfromlocal README.txt input
Vi. Creating a map/reduce Project
1. Create a new project File--new--other--map/reduce project named MR1,
Then create the class Org.apache.hadoop.examples.WordCount, copy overrides from the HADOOP-2.5.2-SRC
(F:\hadoop\hadoop-2.5.2-src\hadoop-mapreduce-project\hadoop-mapreduce-examples\src\main\java\org\apache\hadoop \examples\wordcount.java)
2. Create a log4j.properties file
Create the Log4j.properties file in the SRC directory with the following content:
Log4j.rootlogger=debug,stdout,r
Log4j.appender.stdout=org.apache.log4j.consoleappender
Log4j.appender.stdout.layout=org.apache.log4j.patternlayout
log4j.appender.stdout.layout.conversionpattern=%5p-%m%n
Log4j.appender.r=org.apache.log4j.rollingfileappender
Log4j.appender.r.file=mapreduce_test.log
Log4j.appender.r.maxfilesize=1mb
Log4j.appender.r.maxbackupindex=1
Log4j.appender.r.layout=org.apache.log4j.patternlayout
log4j.appender.r.layout.conversionpattern=%p%t%c-%m%n
Log4j.logger.com.codefutures=debug
3. Settlement Java.lang.unsatisfiedlinkerror:org.apache.hadoop.io.nativeio.nativeio$windows.access0 (Ljava/lang/String;I) Exception issues
(Because your environment and I may not be consistent, you can have the following issues after the change)
Copy source files Org.apache.hadoop.io.nativeio.NativeIO to Project
Then position to 570 lines, directly modify to return true;
As shown in the following:
VII. running environment configuration under Windows(Restart the machine if not in effect)
Need Hadoop.dll,winutils.exe
I am directly copying the contents of the F:\hadoop\hadoop-common-2.2.0-bin-master\bin directory to overwrite F:\hadoop\hadoop-2.5.2\bin
Viii. Running Project
In Eclipse, click Wordcount.java, right-click Run As->run configurations, configure the run parameters, i.e. input and output folders
Hdfs://ubuntu-v01:9000/user/hadoop/input Hdfs://ubuntu-v01:9000/user/hadoop/output
As shown in the following:
Note: If the output directory already exists, then delete or change a name, such as output01,output02 ...
In addition, there are problems can be read more log (http://ubuntu-v01:8088/logs/)
See information:
Hadoop2.4 experience summary of using Eclipse in Win7 to connect virtual machines in Ubuntu
Http://www.aboutyun.com/thread-7784-1-1.html
hadoop2.2 Eclipse Connection HDFs (Hadoop)
Http://www.aboutyun.com/thread-8190-1-1.html
Novice Guide: Using Eclipse to remotely connect Hadoop for program development on Windows
Http://www.aboutyun.com/thread-6001-1-1.html
How Windows Eclipse runs MapReduce encounters permissions issues how to resolve
Http://www.aboutyun.com/thread-7660-1-1.html
Summary of Hadoop development methods and operational guidance
Http://www.aboutyun.com/thread-6950-1-1.html
Summary of Hadoop errors and workarounds I've encountered
http://heipark.iteye.com/blog/786302
HDFS Rights Management User Guide
Http://hadoop.apache.org/docs/r1.0.4/cn/hdfs_permissions_guide.html
windows7+eclipse+hadoop2.5.2 Environment Configuration
