Configuring the Hadoop mapreduce development environment with Eclipse on Windows

Configure Hadoop MapReduce development environment 1 with Eclipse on Windows. System environment and required documents

• Windows 8.1 64bit
• Eclipse (Version:luna Release 4.4.0)
• Hadoop-eclipse-plugin-2.7.0.jar
• Hadoop.dll & Winutils.exe
  

2. Modify the hdfs-site.xml of the master node

Add the following content

<property>     <name>dfs.permissions</name>     <value>false</value></property>

Designed to remove permission checks because I configured the Map/reduce connection to report the following error when I configured Eclipse to connect to the Hadoop server on my Windows machine. Org.apache.hadoop.security.AccessControlException:Permission denied:

3. Configure the Hadoop plug-in

1. Place the downloaded Hadoop-eclipse-plugin-2.7.0.jar file in Eclipse's plugins directory and restart Eclipse to see that the plugin is in effect
2. Create new Hadoop location, Eclipse menu bar, Show View, other MapReduce Tools, Map/reduce Locations

3. Configure to see the effect

4. Create a map/reduce Project

1. New Project File–new–other–map/reduce Project named WordCount
   (Copy Wordcount.java from hadoop-2.6.0-cdh5.4.1 source, F:\DevTools\hadoop-2.6.0-cdh5.4.1\src\hadoop-mapreduce-project\ Hadoop-mapreduce-examples\src\main\java\org\apache\hadoop\examples\wordcount.java)

2. java.lang.UnsatisfiedLinkError:org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I) Exception Issues

Modify the master node in hadoop-2.6.0-cdh5.4.1\src\hadoop-common-project\hadoop-common\src\main\java\org\apache\hadoop\io\ Nativeio\nativeio.java

Then recompile the jar package with Maven in the/src/hadoop-common-project/hadoop-common directory
mvn package -Pdist,native -DskipTests -Dtar -e -X
When the compilation is complete, the corresponding jar package will be generated in Hadoop-common/target only with Hadoop-common-2.6.0-cdh5.4.1.jar replaced by Windows $hadoop_home\share\hadoop\ Jar Package for common directory

5. Under Windows Runtime Environment configuration

Copy the downloaded Hadoop.dll,winutils.exe to the $hadoop_home/bin directory

6. Run WordCount Project

In Eclipse, click Wordcount.java, right-click Run As->run configurations, configure the run parameters, i.e. input and output folders

hdfs://Master:9000/input hdfs://Master:9000/output 

(Note: If the output directory already exists, then delete or change a name, such as output01,output02 ... )

7. Running Results

Using Eclipse to configure the Hadoop mapreduce development environment on Windows