Configure Hadoop MapReduce development environment 1 with Eclipse on Windows. System environment and required documents
- Windows 8.1 64bit
- Eclipse (Version:luna Release 4.4.0)
- Hadoop-eclipse-plugin-2.7.0.jar
- Hadoop.dll & Winutils.exe
2. Modify the hdfs-site.xml of the master node
Add the following content
<property> <name>dfs.permissions</name> <value>false</value></property>
Designed to remove permission checks because I configured the Map/reduce connection to report the following error when I configured Eclipse to connect to the Hadoop server on my Windows machine. Org.apache.hadoop.security.AccessControlException:Permission denied:
3. Configure the Hadoop plug-in
- Place the downloaded Hadoop-eclipse-plugin-2.7.0.jar file in Eclipse's plugins directory and restart Eclipse to see that the plugin is in effect
- Create new Hadoop location, Eclipse menu bar, Show View, other MapReduce Tools, Map/reduce Locations
3. Configure to see the effect
4. Create a map/reduce Project
- New Project File–new–other–map/reduce Project named WordCount
(Copy Wordcount.java from hadoop-2.6.0-cdh5.4.1 source, F:\DevTools\hadoop-2.6.0-cdh5.4.1\src\hadoop-mapreduce-project\ Hadoop-mapreduce-examples\src\main\java\org\apache\hadoop\examples\wordcount.java)
2. java.lang.UnsatisfiedLinkError:org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)
Exception Issues
Modify the master node in hadoop-2.6.0-cdh5.4.1\src\hadoop-common-project\hadoop-common\src\main\java\org\apache\hadoop\io\ Nativeio\nativeio.java
Then recompile the jar package with Maven in the/src/hadoop-common-project/hadoop-common directory
mvn package -Pdist,native -DskipTests -Dtar -e -X
When the compilation is complete, the corresponding jar package will be generated in Hadoop-common/target only with Hadoop-common-2.6.0-cdh5.4.1.jar replaced by Windows $hadoop_home\share\hadoop\ Jar Package for common directory
5. Under Windows Runtime Environment configuration
Copy the downloaded Hadoop.dll,winutils.exe to the $hadoop_home/bin directory
6. Run WordCount Project
In Eclipse, click Wordcount.java, right-click Run As->run configurations, configure the run parameters, i.e. input and output folders
hdfs://Master:9000/input hdfs://Master:9000/output
(Note: If the output directory already exists, then delete or change a name, such as output01,output02 ... )
7. Running Results
Using Eclipse to configure the Hadoop mapreduce development environment on Windows