Configuring the Hadoop mapreduce development environment with Eclipse on Windows

Source: Internet
Author: User
Tags hadoop mapreduce

Configure Hadoop MapReduce development environment 1 with Eclipse on Windows. System environment and required documents
    • Windows 8.1 64bit
    • Eclipse (Version:luna Release 4.4.0)
    • Hadoop-eclipse-plugin-2.7.0.jar
    • Hadoop.dll & Winutils.exe
2. Modify the hdfs-site.xml of the master node

Add the following content

<property>     <name>dfs.permissions</name>     <value>false</value></property>

Designed to remove permission checks because I configured the Map/reduce connection to report the following error when I configured Eclipse to connect to the Hadoop server on my Windows machine. Org.apache.hadoop.security.AccessControlException:Permission denied:

3. Configure the Hadoop plug-in
    1. Place the downloaded Hadoop-eclipse-plugin-2.7.0.jar file in Eclipse's plugins directory and restart Eclipse to see that the plugin is in effect
    2. Create new Hadoop location, Eclipse menu bar, Show View, other MapReduce Tools, Map/reduce Locations


3. Configure to see the effect

4. Create a map/reduce Project
    1. New Project File–new–other–map/reduce Project named WordCount
      (Copy Wordcount.java from hadoop-2.6.0-cdh5.4.1 source, F:\DevTools\hadoop-2.6.0-cdh5.4.1\src\hadoop-mapreduce-project\ Hadoop-mapreduce-examples\src\main\java\org\apache\hadoop\examples\wordcount.java)

2. java.lang.UnsatisfiedLinkError:org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I) Exception Issues

Modify the master node in hadoop-2.6.0-cdh5.4.1\src\hadoop-common-project\hadoop-common\src\main\java\org\apache\hadoop\io\ Nativeio\nativeio.java

Then recompile the jar package with Maven in the/src/hadoop-common-project/hadoop-common directory
mvn package -Pdist,native -DskipTests -Dtar -e -X
When the compilation is complete, the corresponding jar package will be generated in Hadoop-common/target only with Hadoop-common-2.6.0-cdh5.4.1.jar replaced by Windows $hadoop_home\share\hadoop\ Jar Package for common directory

5. Under Windows Runtime Environment configuration

Copy the downloaded Hadoop.dll,winutils.exe to the $hadoop_home/bin directory

6. Run WordCount Project

In Eclipse, click Wordcount.java, right-click Run As->run configurations, configure the run parameters, i.e. input and output folders

hdfs://Master:9000/input hdfs://Master:9000/output 

(Note: If the output directory already exists, then delete or change a name, such as output01,output02 ... )

7. Running Results

Using Eclipse to configure the Hadoop mapreduce development environment on Windows

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.