Build hadoop-2.6.0 local IDEA development environment under Windows

Overview

This document documents the construction of the Hadoop Local development environment under Windows:

OS：windowshadoop运行模式：独立模式

installation package Structure:

Hadoop-2.6.0-Windows.zip    - cygwinInstall // cygwin离线安装包    - hadoop-2.6.0-windows.tar.gz // hadoop-2.6.0 windows安装包

Ps:

hadoop-2.6.0-windows.tar.gz是基于官方发行包hadoop-2.6.0.tar.gz，并为windows单机测试而自定义的包，改动如下：1. 在bin目录下，添加了hadoop.dll和winutils.exe2. 修改了org\apache\hadoop\io\nativeio\NativeIO.java源码第558行*由于源码有改动，此包不能用于生产环境，只限于windows环境下的开发测试

1. Installing Cygwin

1. Double-click the Setup-x86_64.exe under Cygwininstall

2. Choose to install from local

3. Select Installation Location

4. Select Local Installation file location

5. Click Default next to all to make it install, then click Next to start the installation

2. Installing Hadoop

2. Under the Cygwin root directory, create the OPT directory and copy the installation package to the OPT directory

   cd /; mkdir opt

4. Extract

   cd /opt; tar xzf hadoop-2.6.0-windows.tar.gz

6. Setting environment variables

   Set the Windows environment variable Hadoop_prefix point to the Hadoop installation directory

   Set the Windows environment variable Hadoop_home point to the Hadoop installation directory

   Set the Windows environment variable Java_home point to the Java installation directory

   The PATH environment variable is appended:;%HADOOP_HOME%\bin

8. The following environment variables are set in ~/.BASHRC:

   export HADOOP_CLASSPATH=$(cygpath -pw $(hadoop classpath)):$HADOOP_CLASSPATH

10. Make ~/.BASHRC effective and verify that Hadoop is working

    source ~/.bashrc

    Verifying Hadoop:

    hadoop version

    Installation is successful if version information is displayed normally
    

12. Run the sample code to verify that Hadoop is capable of running MapReduce

    cd /opt/hadoop-2.6.0; mkdir input; cd input; echo "Hello World" > test

    hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar wordcount input output

    If normal execution is displayed:
    

    The part-r-00000 file is generated under/opt/hadoop-2.6.0/output and contains the following:

    Hello   1world   1

3. Idea Development Environment Configuration

1. Create a MapReduce project

2. Select project, press F12, add external dependency

3. Add the necessary jar packages under Hadoop_home

4. Go to run Configuration

5. Configure run parameters

   Working directory is set to the Hadoop installation directory

   and set the input parameters

6. Debugging

   Break points as normal Java programs to debug

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Build hadoop-2.6.0 local IDEA development environment under Windows