Getting started with Hadoop-compiling x64-bit Hadoop on Windows

Environment

Windows 7 x64 bit, Visual Studio Professional

Hadoop Source Version 2.2.0

Step (from the book "Pro Apache Hadoop, Second Edition" slightly modified.

2. Ensure that JDK, 1.6 is, or higher is installed. We assume that it's installed in thec:/myapps/jdkl6/ folder, which should has a bin subfolder.

4. Download the hadoop-2.2.x-src.tar.gz files (2.2.0 at the time of this writing) from the Download section of the A Pache Web site for Hadoop. The download link is as follows:http://www.apache.org/dist/hadoop/core/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz.

6. Explode the tar file into a directory. For the purpose of this section, we assume the directory is calledc:/myapps/hadoop/.

8. If using Visual Studio, use Visual Studio-Professional (not 2012). Do not use the Visual Studio Express because it does not the compiling for 64-bit, which presents problems if running on a 64-bit System. Alternatively, download the Microsoft Windows SDK v7.1. At the time of writing, the link is the following:

   http://www.microsoft.com/en-us/download/confirmation.aspx?id=8279.

10. Place the SDK in the folder with full privileges. For this section, we assume the folder isc:/myapps/winsdk71/.

12. Download and install the Linux-like environment for Windows named Cygwin. We assume that Cygwin home folder isc:/myapps/cygwin/. Ensure that the following UNIX command-line tools is installed:sh, mkdir, RM, CP, Tar, and gzip. The download location for Cygwin ishttp://cygwin.com/install.html.

14. Download and install Maven 3.1.1 or a compatible version. Installation consists of simply exploding thetar bundle into a Windows folder. We assume the home folder for Maven download isc:/myapps/apache-maven-3.1.1/. Verify that this folder has the bin subfolder.

16. Download and install Protocol buffers 2.5.0. At the time of writing, the link is http://protobuf.googlecode.com/files/protoc-2.5.0-win32.zip.

18. Simply unzip the zip file into a folder (assumed to be c:/myapps/protobuf/). Verify the installation is correct by ensuring the folder contains theprotoc.exe file.

20. Install ZLIB from Http://zlib.net/zlib128-dll.zip and explode the zip file in the c:/zlib-1.2.8 folder. At runtime, this folder must is accessible from thePATH variable.

22. ADD the environment variables shown in Table A-1.

Table A-1: Environment Variables for Windows

Environment Variable	Value
Java_home	c:/myapps/jdk16/
M2_home	C:/myapps/apache-maven-3.1.1/
Platform	x64 (or Win32 when building on the 32-bit system)
Cygwin_home	c:/myapps/cygwin/
Protobuf_home	c:/myapps/protobuf/
Zlib_home	C :/zlib-1.2.8 12. Add the following to the Windows PATH variable: %java_home%/bin/ %cygwin_ home%/bin/ %m2_home%/bin/ %protobuf_home% %zlib_home

To start compiling, perform the following steps:

Hadoop 2.2.0 has a bug at compile time, refer to https://issues.apache.org/jira/browse/HADOOP-10110 description, then download https://issues.apache.org/ Jira/secure/attachment/12614482/hadoop-10110.patch This patch, modify it according to this patch hadoop-common-project/hadoop-auth/ Pom.xml the file, and then compile it, or it will go wrong.

2. Microsoft Windows SDK v7.1 and start the Windows SDK 7.1 command prompt in Run as Administrator mode.

4. C:/myapps/hadoop. Recall that's the folder in which, the Hadoop source was placed.

6. Execute the following command MVN package with the -pdist,native-win-dskiptests-dtar options. The exact command is this:

   MVN Package-pdist,native-win-dskiptests–dtar

8. If There is no errors, the hadoop-2.2.0.tar.gz file was created in the c:/myapps/hadoop/hadoop-dist/hadoop-2 .2.0/ folder.

After compiling, if there are no errors, the hadoop-2.2.0.tar.gz file will be generated under the Hadoop-2.2.0-src\hadoop-dist\target directory.

The first step is done!



Getting started with Hadoop-compiling x64-bit Hadoop on Windows