hadoop2.7.1 nutch2.3 two times development Windows environment



Hadoop Windows compilation:

refer to Building.txt and http://wiki.apache.org/hadoop/Hadoop2OnWindows, Compile Condition:

* Windows System-win8
* JDK 1.7+-jdk1.8
* Maven 3.0 or Later-maven 3.3
* Findbugs 1.3.9 (if running Findbugs)- None
* Protocolbuffer 2.5.0-2.5 build by vs2013
* CMake 2.6 or newer-cmake3.3
* Windows SDK 7.1 or Visual Studio professional-vs2010
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* Zlib headers (if building native code bindings for zlib)
* Internet Connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from Gnuwin32:sh, MkDir, RM, CP, TAR, gzip. These
Tools must is present on your PATH.

1. Unix Command-Line Tools:http://git-scm.com/download/win

PATH environment variable set C:\Program Files (x86) \git\cmd; C:\Program Files (x86) \git\bin

If you want to use Gnuwin32, the Hosts file needs to be added (domestic access not)

216.34.181.96 getgnuwin32.sourceforge.net

2. Protocolbuffer 2.5.0 can be compiled with vs2010/vs2013, several projects need to add/fs

Plus, #include <algorithm>

Put the compiled protoc.exe/libprotobuf.lib/libprotobuf-lite.lib/libprotoc.lib into c:/windows, open the cmd input protoc,

3. Copy Zlib1.dll (can be downloaded online, preferably with 1.2.7 version) to C:/windows

4. Copy the C:\Java\jdk1.8.0_51\lib\tools.jar to C:\Java\jdk1.8.0_51\lib\ according to the error message prompted during compilation

5. Open cmd, note that there is no program file in the Java_home directory so there are spaces

Call "%vs100comntools%. \.. \vc\bin\vcvars32.bat "

Set Platform=win32

Set java_home=c:\java\jdk1.8.0_51

CD E:/HADOOP-2.7.1-SRC

MVN Package-pdist,native-win-dskiptests-dtar

6. MVN can try several times and eventually tar files in hadoop-dist\target\

Start Hadoop:

To configure Hadoop to fully reference Linux under configuration, it is important to note that:

1. Where the native directory is involved, add file:///, or HDFs Namenode-format will fail

File:///e:/nutch/hdfs/name

2. error:missing ' server ' JVM at ' C:\Java\jdk1.8.0_51\bin\server\jvm.dll ' may be prompted.

Copy the C:\Java\jdk1.8.0_51\jre\bin\server directory to the C:\Java\jdk1.8.0_51\bin\

Eclipse Import Nutch2.3:

1. Under Windows compilation, Ant Runtime and Linux are not very different

2. Import nutch2.3, online has a routine. Ant runtime to import routines later

Open Eclipse,new-java Project-the basic

Can finish, there will be some bugs:

3. Run As-java appliction, select Injectorjob

4. Run successfully, first run Hadoop

5. InjectorJob:java.lang.NullPointerException
At Java.lang.ProcessBuilder.start (Unknown Source)

, while Hadoop_home/bin joins the path to resolve

You may need to restart Eclipse:)

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

hadoop2.7.1 nutch2.3 two times development Windows environment