Win7 Eclipse Debug CentOS hadoop2.2-mapreduce (GO)

I. build your own development environment

Today, I built a set of Centos5.3 + Hadoop2.2 + Hbase0.96.1.1 development environment, Win7 Eclipse debug MapReduce success. May be the version of the reason for the high, out of the problem, the Internet can not find a complete solution, only on their own.

two. Hadoop installation

This is not verbose, online a lot of articles. I downloaded the hadoop-2.2.0.tar.gz.

1. Http://www.cnblogs.com/xia520pi/archive/2012/05/16/2503949.html describes the installation of Hadoop HDFs in great detail. It's not Hadoop2.2, but the configuration is very similar.
2. The MapReduce configuration can refer to http://blog.sina.com.cn/s/blog_546abd9f0101i8b8.html.

After the installation is successful, the following pages can be viewed smoothly, OK. My cluster environment is 200master,201-203slave.

1. Dfs.http.address 192.168.1.200:50070
2. Dfs.secondary.http.address 192.168.1.200:50090
3. Dfs.datanode.http.address 192.168.1.201:50075
4. Yarn.resourcemanager.webapp.address 192.168.1.200:50030
5. Mapreduce.jobhistory.webapp.address 192.168.1.200:19888. It seems to be unable to access. You need to start hadoop/sbin/mr-jobhistory-daemon.sh start Historyserver to access it.

three. hadoop2.x eclispe-plugin

Https://github.com/winghc/hadoop2x-eclipse-plugin

At present, the plug-in is still in development, you can download the source code compiled, the web also released a compiled jar.

Http://blog.csdn.net/zythy/article/details/17397153 the classmate wrote it in detail.

It is important to note that Hadoop installation directory fills in the Hadoop home address under win, and is designed to create a jar where MapReduce project can automatically introduce the MapReduce needed from this place. Unzip the hadoop-2.2.0.tar.gz to local.

four. various issues

When the previous step is complete, create a mapreduce Project, and the runtime discovers a problem.

1. Java code
   1. Java.io.IOException:Could not locate executable Null\bin\winutils.exe in the Hadoop binaries.
   With the code to find out is the problem of hadoop_home. If Hadoop_home is empty, it must be fullexename to Null\bin\winutils.exe. The solution is very simple, obediently configure the environment variable bar, do not want to restart the computer can be added to the MapReduce program System.setproperty ("Hadoop.home.dir", "..."); temporarily slowly. Org.apache.hadoop.util.Shell.javaJava Code

public static final string Getqualifiedbinpath (string executable)

throws IOException {

//construct Hadoop bin path to the specified executable

String fullexename = hadoop_home_dir + file.separator + "Bin"

+ File.separator + executable;

File Exefile = new File (Fullexename);

if (!exefile.exists ()) {

throw New IOException ("Could not locate executable" + fullexename

+ "in the Hadoop binaries.");

}

return Exefile.getcanonicalpath ();

}

Private static String Hadoop_home_dir = Checkhadoophome ();

Private static String Checkhadoophome () {

//First check the Dflag hadoop.home.dir with JVM scope

String home = System.getproperty ("Hadoop.home.dir");

//Fall back to the System/user-global env variable

if (home = = null) {

Home = system.getenv ("Hadoop_home");

}

...

}

2. This time get the full address fullexename, on my machine is D:\Hadoop\tar\hadoop-2.2.0\hadoop-2.2.0\bin\winutils.exe. Continue executing code and find the wrong Java code
   1. Could not locate executable D:\Hadoop\tar\hadoop-2.2. 0\hadoop-2.2.   0\bin\winutils.exe in the Hadoop binaries.
   Just to see, there is no winutils.exe this thing. Go to Https://github.com/srccodes/hadoop-common-2.2.0-bin to download one, put on it.
3. Continue with the problem Java code
   1. At Org.apache.hadoop.util.Shell.execCommand (Shell.java:661)
   2. At Org.apache.hadoop.fs.RawLocalFileSystem.setPermission (Rawlocalfilesystem.java:639)
   3. At Org.apache.hadoop.fs.RawLocalFileSystem.mkdirs (Rawlocalfilesystem.java:435)
   Continue with Code Org.apache.hadoop.util.Shell.javaJava code
   2. Public Static string[] Getsetpermissioncommand (String Perm, boolean recursive,
   4. String file) {
   6. string[] Basecmd = Getsetpermissioncommand (perm, recursive);
   8. string[] Cmdwithfile = arrays.copyof (basecmd, basecmd.length + 1);
   10. Cmdwithfile[cmdwithfile.length- 1] = file;
   12. return cmdwithfile;
   14. }
   18. /** Return a command to set permission */
   20. Public Static string[] Getsetpermissioncommand (String Perm, boolean recursive) {
   22. if (recursive) {
   24. return (WINDOWS)? New string[] {winutils, "chmod", "-R", perm}
   26. : new string[] { "chmod", "-R", Perm};
   28. } Else {
   30. return (WINDOWS)? New string[] {winutils, "chmod", perm}
   32. : new string[] { "chmod", perm};
   34. }
   36. }
   The contents of the Cmdwithfile array are {"D:\Hadoop\tar\hadoop-2.2.0\hadoop-2.2.0\bin\winutils.exe", "chmod", "755", "Xxxfile"}, I took this alone in CMD and found Java code
   1. This program cannot be started because the computer is missing MSVCR100.dll
   Then download a chant http://files.cnblogs.com/sirkevin/msvcr100.rar and throw it into C:\Windows\System32. Again cmd execution, again the problem Java code
   1. Application does not start properly (0xc000007b)
   Download http://blog.csdn.net/vbcom/article/details/7245186 Directx_repair to solve this problem. Remember to restart your computer after the repair is finished. It's great to try it after the cmd is done.
4. Here we have seen the dawn, but the problem comes again Java code
   1. Exception in thread "main" java.lang.unsatisfiedlinkerror:org.apache.hadoop.io.nativeio.nativeio$ WINDOWS.ACCESS0 (ljava/lang/string;i) Z
   Code goes to Java code
   1. /** Windows only method used to check if the current process has requested
   2. * Access rights on the given path. */
   3. Private static native boolean access0 (String path, int requestedaccess);
   Obviously missing DLL file, remember https://github.com/srccodes/ Hadoop-common-2.2.0-bin Download The thing, there is hadoop.dll, the best way is to replace the local Hadoop bin directory with the Hadoop-common-2.2.0-bin-master/bin directory, and configure the Path=hadoop_home/bin in the environment variable to restart the computer.
6. Finally saw the correct output of the MapReduce output99.

Five. Summary

1. Hadoop Eclipse Plugin is not necessary, its role in my opinion is the following three points (this is a wrong understanding, please refer to http://zy19982004.iteye.com/blog/2031172). Study-hadoop is an ordinary project that runs directly (without running on the Hadoop elephant) and can be debugged to mapreduce.
1. Visualize files in Hadoop.
2. Creating a MapReduce project helps you introduce dependent jars.
3. Configuration conf = new Config (), when it contains all of the configured information.
2. Or you download hadoop2.2 source code compiled well, there should be no problem (no pro-test).

Six. Other issues

1. or Java code
   1. Exception in thread "main" java.lang.unsatisfiedlinkerror:org.apache.hadoop.io.nativeio.nativeio$ WINDOWS.ACCESS0 (ljava/lang/string;i) Z
   The code followed the Org.apache.hadoop.util.NativeCodeLoader.java to see the Java code.
   2. static {
   4. //Try to load native Hadoop library and set fallback flag appropriately
   6. if (log.isdebugenabled ()) {
   8. Log.debug ("Trying to load the custom-built Native-hadoop library ...");
   10. }
   12. try {
   14. System.loadlibrary ("Hadoop");
   16. Log.debug ("Loaded the Native-hadoop Library");
   18. nativecodeloaded = true;
   20. } catch (Throwable t) {
   22. //Ignore failure to load
   24. if (log.isdebugenabled ()) {
   26. Log.debug ("Failed to load Native-hadoop with error:" + t);
   28. Log.debug ("java.library.path=" +
   30. System.getproperty ("Java.library.path"));
   32. }
   34. }
   38. if (!nativecodeloaded) {
   40. Log.warn ("Unable to load Native-hadoop library for your platform ... " +  
   42. "using Builtin-java classes where applicable");
   44. }
   46. }
   Here are the following Java code error
   1. DEBUG org.apache.hadoop.util.nativecodeloader-failed to load native-hadoop with error: Java.lang.unsatisfiedlinkerror:hadoop_home\bin\hadoop.dll:can ' t load AMD 64-bit. dll on a IA 32-bit platform
   Suspect is a 32-bit JDK problem, replaced with 64 bit, no problem Java code
   1. 2014-03- :805 DEBUG org.apache.hadoop.util.nativecodeloader-trying to load the Custom-built Native-hadoop library ...
   2. 2014-03- :812 DEBUG org.apache.hadoop.util.nativecodeloader-loaded the Native-hadoop Library
   A common warning Java code is also addressed here
   1. WARN util. nativecodeloader:unable to load native-hadoop library for your platform ... using Builtin-java classes where appli Cable

Win7 Eclipse Debug CentOS hadoop2.2-mapreduce (GO)