Hadoop learning notes (9): How to remotely connect to hadoop for program development using eclipse on Windows

Hadoop is mainly deployed and applied in the Linux environment, but the current public's self-knowledge capabilities are limited, and the work environment cannot be completely transferred to the Linux environment (of course, there is a little bit of selfishness, it's really a bit difficult to use so many easy-to-use programs in Windows in Linux-for example, quickplay, O (always _ success) O ~), So I tried to use eclipse to remotely connect to hadoop for development. The steps below are as follows:

1. first put the hadoop-eclipse-plugin-1.0.4.jar (specific version depends on your hadoop version) in the eclipse installation directory plugins folder, if you re-open eclipse to see the following view, it indicates that your hadoop plug-in has been installed successfully:
　

The "hadoop installation directoryallocation configuration item refers to your hadoopinstallation directory. In Windows, you only need to extract the downloaded hadoop-1.0.4.tar.gz package to a specific location and point it to this location.

2. Configure MAP/reduce locations in eclipse, as shown in:

The host "master" is the host name that I customized in "C: \ windows \ system32 \ drivers \ etc \ hosts:
　　　　218.195.250.80 master

If the following "DFS locations" is displayed in eclipse, it means that eclipse has successfully connected to remote hadoop (Note: Do not forget to switch your view to the map/reduce view, instead of the default Java view ):
　　

3. Now let's test the maxtemperature example program in the hadoop authoritative guide and create the following three classes:

If we run the maxtemperature class, the following error is reported:

12/04/24 15:32:44 warn util. nativecodeloader: Unable to load native-hadoop library for your platform... using builtin-Java classes where applicable 12/04/24 15:32:44 error security. usergroupinformation: priviledgedactionexception as: Administrator cause: Java. io. ioexception: failed to set permissions of path: \ TMP \ hadoop-Administrator \ mapred \ staging \ administrator-519341271 \. staging to 0700 Exception in thread "Main" Java. Io. ioexception: failed to set permissions of path: \ TMP \ hadoop-Administrator \ mapred \ staging \ administrator-519341271 \. Staging to 0700

This is a File Permission issue in windows. It can run normally in Linux and does not exist.
The solution is to modify the hadoop-1.0.4/src/CORE/org/Apache/hadoop/fs/fileutil. in Java, the checkreturnvalue can be commented out (a little rough, in the window, you do not need to check ):
Re-compile the packaging hadoop-core-1.0.4.jar, replace the hadoop-1.0.4 under the hadoop-core-1.0.4.jar root directory. (When I re-package out of some problems, directly to download the hadoop-core-1.0.2.jar from the Internet instead of the hadoop-core-1.0.4.jar, this can also run normally,: https://skydrive.live.com /? Cid = cf7746837803bc50 & id = cf7746837803bc50 % 211276)
(In fact there is another simple way, we just need to put the hadoop-1.0.4/src/CORE/org/Apache/hadoop/fs/fileutil. java modified and re-compiled class file added to the original hadoop-core-1.0.4.jar package to replace the original fileutil. class file, there is a good hadoop-core-1.0.4.jar package for the Windows environment, you can download)

In addition, I encountered the following error:

Org. Apache. hadoop. Security. accesscontrolexception: Permission denied: User = administrator, access = write, inode = "tmp": Root: supergroup: rwxr-XR-X.

This error is a bit confusing because I have configured the user name hadoop in MAP/reduce locations (hadoop is the user name for running hadoop clusters on Linux ), I don't know why hadoop is positioned as an administrator user. The solution is as follows:

Cause: the local user administrator (a Windows user on the local machine) wants to remotely operate the hadoop system and has no permissions.
Solution:
A. In the test environment, you can cancel the hadoop HDFS user permission check. Open the conf/hdfs-site.xml and find the DFS. Permissions attribute changed to false (default: True) OK.
B. Modify the hadoop location parameter. On the advanced parameter tab, locate hadoop. Job. ugi and change this item to the hadoop startup user name. (Note that the hadoop. Job. ugi parameter may not be set for the first time. After an error is reported, you can check it again .)
C. When eclipse uses the hadoop plug-in to submit jobs, jobs are written to the HDFS file system as drwho by default, corresponding to/user/hadoop on HDFS, the drwho user does not have the write permission on the hadoop Directory, which causes exceptions. Solution: Open the hadoop directory and run the following command: $ hadoop FS-chmod 777

The result is as follows:
　　
Reprinted please indicate the source: http://www.cnblogs.com/beanmoon/archive/2013/01/05/2845579.html