Eclipse-based hadoop environment Setup Guide.

 

Reproduced in: http://blog.csdn.net/lazythinker/article/details/6931651

 

 

1. Download and configure cygwin

1. Create a system variable cygwin with the variable value ntsc tty]

2. Edit the system variable "path" and add "C: \ cygwin \ bin]

Ii. install Java

Configure java_home;

3. Configure eclipse

1. Put the conins hadoop-*-eclipse-plugin.jar under the contrib/directory under the hadoop installation package into the Plugins directory of the eclipse directory;

2. For other configuration procedures, refer to other network documents;

3. For runtime configuration, select the run as-> open debug dialog option and set it on the arguments tab: [D: \ workspace \ hadoop-test \ In D: \ workspace \ hadoop-test \ out]. Fill in two directories, which are the data input directory and the output directory, separated by spaces;

4. Possible Problems

1. If this error occurs: Org. Apache. hadoop. Security. accesscontrolexception: Permission denied: User = administrator, access = execute, inode = "job_201111031322_0003": heipark: supergroup: rwx-

Cause: The local user administrator (a Windows user on the local machine) wants to remotely operate the hadoop system, which is caused by lack of permissions.

Solution:

A,Test EnvironmentTo cancel the hadoop HDFS user permission check. Open the conf/hdfs-site.xml and find the DFS. Permissions attribute changed to false (default: True) OK.

B. Modify the hadoop location parameter. On the advanced parameter tab, locate hadoop. Job. ugi and change this item to the hadoop startup user name.

Note that the hadoop. Job. ugi parameter may not be set for the first time. After an error is reported, you can see it.

2. When running the example: Ioexception: cannot run program "chmod": CreateProcess error = 2,

Cause:Cygwin is not installed or environment variables are not configured;
Solution:Install and configure the environment variable bin to restart eclipse under path to solve the problem;

3. If the following problems occur

11/11/03 12:21:31 warn mapred. localjobrunner: job_local_0001
Java. Lang. outofmemoryerror: Java heap Space
At org. Apache. hadoop. mapred. maptask $ mapoutputbuffer. <init> (maptask. Java: 781)
At org. Apache. hadoop. mapred. maptask. runoldmapper (maptask. Java: 350)
At org. Apache. hadoop. mapred. maptask. Run (maptask. Java: 307)
At org. Apache. hadoop. mapred. localjobrunner $ job. Run (localjobrunner. Java: 177)
11/11/03 12:21:31 info mapred. jobclient: Map 0% reduce 0%
11/11/03 12:21:31 info mapred. jobclient: job complete: job_local_0001
11/11/03 12:21:31 info mapred. jobclient: counters: 0
Exception in thread "Main" Java. Io. ioexception: Job failed!
At org. Apache. hadoop. mapred. jobclient. runjob (jobclient. Java: 1252)
At mapred. wordcount. Main (wordcount. Java: 110)

Cause: the memory allocation of the Java Virtual Machine is insufficient when the client is running.

Solution: Modify run configuration, select the arguments tab, and add-xmx1024m to the VM arguments column. Save the settings to solve the problem.