Hadoop standalone deployment and eclipse Development

A) version

Hadoop-1.0.3

Jdk1.6.0 _ 32

Eclipse hadoop development plug-in hadoop-eclipse-plugin-1.0.3.jar

Hadoop-1.0.3 is installed in standalone mode, port HDFS 9000 mapred 9001

B) Environment

1) Installation Steps for the hadoop-1.0.3 can refer to the http://hadoop.apache.org/docs/r1.0.3/single_node_setup.html

Conf/core-site.xml

<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Put site-specific property overrides in this file. --><configuration><property>          <name>fs.default.name</name>          <value>hdfs://xxx.xxx.xxx.xxx:9000</value>  </property><property>        <name>hadoop.tmp.dir</name>        <value>/opt/hadoop-1.0.3/tmp</value> </property></configuration>

Conf/mapred-site.xml:

<configuration>     <property>         <name>mapred.job.tracker</name>         <value>xxx.xxx.xxx.xxx:9001</value>     </property></configuration>

Conf/hdfs-site.xml

<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Put site-specific property overrides in this file. --><configuration>        <property>                  <name>dfs.replication</name>                  <value>1</value>          </property>          <property>                 <name>dfs.permissions</name>                 <value>false</value>         </property></configuration>

Change XXX. XXX to the IP address of your server.

2) development

1. Use eclipse in windows. Copy the plug-in to the plugin directory.

2. Extract the hadoop-1.0.3 to the programmer's development machine.

3. Set hadoop MAP/reduce to the directory you extract

4. Configure MAP/reduce location

5,

Create a map/reduce project

Import wordcount. Java in examples \ org \ apache \ hadoop \ examples to the Project

If the problem arises, a permission error is prompted.

12/09/24 13:52:27 error security. usergroupinformation: priviledgedactionexception as: zzj cause: Java. io. ioexception: failed to set permissions of path: \ TMP \ hadoop-zzj \ mapred \ staging \ zzj-743566479 \. staging to 0700
Exception in thread "Main" Java. Io. ioexception: failed to set permissions of path: \ TMP \ hadoop-zzj \ mapred \ staging \ zzj-743566479 \. Staging to 0700

To solve this problem, you can directly use it.

conf.set("mapred.job.tracker", "218.249.75.144:9001");

To manually specify

Use RunAs to select run on hadoop to run this project. Eclipse automatically transfers the jar package to hadoop and runs