Eclipse-hadoop Development Configuration Detailed

Eclipse_hadoop Development Detailed

Eclipse-hadoop Development Configuration Detailed

The prerequisite Summary is a summary of the configuration issues encountered during the Hadoop-eclipse development environment. The information summarized in this article is primarily a development installation configuration for the Hadoop-2.2.0 version of Eclipse. The local development environment is Win7 (64-bit) +eclipse (Kepler Service Release 2 64-bit), Win7 (64-bit), JDK (1.6.0_31 64-bit), and the server that is developing the connection is the CentOS Linux Release 7.2.1511 (Core).

Environment preparation

• CentOs (CentOS Linux release 7)
• JDK (1.6.0_31 64-bit)
• Eclipse ( Kepler Service Release 2 ) +hadoop plug-in
• Hadoop-2.2.0
• Win7 (64-bit)

Installation and configuration of the Eclipse plugin

Install the Eclipse plug-in and set the location of the hadoop-2.2.0 file package.

• After Eclipse ( Kepler Service Release 2 ) is decompressed, Put the Hadoop-eclipse-plugin-2.6.0.jar package in the Plugins folder, open Eclipse, and you'll see the options for Hadoop map/reduce in the menu bar (Window) submenu (Preferences).

• After downloading the hadoop-2.2.0.tar.gz compressed package to the WIN7 environment, unzip the hadoop-2.2.0.tar.gz to hadoop-2.2.0.
• Configure the variable environment Hadoop_home and path as shown in

• Set the location of Hadoop in the options for Hadoop map/reduce.

Unzip hadoop2.6 (x64). zip to hadoop2.6 (x64), copy all files to Hadoop-2.2.0bin directory hadoop.dll copy C:\Windows\System32 directory.

Next, you'll be able to develop Hadoop

Open the MapReduce view

Window---Open perspective, other select Map/reduce, icon is a blue elephant

Add a connected Map/reduce environment

• In the Map/reduce locations view, right-click to select "New Hadoop location ..." and:

Configure the Popup dialog box to fill in the following:

1. Location Name:

   This does not care, is the mark of this mapreduce, as long as can help you remember

2. Map/reduce Master section Related definitions:

   Host: The IP address of the machine where the Jobtracker is located in the cluster in the previous section Port:jobtracker port (default is 9001) two parameters are Mapred.job.tracker IP and port in Mapred-site.xml DFS Master section: Host: Is the last section of the cluster Namenode machine IP port: Namenode Port These two parameters are in the Core-site.xml fs.default.name inside the IP and port

3. User name: Is the username used to build the Hadoop cluster, I use had. After filling out the above information, close Eclipse and restart.

Local files are uploaded to the remote Dfs file system via the DFS plug-in operation

Attention

When uploading files to the Dfs file system, the name node is in Safe mode can appear

When the Distributed file system starts, there will be a security mode at the beginning, and when the Distributed file system is in Safe mode, the contents of the file system are not allowed to be modified or deleted until the end of safe mode. The Safe mode is to check the validity of the data blocks on each datanode when the system is started, and to copy or delete some data blocks according to the policy. The run time can also be entered in safe mode through commands. In practice, when the system starts to modify and delete files will also have a safe mode does not allow the modification of the error prompt, only need to wait a while. If you do not want to wait for Hadoop dfsadmin -safemode leave the security mode of Hadoop to be closed by using a command or modify dfs.safemode.threshold.pct to a smaller value, the default is 0.999.

用户可以通过dfsadmin -safemode value  来操作安全模式，参数value的说明如下：enter - 进入安全模式leave - 强制NameNode离开安全模式get -  返回安全模式是否开启的信息wait - 等待，一直到安全模式结束。

Security Mode exit premise-The minimum backup level is reached for the blocks of the entire file system (99.9% by default, which can be set by dfs.safemode.threshold.pct) (default is 1, Can be set via Dfs.replication.min).

HDFs Directory

Execute the command on the server to see if there [[email protected] /]# hdfs dfs -mkdir /input is an input directory, and if you have permission to view [[email protected] /]# hdfs dfs -ls / it, you can also use the plugin to see if an HDFs entry directory is usedhadoop fs -chmod 777 /你新建的目录

[[email protected] /]# hdfs dfs -ls /17/03/30 16:49:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableFound 3 itemsdrwxr-xr-x   - had   supergroup  0 2017-03-30 09:34 /inputdrwxr-xr-x   - Administrator supergroup  0 2017-03-30 15:25 /outputdrwxr-xr-x   - Administrator supergroup  0 2017-03-30 15:20 /temp

New Hadoop-map/reduce Project

Step: File-->new-->other-->map/reduce Project

Import project files to experience the wordcount of Hadoop.

The project name can be taken casually, such as WordCount. Copy the Hadoop installation directory Src/example/org/apache/hadoop/example/wordcount.java to the project you just created.

In the project it may involve your local users uploading HDFs permissions to the Nativeio.java class of the problem Org.apache.hadoop.io.nativeio. http://www.genshuixue.com/i-cxy/p/15193768

8) Running the program

Right-click the WordCount project, select Run As-->runconfigurations ..., and the Run Configurations dialog box pops up. As shown in the following:

Right-click on the left Java application and choose New menu to create a configuration item named WordCount. Set program arguments in the Arguments tab:

Hdfs://192.168.0.220:9000/input/test HDFS://192.168.0.220:9000/OUTPUT/OUT1

The first behavior is the input file, and the second behavior outputs the result file.

If you run the Times Java.lang.OutOfMemoryError:Java heap space to configure VM parameters:-xms512m-xmx1024m-xx:maxpermsize=256m When you are finished setting up, click Run.

Common errors See http://blog.csdn.net/yonghutwo/article/details/9206059 http://blog.csdn.net/collaboom/article/details/ 52015967 http://blog.csdn.net/yonghutwo/article/details/9206059

Eclipse-hadoop Development Configuration Detailed