Using Java to call the Hadoop interface Learning notes

Introduction

Since Hadoop is installed on Linux systems, it is necessary to use Eclipse to develop Java on Linux systems. Many companies now require the experience of developing Java on Linux, so this is a great opportunity for a practiced hand. Learning Hadoop is not just learning Hadoop, I believe that the introduction of Hadoop, Linux is also a primer, the shell is also introduced, Linux Java will be developed, perhaps this is the charm of learning, so that people can not stop! Don't say much nonsense, get to the point.

"Install Eclipse"

工欲善其事, its prerequisite, to develop Hadoop programs on Linux with Java, it is essential to build the environment.

Since JDK and Hadoop have already been installed on Linux, it is only necessary to install Eclipse on Linux, I am installing eclipse-standard-kepler-sr1-linux-gtk.tar.gz This version of Eclipse. First, use Securefx to upload the package to the Linux system.

Unzip the eclipse-standard-kepler-sr1-linux-gtk.tar.gz into the /usr/local/src directory (personal habits).

TAR-ZXVF eclipse-jee-luna-sr1a-linux-gtk.tar.gz-c/USR/LOCAL/SRC

After extracting the theory, you can open eclipse for development directly. Go to the Linux desktop and go to the /usr/local/src/eclipse directory:

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/59/E8/wKioL1TvFNzCYLsFAALXxvP6SeY349.jpg "title=" Image 1.png "alt=" Wkiol1tvfnzcylsfaalxxvp6sey349.jpg "/>

Set up a shortcut to eclipse

Click Eclipse to start the Eclipse program, but this is cumbersome, each time you need to go to the /usr/local/src/eclipse directory to start, so it is necessary to set up shortcuts.

First go back to Centeros's desktop, right--"create launcher,

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/59/E8/wKioL1TvGByCR63BAAQv9uOz8B4851.jpg "title=" Image 4.png "alt=" wkiol1tvgbycr63baaqv9uoz8b4851.jpg "/> Input Name--" Click Browse, select /usr/local/src/eclipse/eclipse-" Click on the spring icon and select /usr/local/src/eclipse/icon.xpm-"OK

OK, you are done.

"Import a Hadoop jar package"

Open eclipse, is not very familiar with, here my workspace selected is the default /root/workspace

Create a Java project, my project name is called Hadoop. Create a new folder in the project Lib is used to store jar packages, are these operations simple?

Next is the import of Hadoop jar package, I choose to use the command line to import jar package, practice practiced hand!

[[email protected] ~]# Cd/usr/local/src/hadoop-2.2.0/share/hadoop/[[email protected] hadoop]# ll total dosage 24drwxr-xr-x. 6 67974 users 4096 October 7 commondrwxr-xr-x. 7 67974 users 4096 October 7 hdfsdrwxr-xr-x. 3 67974 users 4096 October 7 httpfsdrwxr-xr-x. 5 67974 users 4096 February 23:19 mapreducedrwxr-xr-x. 4 67974 users 4096 October 7 toolsdrwxr-xr-x. 6 67974 users 4096 October 7 yarn

We can see that in the /usr/local/src/hadoop-2.2.0/share/hadoop/ directory, there are all the jar packages of Hadoop,

First, you need to import the jar package under the common directory.

[[email protected] hadoop]# cd common/[[email  Protected] common]# ll Total dosage  4152-rw-r--r--.  1 67974 users 2735584 10 Month   7 2013 hadoop-common-2.2.0.jar-rw-r--r--.  1 67974 users 1352859  10 month   7 2013 hadoop-common-2.2.0-tests.jar-rw-r--r--.  1 67974 users   139536 10 Month   7 2013 hadoop-nfs-2.2.0.jardrwxr-xr-x. 2 67974  users    4096 10 Month   7 2013 jdiffdrwxr-xr-x. 2  67974 users    4096 10 Month   7 2013 libdrwxr-xr-x. 2  67974 users    4096 10 Month   7 2013 sourcesdrwxr-xr-x. 2  67974 users    4096 10 Month   7 2013 templates[[email  protected] common]# 

We need to import all the jar packages under common and all the jar packages in the Lib directory in common

[[email protected] common]# cp *.jar/root/workspace/hadoop/lib/#导入common下的所有jar [[email protected] common]# CD Lib/[[em AIL protected] lib]# cp *.jar/root/workspace/hadoop/lib/#导入common下lib目录中的所有jar [[email protected] lib]#

Because the next main thing is to develop an HDFS program, you also need to import the relevant jar packages of HDFS

[[email protected] lib]# cd .. [[email protected] common]# cd .. [[email protected] hadoop]# cd hdfs/      #跳转到hdfs目录 [[email  Protected] hdfs]# ll Total dosage  7156-rw-r--r--.  1 67974 users 5242252 10 Month   7 2013 hadoop-hdfs-2.2.0.jar-rw-r--r--.  1 67974 users 1988460  October   7 2013 hadoop-hdfs-2.2.0-tests.jar-rw-r--r--.  1 67974 users    71689 10 Month   7 2013 hadoop-hdfs-nfs-2.2.0.jardrwxr-xr-x. 2 67974  users    4096 10 Month   7 2013 jdiffdrwxr-xr-x. 2  67974 users    4096 10 Month   7 2013 libdrwxr-xr-x. 2  67974 users    4096 10 Month   7 2013 sourcesdrwxr-xr-x. 2  67974 users  &nbsP; 4096 10 Month   7 2013 templatesdrwxr-xr-x. 7 67974 users     4096 10 Month   7 2013 WEBAPPS[[EMAIL PROTECTED] HDFS]# CP  *.jar /root/workspace/hadoop/lib/   #导入hdfs相关的jar包 [[email protected] hdfs]#

Well, the development needs of the jar package has been put into the project

Go back to the Eclipse interface, refresh the project, and refresh the jar. Because it is a Java project, so also to the Lib under all the jar package into a small bottle, this is not a stranger to believe that everyone.

Create a new package, Cn.hucc.hadoop, create a new Java file, Hadoopdemo.java. Final results:

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/59/E8/wKioL1TvHbXxGchPAARx2_ze7DE207.jpg "title=" Image 5.png "alt=" Wkiol1tvhbxxgchpaarx2_ze7de207.jpg "/>

Importing a source bundle for Hadoop

Now although it can be developed, but still need to import Hadoop source package, because as a high-quality programmer, see the source is essential (hehe ~). Click FileSystem class to view the source code, at this time must not see the source code, so need to download the source package and upload to Linux. Download and upload the hadoop-2.2.0-src.tar.gz to the Linux system using SECUREFX.

Unzip the source package into the /usr/local/src/resource directory (personal habits)

TAR-ZXVF hadoop-2.2.0-src.tar.gz-c hadoop-2.2.0-src.tar.gz

Then come back to eclipse to link to the directory, which is not difficult for people with Java development experience.

"Developing Hadoop Programs"

Write the following code in Hadoopdemo.java:

package cn.hucc.hadoop;import java.io.fileoutputstream;import  java.io.InputStream;import java.io.OutputStream;import java.net.URI;import  org.apache.hadoop.conf.configuration;import org.apache.hadoop.fs.filesystem;import  org.apache.hadoop.fs.path;import org.apache.hadoop.io.ioutils;public class hadoopdemo { Public static void main (String[] args)  throws Exception {URI uri  = new uri ("hdfs://hucc01:9000");//namenode addressconfiguration config = new  configuration (); Filesystem fs = filesystem.get (Uri, config); Inputstream is = fs.open (new  path ("/JDK")); Outputstream os = new fileoutputstream ("/root/jdk1.7.tar.gz"); Outils.copybytes (Is, os, 4096, true);}} 

Run the program, execute successfully, go back to the root directory and see if jdk1.7.tar.gz is lying quietly! Here, the first Hadoop program developed with Java program is finished, think of a little excitement!

Summary

Overall look down, in fact, there is not much difficulty, Hadoop seems very mysterious, really to learn when the feeling is so, did not learn the three frame when the feeling of the hard feelings!

Every day to work, come back to learn about 3 hours, adhere to, and blog down, I believe that the long-term past, Hadoop will always be my conquest ...

The first half of the graduation horse, has been doing a crud live, with those several frameworks, I do not want to always do, do not want to think of the yards, I have to learn big data, learn python, the final job of data mining is the king.

Struggle, youth, while still young now!

This article is from the "Mo" blog, please be sure to keep this source http://flycc258.blog.51cto.com/8624126/1615545

Using Java to call the Hadoop interface Learning notes