Hadoop example program code

Hadoop example code:

1. creatinga configuration object: to be able to read from or write to HDFS, you need tocreate a configuration object and pass configuration parameter to it usinghadoop configuration files.

ImportOrg. Apache. hadoop. conf. configuration;

ImportOrg. Apache. hadoop. fs. path;

 

Public
ClassMain {

Public
Static voidMain (string [] ARGs ){

Configurationconf =NewConfiguration ();

Conf. addresource (NewPATH ("C: \ documentsand Settings \ mz50947 \ Desktop \ hadoop-1.2.1 \ conf \ core-site.xml "));

Conf. addresource (NewPATH ("C: \ documentsand Settings \ mz50947 \ Desktop \ hadoop-1.2.1 \ conf \ hdfs-site.xml "));

System.Out. Println (CONF );

}

}

Output:

Configuration: core-default.xml, core-site.xml, C:/Documents andsettings/mz50947/desktop/hadoop-1.2.1/CONF/core-site.xml, C:/Documents andsettings/mz50947/desktop/hadoop-1.2.1/CONF/hdfs-site.xml

 

2. addingfile to HDFS: Create a filesystem object and use a file stream to add a file.

ImportJava. Io. bufferedinputstream;

ImportJava. Io. file;

ImportJava. Io. fileinputstream;

ImportJava. Io. ioexception;

ImportJava. Io. inputstream;

 

ImportOrg. Apache. hadoop. conf. configuration;

ImportOrg. Apache. hadoop. fs. fsdataoutputstream;

ImportOrg. Apache. hadoop. fs. filesystem;

ImportOrg. Apache. hadoop. fs. path;

 

Public
ClassMain {

Public
Static voidMain (string [] ARGs)
ThrowsIoexception {

Configurationconf =NewConfiguration ();

Conf. addresource (NewPATH ("C: \ documentsand Settings \ mz50947 \ Desktop \ hadoop-1.2.1 \ conf \ core-site.xml "));

Conf. addresource (NewPATH ("C: \ documentsand Settings \ mz50947 \ Desktop \ hadoop-1.2.1 \ conf \ hdfs-site.xml "));

System.Out. Println (CONF );

Filesystem = filesystem.Get(CONF );

// Check ifthe file already exists

Path =NewPATH ("C :\\ documents andsettings \ mz50947 \ Desktop \ output \ file. Ext ");

If(Filesystem. exists (PATH )){

System.Out. Println ("file" +
Path + "already exists ");

Filesystem. Delete (path,
True);

}

 

// Create anew file and write data to it.

Fsdataoutputstream out = filesystem. Create (PATH );

Inputstream in =NewBufferedinputstream (NewFileinputstream (NewFile ("C: \ documentsand
Settings \ mz50947 \ Desktop \ hadoop-1.2.1 \ conf \ hdfs-site.xml ")));

 

Byte[] B =
New byte[1024];

IntNumbytes = 0;

While(Numbytes = in. Read (B)> 0 ){

Out. Write (B, 0, numbytes );

}

 

// Closeall the fileDescripters

In. Close ();

Out. Close ();

Filesystem. Close ();

}

}

Will create file. file. ext and file under folder output

3. deployto Linux,

(1). c: \ Documents and Settings \ mz50947 \ Desktop> pscpc: \ eclipse \ workspaces \ 3.7 _ may_20 _

Dashboard \ hadoop_begin \ bin \ Main.classyj70978@retailvm1d.nam.nsroot.net:/home/YJ

70978/hadoop/hadoop-1.1.2/src/test

(2)./export/opt/jrockit/6.0 _ 14r27. 6.5l64/bin/jar-CVF main. jar-cwordcount_classes /.

(3)./home/yj70978/hadoop/hadoop-1.1.2/bin/hadoop JAR/home/yj70978/hadoop/hadoop-1.1.2/src/test/Main. Jar main