Hadoop example program code

Hadoop example code:

1.      Creatinga configuration object: to be able to read from or write to hdfs, you need tocreate a configuration object and pass configuration parameter to it usinghadoop configuration files.

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.Path;

 

public
class Main {

     public
static void main(String[] args){

      Configurationconf = newConfiguration();

      conf.addResource(new Path("C:\\Documentsand Settings\\mz50947\\Desktop\\hadoop-1.2.1\\conf\\core-site.xml"));

        conf.addResource(new Path("C:\\Documentsand Settings\\mz50947\\Desktop\\hadoop-1.2.1\\conf\\hdfs-site.xml"));

        System.out.println(conf);

     }

}

Output:

Configuration:core-default.xml, core-site.xml, C:/Documents andSettings/mz50947/Desktop/hadoop-1.2.1/conf/core-site.xml, C:/Documents andSettings/mz50947/Desktop/hadoop-1.2.1/conf/hdfs-site.xml

 

2.      Addingfile to hdfs: create a filesystem object and use a file stream to add a file.

import java.io.BufferedInputStream;

import java.io.File;

import java.io.FileInputStream;

import java.io.IOException;

import java.io.InputStream;

 

import org.apache.hadoop.conf.Configuration;

importorg.apache.hadoop.fs.FSDataOutputStream;

import org.apache.hadoop.fs.FileSystem;

import org.apache.hadoop.fs.Path;

 

public
class Main {

     public
static void main(String[] args)
throws IOException{

      Configurationconf = newConfiguration();

      conf.addResource(new Path("C:\\Documentsand Settings\\mz50947\\Desktop\\hadoop-1.2.1\\conf\\core-site.xml"));

        conf.addResource(new Path("C:\\Documentsand Settings\\mz50947\\Desktop\\hadoop-1.2.1\\conf\\hdfs-site.xml"));

        System.out.println(conf);

        FileSystem fileSystem = FileSystem.get(conf);

       

        // Check ifthe file already exists

        Path path = new Path("C:\\Documents andSettings\\mz50947\\Desktop\\output\\file.ext");

        if (fileSystem.exists(path)){

            System.out.println("File" +
path + "already exists");

            fileSystem.delete(path,
true);

        }

 

        // Create anew file and write data to it.

        FSDataOutputStream out =fileSystem.create(path);

        InputStream in = new BufferedInputStream(new FileInputStream(new File("C:\\Documentsand
Settings\\mz50947\\Desktop\\hadoop-1.2.1\\conf\\hdfs-site.xml")));

 

        byte[] b =
new byte[1024];

        int numBytes = 0;

        while ((numBytes = in.read(b)) > 0) {

            out.write(b, 0, numBytes);

        }

 

        // Closeall the file descripters

        in.close();

        out.close();

        fileSystem.close();

     }

}

Will create file .file.ext and file under folder output

3.      Deployto linux,

(1). C:\Documents and Settings\mz50947\Desktop>pscpC:\eclipse\workspaces\3.7_MAY_20_

dashboard\Hadoop_Begin\bin\Main.classyj70978@retailvm1d.nam.nsroot.net:/home/yj

70978/hadoop/hadoop-1.1.2/src/test

(2). /export/opt/jrockit/6.0_14R27.6.5l64/bin/jar -cvf Main.jar -Cwordcount_classes/ .

(3). /home/yj70978/hadoop/hadoop-1.1.2/bin/hadoop jar/home/yj70978/hadoop/hadoop-1.1.2/src/test/Main.jar Main