Getting started with hbase MapReduce program example

1. First, let's look at a standard hbase as an example of Data Reading source and output source:

View Code JAVA

12345678

Configuration conf = HBaseConfiguration.create();Job job = new Job(conf, "job name ");job.setJarByClass(test.class);Scan scan = new Scan();TableMapReduceUtil.initTableMapperJob(inputTable, scan, mapper.class,Writable.class, Writable.class, job);TableMapReduceUtil.initTableReducerJob(outputTable, reducer.class, job);job.waitForCompletion(true);

First, create the configuration information and job object, and set the job class. Like normal mapreduce, the only difference is the description of the data source, which is implemented by the initTableMapperJob and initTableReducerJob methods of TableMapReduceUtil.

Use the code above:
The data input source is the hbase inputTable table, and the mapper. class is executed for the map process. The output key/value type is
ImmutableBytesWritable and Put types. The last parameter is the job object. It should be noted that a scan should be declared to read the object scan, and the number of table scan reads should be calculated.
Data is used. You can configure parameters for scan.
The data output target is the outputTable table of hbase, the reduce process that the output executes is the CER Cer. class, and the operation job target is job. Comparison with map
The annotation of the output type is missing because they are not necessary. After reading the source code, you will know the write (key, value) in the TableRecordWriter of mapreduce)
The key value is not used in the method. The value can only be Put or Delete. The write method automatically determines that it is not specified by the user.

Next is the mapper class:

View Code JAVA

1234567891011

Public class mapper extendsTableMapper <KEYOUT, VALUEOUT> {public void map (Writable key, Writable value, Context context) throws IOException, InterruptedException {// context er logical Context. write (key, value );}}}

It inherits the TableMapper class provided in hbase. In fact, this class is also the inherited MapReduce class. The following two generic Parameters specify that the type is the data type exported by Er er. This type must inherit from the Writable class, such as put and delete. It must be noted that the data type specified with the initTableMapperJob method remains unchanged. This process automatically reads data from one row in the specified hbase table for processing.

Then the CER class:

View Code JAVA

12345678

Public class countUniteRedcuer extends <KEYIN, VALUEIN, KEYOUT> {public void reduce (Text key, Iterable <VALUEIN> values, Context context) throws IOException, InterruptedException {// CER logic context. write (null, put or delete );}}

CER inherits the TableReducer class. Specify three generic parameters. The first two must correspond to the output key/value type of the map process, and the third must be put or delete. When writing, you can write the key null, which is unnecessary. In this way, the data output by CER is automatically inserted into the table specified by outputTable.

2. Sometimes the data source is hdfs text and the output object is hbase. At this time, the change is also very simple:

View Code JAVA

1234567891011

Configuration conf = HBaseConfiguration.create();Job job = new Job(conf, "job name ");job.setJarByClass(test.class); job.setMapperClass(mapper.class);job.setMapOutputKeyClass(Text.class);job.setMapOutputValueClass(LongWritable.class);FileInputFormat.setInputPaths(job, path); TableMapReduceUtil.initTableReducerJob(tableName,reducer.class, job);

You will find that you only need to specify the mapper execution class and output key/value type, specify the FileInputFormat. setInputPaths data source path, and the output Declaration remains unchanged, just like the normal mapreduce job declaration process. The command declaration process for reading data from hdfs text and outputting data to hbase is completed. Mapper and reducer are as follows:

View Code JAVA

1234567891011121314

Public class mapper extends Mapper <LongWritable, Writable, Writable> {public void map (LongWritable key, Text line, Context context) {// mapper logical context. write (k, one) ;}} public class redcuer extendsTableReducer <KEYIN, VALUEIN, KEYOUT> {public void reduce (Writable key, Iterable <Writable> values, Context context) throws IOException, interruptedException {// CER logic context. write (null, put or delete );}}

The ER er still inherits the mapper in the original MapReduce class. Also note that the key/value of the data type is always the same.

3. Read from hbase tables as data sources and hdfs as data output. The following is a simple example:

View Code JAVA

12345678910

Configuration conf = HBaseConfiguration.create();Job job = new Job(conf, "job name ");job.setJarByClass(test.class);Scan scan = new Scan();TableMapReduceUtil.initTableMapperJob(inputTable, scan, mapper.class,Writable.class, Writable.class, job);job.setOutputKeyClass(Writable.class);job.setOutputValueClass(Writable.class);FileOutputFormat.setOutputPath(job, Path);job.waitForCompletion(true);

Mapper and reducer are as follows:

View Code JAVA

12345678910111213141516171819202122

Public class mapper extendsTableMapper <KEYOUT, VALUEOUT> {public void map (Writable key, Writable value, Context context) throws IOException, InterruptedException {// context er logical Context. write (key, value) ;}} public class CER extendsReducer <Writable, Writable> {public void reducer (Writable key, Writable value, Context context) throws IOException, interruptedException {// CER logic context. write (key, value );}}}

Finally, let's talk about the nature of TableMapper and TableReducer. In fact, these two classes are designed to simplify code writing, because the four generic parameters passed in have fixed parameter types, therefore, it is the simplified version of Mapper and Reducer. In essence, there is no difference between them. The source code is as follows:

View Code JAVA

1234567

public abstract class TableMapper<KEYOUT, VALUEOUT>extends Mapper<ImmutableBytesWritable, Result, KEYOUT, VALUEOUT> {} public abstract class TableReducer<KEYIN, VALUEIN, KEYOUT>extends Reducer<KEYIN, VALUEIN, KEYOUT, Writable> {}

Now, you can write the first wordcount hbase mapreduce program.

Ps: For more information about read/write optimization, see the following article.