Mapreduce programming Series 5 mapreduce main process sorting

The previous four articles show you how to write a simple log extraction program, read all CSV log files in the HDFS share/logs directory, and then extract the data, output to the share/output directory.

This article will take a look at the main process and propose new improvement objectives.

First, declare that all the code is Maven project and does not use any IDE. This is my consistent programming style, developed using Emacs + jdee. You only need to learn how to use Maven in IDE.

Comparable serialization

The first is serialization, which is commonly used in various programming technologies. The special feature of mapreduce is that keys are used for sorting. All keys must support both serialization and deserialization, and also large-sized operations. Therefore, the writablecomparable interface is generally used, which is inherited from the writable interface and Java. Lang. Comparable interface. The former is responsible for serialization, implementing functions similar to stream, and the latter is responsible for comparison.

Mapreduce computing process

Here is a summary of the main steps:

1. PassInputformatRead all the lines of the log file in the HDFS directory and split the contents into blocks. Then each block corresponds to a mapper.

2. Call eachMapperThe map function of is used to convert the data in the content block into the <key, value> format by row and pass the data as parameters. the code of the map function is implemented by the programmer. Generally, the key is data, and the value is an integer, which facilitates statistics. In this way, the parameter <key, value> is changed to another <key, value> that conforms to the business logic. The context. Write method is used.

Write it out, and then the framework is handed over to Cer.

3.PartitionerAt present, my program does not implement its own class, but it simply uses CER, which will be added later.

4. The framework groups keys to form <key, values> pairs and callsReducerReduce function. The function receives the <key, values> passed by mapper and then performs statistics.

5. What format file is output fromOutputformatTo control.

Note that the preceding bold characters are five mapreduce components. Each component is a class that we can inherit, and then the mapreduce Framework calls back and forth the implementation method of our subclass through polymorphism.

Mapreduce job Configuration

With the above implementation, you also need to configure the job and submit it in the hadoop command line.

For configuration, you can directly create a job class and call the Set Method for corresponding settings. The parent class of a job is jobcontext.

You can set the top five component classes and replace them with your own classes. You can also set the number of reducers.

The analysis is here, and more programming practices will be carried out later.

For example, you can customize your three major components: inputformat, partitioner, and outputformat.

Mapreduce programming Series 5 mapreduce main process sorting