Optimization of mapreduce tasks in hadoop

1. Task Scheduling

(1) computing: hadoop will always give priority to assigning tasks to idle machines. All people can share system resources fairly.

(2) I/O: hadoop tries its best to allocate map tasks to the machine where input split is located to reduce network I/O consumption.

2. data preprocessing and input split size

(1) Before submitting a mapreduce task, you can perform a preprocessing of the data and merge the data to improve the execution efficiency of the mapreduce task. Refer to the execution time of map tasks. When the execution time of a map task is short, more data can be allocated. Set the input data size of map to adjust the running time of map.

3. Number of MAP and reduce tasks

(1) map task slots and reduce task slots. It refers to the maximum number of MAP/reduce tasks that the cluster can run simultaneously. The main reference for setting the map quantity of mapreduce tasks is the running time of map tasks, and the number of reduce tasks only needs to refer to the task slot settings.

4. Combine Functions

This is a function used to merge data locally. There is a lot of repeated data in the map. It is very time-consuming to transmit these records to reduce tasks one by one. Therefore, the mapreduce framework runs a combine function, used for local merge, which greatly reduces the consumption of network I/O operations. Job. setcombineclass (combine. Class );

5. When writing a mapreduce program, you can select to compress the map output and final output results (you can also select the compression mode ). In some cases, the intermediate output of map may be very large. compressing it can effectively reduce the amount of data transmitted over the network.

6. Custom compator

In hadoop, you can customize data types for more complex purposes. When you want to implement the K-means algorithm (a basic Clustering Algorithm), you can define a set of K integers.