Hadoop learning notes (2): Data streams from map to reduce

A mapreduce job is a unit of work that you want to execute. It includes input data, mapreduce programs, and configuration information. Hadoop runs a job by dividing it into tasks (MAP tasks and reduce tasks.

There are two types of nodes used to control the execution of each job:JobtrackerDistribute tasks to variousTasktrackerS to run and coordinate all jobs running on the system. Tasktrackers runs the task and reports the progress to jobtracker. jobtracker maintains the global progress of each job. If a task fails, jobtracker resends the task to another tasktracker (that is, jobtracker is not only responsible for global job scheduling, but also responsible for the maintenance of all tasks of all jobs, this problem leads to mapreduce2 -- yarn, which will be detailed later ).

Hadoop divides the input of a mapreduce job into fixed-size partitions, which are called input splits or splits. Hadoop creates a map task for each split. In this way, the concurrent processing feature of hadoop can be well utilized. If a node is fast in processing, we can allocate more splits to it. However, the more splits, the better. If splits is too small, the additional overhead for managing splits and creating map tasks will account for a large proportion of the job execution time. For most jobs, the default size of a Good splits is the block size of HDFS, Which is 64 MB (this value can be changed at the cluster level, which will affect all new files in the future; you can also set a specific overload value for each newly created file ).

If the map task node runs on a redundant backup of input data (replica ), in this case, hadoop can achieve its maximum performance (because no additional data transmission is required to save time), which is called Data Locality optimization (Local Data Optimization ). However, sometimes this is not ideal, because it is possible that the node storing replica is running other tasks at full capacity at this time, the task will be distributed to other nodes to run, then, replica is retrieved from the nearest node where replica is stored. In general, there are three situations:

Data-local (A), rack-local (B), and off-Rack (c) map tasks

This is the reason why the size of splits should be exactly equal to the size of the block, because this is the maximum data size that HDFS can ensure on a node, if the size of splits exceeds the size of the block, it will occupy two blocks. The probability that the two blocks are stored on one node at the same time is very small, in this way, network transmission is inevitable, which increases the execution time.

Map tasks writes their output data to the local file system instead of HDFS. The reason for this is that most map tasks output intermediate data: it will be processed by the reduce task to generate the final output, and the intermediate data will be deleted once the job is completed successfully, if you store them on HDFS, it will cause a lot of extra costs. If the node running the map task runs on the machine before the reduce task removes the intermediate data output by the map task (correspondingly, the intermediate data is lost ), hadoop automatically restarts the map task on another node to regenerate the intermediate data.

Reduce masks does not have the advantage of Data localization, because the input data of a single reduce task is usually output by all mapper. Therefore, the sorted map output must be transmitted to Cer CER over the network. Reduce outputs are generally stored in HDFS for reliability (the first replica is stored on the local node, and the other two are stored in a different rack from the local node ).
The data flow of only one reduce task is as follows (the dotted arrow shows the local data flow, and the solid arrow shows the data flow between clusters ):

The number of reduce tasks is not determined by the size of input data, but by the user. When there are many reducers, map tasks will partition their output data and create a partition for each CER Cer ). A partition may contain multiple keys (and the values associated with them), but the record corresponding to each given key (Records) is stored in a partition. You can specify the partition method used for partitioning. However, a default partition or is usually used. It uses the hash function to partition records. Data streams with multiple reducers are shown as follows:

We can also see why data streams from map to reduce are called shuffle (meaning "shuffling, chaos"), because each reduce input is output by many maps. The shuffle stage is far more complex than the demo. If the shuffle stage is well tuned, the execution time of the job will be greatly affected. The details will be discussed later.

Reprinted please indicate the source: http://www.cnblogs.com/beanmoon/archive/2012/12/06/2805636.html