The execution process of hive query on MapReduce

The hive query is first converted to a physical query plan, and the physical query plan typically contains multiple mapreduce jobs, and the output of one mapreduce job can be used as input to another mapreduce job. The MapReduce job designed by Hive for hive queries has a fixed pattern: The Mapper class is the Org.apache.hadoop.hive.ql.exec.execmapper,reducer class Org.apache.hadoop.hive.ql.exec . Execreducer. The map process may have different inputformat for the MapReduce job, which InputFormat is determined when the physical query plan is generated. A operator tree is generated in the Execmapper initialization method, and the root is a Mapoperator object. In the Execmapper map method, the value of the Key-value pair from InputFormat is distributed to the root node of the operator tree, the Mapoperator object. Mapoperator converts the received value to a record (row) and passes it down the tree down to each subordinate operator processing, and the record (Row) may be filtered during delivery, Converting or attaching a key (the record itself is serialized as value) is output to the following reduce process. The reduce process , which is output from the map process, key-value the reduce method passed to Execreducer after the MapReduce framework is grouped by key. In the initialization method of Execreducer, a operator tree is also generated, and the groups Key-value passed to the reduce method are passed to the operator tree by order. Pass each set of Key-value to the start and end of each operator in the operator tree is notified.