Hive MapJoin Mechanism

Hive MapJoin Mechanism

What is MapJoin?
MapJoin, as its name implies, is the connection between tables in the Map stage. The connection does not need to enter the Reduce stage. This saves a lot of data transmission in the Shuffle phase. Thus it plays a role in optimizing the job.

Principle of MapJoin:
That is, join is performed on the map side. The principle is broadcast join, that is, the small table is used as a complete driving table for join operations. Generally, the data in each table to be connected is distributed in different maps for processing. That is, the Value corresponding to the same Key may exist in different maps. In this way, you must wait until the Reduce node is connected. To make MapJoin smooth, the following conditions must be met: except that the data of a table is distributed in different maps, the data of other connected tables must be completely copied in each Map. MAPJION reads all small tables into the memory. In the map stage, MAPJION directly matches the data of another table with the data in the memory. Because the join operation is performed on the map, reduces the efficiency of reduce operations.

Applicable scenarios of MapJoin:
Applicable scenarios of mapjoin, such as join operations with a table that is very small and not equivalent. Through the above analysis, you will find that not all scenarios are suitable for MapJoin. it is usually used in the following scenarios: in the two tables to be connected, there is a large table, there is a small table, this small table can be stored in the memory without affecting performance. In this way, we copy the small table file to the local location of each Map task, and then let the Map read the file to the memory for use.

Implementation of MapJoin:
1) Use the static DistributedCache. addCacheFile () method in the Map-Reduce driver to add the small table file to be copied ,. JobTracker obtains the URI list before the job starts and copies the corresponding files to the local disks of each TaskTracker.
2) use the DistributedCache. getLocalCacheFiles () method in the setup method of the Map class to obtain the file directory, and use the standard file read/write API to read the corresponding file.

Hive version differences:
One of the built-in optimization mechanisms provided by Hive includes MapJoin.
Before Hive v0.7, you need to provide MapJoin instructions so that Hive can optimize MapJoin. Hive v0.7 and later versions are optimized without the MapJoin instruction. It is controlled by the following configuration parameters: hive> set hive. auto. convert. join = true; after hive 0.11, when the table size meets the settings (hive. auto. convert. join. noconditionaltask = true, hive. auto. convert. join. noconditionaltask. size = 10000, hive. mapjoin. smalltable. filesize = 25000000). By default, the join is converted to map join (hive. ignore. mapjoin. hint is true, hive. auto. convert. true for join). However, there are many map join bugs in hive0.11. You can disable map join convert by default and set hint: hive if necessary. auto. convert. join = false. Hive. ignore. mapjoin. hint = false. Hive v0.12.0 is enabled by default. That is, hive. auto. convert. join = true. Hive also provides another parameter-the table file size as the threshold for enabling and disabling MapJoin. Hive. mapjoin. smalltable. filesize = 25000000

Hadoop cluster-based Hive Installation

Differences between Hive internal tables and external tables

Hadoop + Hive + Map + reduce cluster installation and deployment

Install in Hive local standalone Mode

WordCount word statistics for Hive Learning

Hive operating architecture and configuration and deployment

Hive details: click here
Hive: click here

This article permanently updates the link address: