Hadoop performs join operations on multiple data tables

When using hadoop today, it is difficult to merge and join several large tables with certain relationships. However, after careful analysis, it is better to solve the problem. In addition, this is a very common requirement for massive data processing. So write it down and share it with you. If there is a better way to do this, we can also discuss it. Welcome to shoot bricks, haha.

The following two types of data are required:

Data

Key value1 value2 value3. .. valuen

Data B

Value1 A1 A2 a3...

Value2 B1 B2 b3... bn

...

Valuen X1 X2 x3. .. XN

Target Data

Key value1 A1 A2 a3... an value2 B1 B2 b3... bn... valuen X1 X2 x3.... XN

That is, to merge all datasets into one row. Of course, all the data is massive, and the data on terabytes cannot be merged on a single machine. So I thought of a way to use two rounds of mapreduce ON THE hadoop cluster to complete the process. The following is a method attached. Thank you for choosing one.

Map in the first round

Use valuen as the key, red as the key, and the whole row of data as the value, that is

Value11 key value1 value2 value3. .. valuen

Value21 key value1 value2 value3. .. valuen

...

Valuen1 key value1 value2 value3. .. valuen

Value12 A1 A2 a3...

Value22 B1 B2 b3... bn

...

Valuen2x1X2 x3. .. XN

First round of Reduce

After the map stage, the standard input data should be in the following format. (Why don't I need to talk about it? If you don't understand it, we suggest you first look at the mapreduce mechanism. For more information, see the previousArticle)

Value11 key value1 value2 value3. .. valuen

Value12 A1 A2 a3...

Value21 key value1 value2 value3. .. valuen

Value22 B1 B2 b3... bn

...

Valuen1 key value1 value2 value3. .. valuen

Valuen2x1X2 x3. .. XN

Merge

The following result is displayed.

Key value1 value2 value3. .. valuenValue1 A1 A2 a3...

Key value1 value2 value3. .. valuenValue2 B1 B2 b3... bn

...

Second round of Map

ContinueKey value1 value2 value3.. valuen Integer Behavior key, Print Output

Key value1 value2 value3. .. valuenValue1 A1 A2 a3...

Key value1 value2 value3. .. valuenValue2 B1 B2 b3... bn

...

Second round of Reduce

After directly merging the above results, sort out the output format to get the expected data.

 

After thinking for a long time, I did not find a better way to complete the process in less than two rounds of Map-reduce. In this article, I hope you can give suggestions. Thank you.