Shuffle Correlation analysis

Shuffle description is a process that shows many-to-many dependencies. Shuffle is the link between the map phase and the reduce phase, where each reduce task reads a piece of data from the data generated by the map task. Shuffle is usually divided into two parts: data preparation in the map phase and a copy of the data in the reduce phase.

The map phase determines the number of data shards for each map task output based on the amount of task in the reduce phase, which may be stored in memory or on disk, which may be in the form of one file per shard or multiple shards in a single data file. Plus a lasso to record the offset of each shard in the data file. (except for the narrow dependencies in the Rdd, which happens to be a one-on.)

1, Shuffle write

The shufflemaptask of shuffle output in spark creates buckets for each resulttask, The result of Shufflemaptask will be bucketid according to the partitionner of the set. Then fill in the corresponding bucket, so each shufflemaptask created bucket data is equal to the number of resulttask.

Shufflemaptask creates a bucket corresponding to a file on the disk that stores the results, This file has also been blockfile. The buffer configured through the SPARK.SHUFFLE.FILE.BUFFER.KB property is the one used to create the Fastbufferedoutputstream output stream. If the Spark.shuffle.consolidateFiles property is set to True in the configuration file, the bucket generated by Shufflemaptask does not necessarily correspond to a single file, but is part of the corresponding file. Doing so will significantly reduce the number of blockfile files produced.

2, Shuffle read

Spark can read data in two ways, one common socket and the other using the Netty framework. The Netty mode can be started by configuring the Spark.shuffle.use.netty property to True. Netty The framework, Blockmanager creates shufflesender specifically for sending data, and if the data required by resulttask happens to be in this node, it is read directly to the disk and is no longer available over the network. When MapReduce takes data, even if the data is on-premises, it is going to be transmitted over the network.

Shuffle Correlation analysis