Spark Shuffle Optimization __spark

Spark.shuffle.file.buffer default value: 32k parameter description: This parameter is used to set the buffer buffer size for the bufferedoutputstream of the shuffle write task. Writing data to a disk file is written to the buffer buffer before it is written to the disk until the buffer is full. Tuning recommendation: If the available memory resources for the job, you can increase the size of this parameter (such as 64k), thereby reducing the number of shuffle in the write process, you can reduce the number of disk IO, and thus improve performance. In practice, it is found that the performance of 1%~5% can be improved by adjusting the parameters rationally. Spark.reducer.maxSizeInFlight default value: 48m parameter description: This parameter is used to set the buffer buffer size for the shuffle read task, which determines how much data can be pulled at a time. Tuning suggestion: If the available memory resources of the job, you can appropriately increase the size of this parameter (such as 96m), so as to reduce the number of pull data, you can reduce the number of network transmission, and thus improve performance. In practice, it is found that the performance of 1%~5% can be improved by adjusting the parameters rationally. Spark.shuffle.io.maxRetries Default value: 3 parameter Description: Shuffle read task automatically retries if the pull fails because of a network exception when it pulls data from the node that is the shuffle write task. This parameter represents the maximum number of times that can be retried. If pulling or not succeeds within a specified number of times, the job execution may fail. Tuning recommendation: For jobs that contain particularly time-consuming shuffle operations, it is recommended that you increase the maximum number of retries (such as 60) to avoid data pull failures due to factors such as the JVM's full GC or network instability. In practice, it is found that adjusting this parameter can greatly improve the stability of the shuffle process for the large amount of data (billions of ~ tens of billions of dollars). Spark.shuffle.io.retryWait Default value: 5s parameter Description: specifically explained above, this parameter represents the wait interval for each retry pull data, the default is 5s. Tuning suggestion: It is recommended to increase the length of the interval (such as 60s) to enhance the stability of the shuffle operation. Spark.shuffle.memoryFraction default value: 0.2 parameter Description: This parameter represents the proportion of memory in executor memory that is allocated to the shuffle read task for aggregation operations, by default, by 20%. Tuning suggestion: This parameter is explained in resource parameter tuning. If there is sufficient memory, and veryWith less persistence, it is recommended that you increase this ratio to operate more memory for the aggregation of shuffle read to avoid the frequent reading and writing of disks during aggregation due to insufficient memory. In practice, it is found that a reasonable adjustment of this parameter can improve the performance by about 10%. Spark.shuffle.manager default value: Sort parameter Description: This parameter is used to set the type of Shufflemanager. After Spark 1.5, there are three options: hash, sort, and tungsten-sort. Hashshufflemanager is the default option before Spark 1.2, but spark 1.2 and subsequent versions are Sortshufflemanager by default. Tungsten-sort is similar to sort, but uses the tungsten-heap memory management mechanism in the plan to be more efficient in memory usage. Tuning recommendation: Because Sortshufflemanager sorts data by default, you can use the default Sortshufflemanager if you need the sorting mechanism in your business logic, and if your business logic doesn't need to sort the data, Then it is recommended to refer to the following parameters tuning, through the bypass mechanism or optimized hashshufflemanager to avoid sorting operations, while providing better disk read and write performance. Note here that Tungsten-sort should be used with caution, because some of the corresponding bugs were found before. Spark.shuffle.sort.bypassMergeThreshold Default value: 200 parameter Description: When Shufflemanager is Sortshufflemanager, if shuffle read The number of tasks is less than this threshold (the default is 200), then the shuffle write process does not perform a sort operation, but instead writes the data directly in an hashshufflemanager way, In the end, however, all the temporary disk files produced by each task are merged into one file and a separate index file is created. Tuning suggestion: When you use Sortshufflemanager, if you do not need to sort operations, then it is recommended to increase this parameter larger than the number of shuffle read task. The bypass mechanism is automatically enabled at this time, and map-side is not sorted, reducing the performance overhead of sorting. But in this way, a lot of disk files will still be generated, so shuffle write performance needs to be improved. Spark.shuffle.consolidateFiles Default value: False parameter Description: If you use HASHSHUFFLEmanager, this parameter is valid. If set to true, the consolidate mechanism will be turned on, greatly merging the output files of shuffle write, which can significantly reduce disk IO overhead and improve performance for shuffle read tasks in particular. Tuning suggestion: If you do not need the Sortshufflemanager sorting mechanism, then in addition to using the bypass mechanism, You can also try to manually specify the Spark.shffle.manager parameter as a hash, using Hashshufflemanager, and opening the consolidate mechanism. In practice, it is found that its performance is higher than the sortshufflemanager of bypass mechanism. 10%~30%.