Storage Management for Spark

The storage and management of the RDD is implemented and managed by Spark's memory management module. This paper introduces the storage management module of spark from two angles of architecture and function. Architectural Perspective

From the architecture perspective, the Storage Management module is divided into the following two layers: the communication layer: the Storage Management module adopts the master-slave structure to realize the communication layer, the master node and the Slave node transmit control information and state information. Storage layer: The storage Management module needs to store the data on the hard disk or in memory and, if necessary, to the remote, which is implemented by the storage layer and provides the corresponding interface. communication Layer Message delivery

In the communication layer of the storage Management module, the Blockmanager on each executor is only responsible for managing the data block original information owned by its own executor, and does not manage the data block meta information on other executor Whereas the driver-side Blockmanager has all the registered Blockmanager information and block meta-information, the blockmanager of executor often obtains the required non-local data by sending information to driver. Storage-tier architecture

The RDD is made up of different partitions, and the transformations and actions we perform are performed on each separate partition. In the storage Management module, the RDD is also considered to be composed of different data blocks, the access to the RDD is in the data Block unit, in essence the partition (partition) and data block (block) is equivalent, but the angle is different. At the same time, the smallest unit of data accessed in the Spark Storage Management module is the data block, and all operations are in chunks. Data blocks (block)

As mentioned in the previous section, the Storage Management module manages data in blocks, which are the smallest operating units in the storage Management module. Various blocks of data are managed in the storage Management module, which provides different capabilities for the spark framework, and several of the main blocks of data managed in the Spark storage Management module are: RDD data blocks: Used to store the cached RDD data. Shuffle data block: Used to store persisted shuffle data. Broadcast variable data block: Used to store the stored broadcast variable data. The task returns the result data block: The task returned results that are used to store inside the storage management module. Typically, the task returns results with the task returned to the driver side with the Akka. However, when the task returns a large result, it causes the Akka frame overflow, another option is to put the return result as a block in the storage management module, and then obtain the data block on the driver side, because the storage Management module internal Data block transfer is connected through the socket, Therefore, there will be no akka frame overflow. Streaming BLOCK: Used only in spark streaming to store the received streaming data block. from a functional perspective

From a functional point of view, the storage Management module can be divided into the following two main parts: the RDD cache: The entire Storage Management module works as a cache for the RDD, including memory-based and disk cache. Shuffle data persistence: Shuffle data for intermediate results is also managed by the storage management module. The performance of shuffle directly affects the overall performance of the spark application, so the processing of shuffle data in the storage Management module differs from the traditional RDD cache. Rdd Persistence

The storage Management module can be divided into two chunks, one is the cache of the RDD, and the other is the persistence of the shuffle data. The next step is to describe how the storage management module caches the RDD from memory and disk two. the relationship between the RDD partition and the data block

For the various operations of the RDD, such as the conversion operation, the operation, we put the operation function on the RDD, and eventually these operations will be executed on each partition, so to speak, all operations on the RDD are partition-based. In the storage Management module, we are often exposed to the concept of data block, in the storage management module for data access is in the data block units. Partitioning is a logical concept, and data blocks are physical data entities, the partitions and chunks of data that we manipulate, and what is the relationship between them. In this section we will describe the relationship between partitions and data blocks.

In spark, partitions and blocks of data correspond to one by one, a partition in an RDD corresponds to a block of data in the storage management module, the Storage Management module does not touch or care about the RDD, it only cares about the data blocks, and the mappings between the blocks and partitions are made by the conventions on the names.

The Convention on this name is established as follows: Spark maintains a separate ID number for each RDD within it, and a separate index number for each of the RDD's partitions, so as long as you know the ID number and index number, we can find the appropriate partition in the RDD, meaning "ID number + index number "You can determine this partition globally and uniquely. In this way, the "ID number + index number" as the name of the block naturally establishes the partition and block mapping.

When we show that the call function caches the rdd we need, Spark creates a mapping between the RDD partition and the data block within it, and when we need to read the cached Rdd, we can get the block corresponding to the partition from the storage management module according to the mapping relationship mentioned above. The following figure shows the mapping between the RDD partition and the data block.

Internal Cache

When an RDD is cached with default or memory-based persistence, the data block for each partition in the RDD is managed by the memory store in the storage Management module. The In-memory cache maintains a hash table with the data block name key and the block content as a value.

An important issue in the memory cache is what to do if the memory is not or has reached the threshold you have set. In Spark, there is a configuration for memory thresholds that can be used in the memory cache: Spark.storage.memoryFraction. By default it is 0.6, which means that 60% of the JVM's memory can be used by the memory cache to store block content. When we store data blocks that occupy more than 60% of memory, Spark takes some policy to free up memory cache space: Discard some chunks of data, or store some chunks of data on disk to free up memory cache space. is discarded or stored on disk, depending on the persistence options of the data blocks that are being manipulated, if the persistence option contains disk caches, the blocks are cached in the disk, and vice versa.

So does the immediate removal affect the Spark program's error recovery mechanism? This depends on the traceability of the dependency, and if the RDD depends on the ancestor Rdd that can be traced back and available, then the corresponding block of the RDD is deleted without affecting the error recovery. Conversely, if the RDD is already an ancestor Rdd and the data cannot be traced back, then the program will go wrong. Lost executor error is the reason.

As can be seen from the above introduction, the memory cache for the management of the data block is very simple, essentially a hash table plus some access policies. Disk Cache

The disk cache manages blocks of data in such a way that, first, the blocks are stored in a specific directory on disk. When we configure Spark.local.dir, we configure the directory where the storage Management module disk cache holds data. When the disk cache is initialized, the Spark disk cache folder is created under these directories, and the folder is named: Spark-local-yyyymmddhhmmss-xxxx, where xxxx is a random number. All block content is stored in these created directories.

Second, in the disk cache, a data block corresponds to a file in the file system, and the mapping of the file name and the block name is computed by the hashing algorithm.

In summary, the file path for the data block is: dirid/subdirid/block_id. This allows us to establish a correspondence between the block and the file, and the access block content becomes the writing and reading of the corresponding file. Persistence Options

The cached data block is fault-tolerant, and if one of the RDD's partitions is lost, he is automatically re-acquired through the inheritance relationship.

For the persistence of the RDD, Spark gives us different options that allow us to persist the RDD to memory, disk, or to serialize to memory, and set up multiple copies to be stored between different nodes in the cluster. All of these different storage policies are determined by different persistence options. Shuffle Data Persistence

The storage Management module can be divided into two chunks, one is the cache of the RDD, and the other is the persistence of the shuffle data. After the RDD cache is introduced, shuffle data persistence is described next.

The following diagram illustrates the process of shuffle operations in spark

First, each map task creates the corresponding bucket based on the amount of data in the reduce task, the number of buckets is m*r, where M is the number of map tasks, and R is the number of reduce tasks.

Second, the result of the map task is populated into each bucket based on the partitioning algorithm that is set. The partitioning algorithm here is customizable, and the default algorithm is, of course, based on the key hash to a different bucket.

When the reduce task starts, it processes the bucket as input to the task from the remote or local storage management module based on the ID of its task and the ID of the map task it depends on.

The difference between the shuffle data and the RDD persistence is that the shuffle data block must be cached on disk, not in-memory, and in the RDD disk-based persistence, each chunk corresponds to a file, and in shuffle data block persistence, There are two ways to store shuffle data blocks: One is to map shuffle blocks of data into files, which is the default, and the other is to map shuffle blocks of data into a section of a file. This way you need to set Spark.shuffle.consolidateFiles to true.

The default way is to generate a large number of files, such as 1000 map tasks and 1000 reduce tasks, will produce 1 million shuffle files, which will have a significant impact on disk and file system performance, so there is a second way to implement, The shuffle data blocks resulting from the time-sharing map task are merged into the same file to reduce the total number of shuffle files. For the second way of storage, the schematic is as follows:

The access to shuffle data blocks is described earlier, and we describe the reading and transmission of shuffle data blocks. Shuffle is a process of reorganizing the output of a set of tasks as input to the next set of tasks, because the tasks are distributed on different nodes, so in order to use the reorganization results as input, it is necessary to involve the reading and transmission of the shuffle data.

In the Spark storage Management module, there are two ways to read and transmit shuffle data: one is to obtain data based on a socket connection in NIO, and the other is to obtain data based on OIO through the Netty server.

The former is the default fetch method, and by configuring Spark.shuffle.use.netty to True, you can enable the second method of fetching. There are two ways to get shuffle data, because the default mode in some cases does not make full use of network bandwidth, users can compare the performance differences between the two ways to decide which shuffle data acquisition method.

In general, the Spark Storage Management module has done many things that are different from RDD persistence for shuffle data persistence, including how to access shuffle data blocks, and how to read and transmit shuffle blocks of data. All of these implementations are designed to enable shuffle to achieve better performance and fault tolerance.