TFS data server startup Optimization

The TFS Data Server (DS) records the correspondence between the Logical Block and the physical block by storing a blockprefix structure in the header of the physical block. When DS is used, it reads every block that has been used, read the block header information and report it to the NS. The NS establishes a ing relationship between the Logical Block and the DS through the information. Initially, TFS uses a 1500 GB small disk. Each primary block is 64 MB and each extended block is calculated at 2 MB. The total number of physical blocks is close, even if all these blocks are used (the utilization of the extended blocks is not too high under normal conditions), loading these physical blocks will be less than 30 s, which is tolerable, however, at present, TFS uses 2 TB disks, and the total number of physical blocks is about. The loading time will linearly expand with the number of blocks. When all the blocks are used, the loading process may take up to ten minutes, which is intolerable for the server and must be optimized.

 

The ds startup process mainly takes time in the blockfilemanager: Bootstrap Process, which executes the following tasks:

1. Load the super block load_super_blk

2.Load the data block load_block_file Based on the super block information(The main time is spent here)

• Traverse the master block set in the middle of the super block normal_bitmap
• Read the blockprefix of the main block header, and load all the extended blocks in a chain accordingly.
• Load the index of the Logical Block. During the loading process, the blockinfo of the index header is read and checked with the information of the physical block header.

During the load_block_file process, for each logical block, you must first read the header information of the corresponding physical block (one main block, multiple expansion blocks) and establish the correspondence between the Logical Block and the physical block, then MMAP corresponding index information.

The chain loading of logical blocks is based on the blockprefix of the block header. The blockprefix structure is as follows:

Struct blockprefix
{
Uint32_t logic_blockid _; // logical block number corresponding to the physical block
Uint32_t prev_physic_blockid _; // block number of the next physical block
Uint32_t next_physic_blockid _; // The block number of the last physical block. 0 indicates the last block.
};

Index loading includes open and MMAP operations, and reads blockinfo information mapped to the memory area header for reporting to ns.

 

Comparison and test: physical block loading and IndexWho is more time-consuming to load?

Test environment: 2 TB disks close to full, with 22134 logical blocks on DS

1. When DS is started normally, it takes about 14 minutes to start. (Assume that each physical block uses an extended block, and the number of random reads should be 2 of the number of logical blocks.Times)
2. Clear the page cache, use vmtouch to load the index data to the memory, and then start Ds. the startup time is about 13 minutes.
3. Modify the code. Change the cycle in load_block_file to two times to load the block and index, clear the page cache, and start Ds, the start time is about 13 min + 6 S (6 s is the time when the index is loaded in a centralized manner ).

From the test above, we can see that,The time is mainly used for loading physical blocks.Because the space of the physical block is pre-allocated, the random read operation is triggered when the block header is constantly read.

Optimization Scheme

Will be scatteredBlockprefixCentralized storage in a file (because the number of physical blocks is determined, the file size is determined, and the file space can be allocated in advance ), during the loading process, only the file is read in a centralized manner to establish the correspondence between the Logical Block and the physical block, thus reducing a large number of random reads.

The header loading and updating of a block are completed through the interfaces load_block_prefix and dump_block_prefix. To use the new scheme, you need to modify the implementation of these two interfaces.BlockHeader readSwitch fromSeparate file reading,If you find that the blockprefix file is not stored separately, use the original solution to start it.

Supporting Tool changes

1. Create a file to store blockprefix in the format file system.
2. A tool is added to generate a separate blockprefix File Based on the block header, which is used to enable the active DS instance in a new solution.

Optimization results

The centralized storage blockprefix solution is used to test the running DS and the DS created using the new solution. The ds startup time is within 10 s, and DS can normally complete read/write service requests.