BT source code learning experience (11): client source code analysis (Storage Management) Author: Wolfenstein this time analysis BT storage management. We know that Bt divides the resources to be shared into uniform chunks, and records the message digest values of each shard in the seed file to determine whether a Shard has been correctly downloaded during download. In addition, we can see that the size of other parts is the same unless it is the last part, therefore, it is very likely that the number of bytes at the beginning of a file belongs to a certain segment, and the number of bytes offset from the center belongs to another segment, or, if the file is small, a part contains several files. The storage management part of Bt shields these differences from other parts of the program, that is, for other parts, it only needs to be accessed by block. First, let's look at the filepool class, which is defined in multitorrent, that is, there is only one global. Therefore, it can ensure that the number of files opened on the hard disk when multiple seed files are downloaded is limited to a certain number. The following variables are maintained internally: handlebuffer is a list of all opened files. allfiles is a dictionary that records the owner of all files, that is, which file belongs to which seed file. The keyword is the file name of each file, and the value is the corresponding _ singletorrent object. Handles is a dictionary that records the relationship between file names and corresponding handles. whandles also describes which files are writable. Note that the corresponding handles are not stored in whandles, that is, if a file appears in handles, you can use handles to directly obtain its handle to avoid opening or closing the file multiple times. If it appears in whandles, it can still be written. Storage is defined in every _ singletorrent. Creating storage requires a list of files and their corresponding sizes. The file size information can be obtained from the metadata of the seed file. In addition, you must add each item in the Object List of the metadata of the seed object to the actual directory saved on the hard disk so that it can directly correspond to a specific object. When storage is created, it establishes a ing relationship between the file name and the global bytes, that is, list ranges. Each list item in this list is a triple, starting offset, ending offset, and file name. It indicates the file in the content of the seed file, from the nth byte to the nth byte. In addition, if the content of all information is divided into blocks in the seed file and the length of the block is set to piecelen, each segment has a byte offset, for example, from the first byte to the second byte piecelen (not included), it belongs to the first byte and then to the second byte. Therefore, this storage class must solve the conflict between downloading by Bt block and storing by file on hard disk. Here, I will try again to translate piece into parts because each part is further divided into several slice parts during the subsequent BT download process, and I am used to translating slice into slices. In storage, there are two private functions: _ intervals and _ get_file_handle. They provide two important functions for read and write. Read and Write are important interfaces provided by storage. The _ intervals task provides a global offset and length. A table is returned, indicating the number of bytes starting from the first few bytes of the files to access the data. In this way, you can use for... in_intervals (XX, XX) in read and write. While _ get_file_handle is used to obtain the actual handle of a file for reading and writing, the reason why a function must be used to obtain the file handle in storage is that the number of opened files must be limited. Therefore, you must deal with filepool in _ get_file_handle, when opening a file handle, you must maintain some variables in the filepool. Another function of storage is to read and write a "quick recovery" state file. It is only responsible for reading and writing a part of data in this file, which is read and written by the storagewrapper class. The storage class is responsible for reading and writing part of the file header, including 'bittorintoresumestatefile, version1', and total data volume, as well as the size and change time of each file. The storagewrapper class is defined in _ singletorrent. _ start_download. The provided interfaces must be more advanced. For example, it provides access by a certain block, and then obtains the actual byte offset by multiplying the block number by the size of a block internally, and then allows storage to read and write data. In addition, it maintains a bit array have of which blocks are locally owned to facilitate decision-making. There are two arrays indicating the block storage status, places and rplaces, they mean the first few pieces of data stored on the hard disk and the first few pieces of data stored on the hard disk are actually the first few pieces of data. This array is based on the abstraction of two parts of data: the first part of the data abstraction refers to the content (that is, shared resources) represented in the seed file as a continuous data, there are several pieces of data. The data abstraction in the second part refers to the object stored on the hard disk as a piece of continuous data, that is, the continuous storage space, which is also divided into several blocks. When all download tasks are completed, there should be 0 to self. numpieces-1 with places [I] = rplaces [I] = I. In the download process, because of certain strategies, it is not necessarily the first 0th blocks, and then the next 1st blocks, therefore, it is possible that the value in places [I] And rplaces [I] is not equal to I. Let's take a look at the bitfield, which is defined in BitTorrent/bitfield. py. It stores bits in the most economical space, that is, the number of BITs except 8 bytes. In addition, it implements _ setitem _ and _ getitem __, so that you can directly perform read and write operations on have [I] to complete the value operation. Note that there is assertval in the implementation of _ setitem _, which means that only one item in the array can be assigned to 1. This function is applicable to the scenario where a block is owned, that is, a block can only be available from nothing to gain or lose ". Storagewrapper also has a very important function, which is to split each piece of data into several slice again, and one slice is the smallest unit of data exchange between two peer customers through the network. On this basis, it is responsible for generating requests. inactive_requests stores all possible requests. When reading this part of the code, pay attention to the difference between 1 and L. during initialization, the value of inactive_requests [I] is 1, this indicates that a certain part is not yet available (so you can generate a network request for this). When you know that a part has been obtained, the value of inactive_requests [I] becomes none, the specific operation to be performed when you know that a piece has been obtained is markgot. It means that the first POS block of the piece block in the storage space on the hard disk is found. In addition, during initialization, call _ check_partial and _ make_partial to check a specific block to see which slice needs to be downloaded. Put these requests in inactive_requests. When other parts of the program decide to start the next part, storagewrapper will generate the corresponding network Request Parameters for it (the number of parts, the offset, new_request is used to complete this task. In addition, piece_came_in and get_piece provide data read/write operations. When calling them, you must specify the index (number of parts ), begin (intra-block offset), length (length, which is piece in piece_came_in, that is, the data itself, you can directly obtain its length ). Finally, we need to mention the storage management part, that is, an early BT version applied for the corresponding hard disk space at the beginning of the download. Now, the file size increases with the download process. However, the download sequence may not be as follows: First, 0th, and then 1st. Therefore, the storage sequence of files is different. When new data is downloaded to the hard disk, it is very likely that we need to adjust the starting point and try to make them "seated ". The _ move_piece function can move data, refer to the code called for _ move_piece at the beginning of piece_came_in to understand the process in which BT gradually sets the block sequence "seat check" during the download process.
