EMule source code parsing-2

EMule source code parsing-2 2006-10-16

1. Block Mechanism-ensure the correct transmission of resources. To speed up content delivery, block processing is a simple and effective method. EMule processes each file in blocks. In addition, if the hash value of each part is retained, the validity of the downloaded content can be determined only when a part of the file is downloaded.

When obtaining information about each shared file, eMule performs multipart processing on it. Therefore, if you need to know the multipart processing and Restoration Mechanisms in eMule, check cknownfile :: just implement the createfromfile function.

The classes involved in this function and related to block processing and hash computing are all in shahashset. cpp and shahashset. h.

The following describes several main classes:

The caichhash class is only responsible for one hash value and provides basic operations such as direct assignment and comparison between two caichhash classes. Caichhashalgo is a common interface of the hash algorithm. Other hash algorithms can be used as long as they are implemented. This makes it easy to use different hash algorithms to calculate hash values. Caichhashtree is a tree-based hash value organization. It has a member variable of the left and right subtree, and its type is a pointer to caichhashtree. This is a typical method to implement the tree structure. Caichhashset contains a variable of the caichhashtree type, which is directly responsible to cknownfile, representing the block information of a file. The comments at the beginning of the shahashset. h file explain the way it is segmented. Here two constants 9728000 and 184320 are used, which are 180 K and K respectively. This is two different partitioning methods in eMule. That is, a large file is first divided into several k blocks and these blocks are organized into a tree structure, then each such block is broken down into several k blocks (52 blocks, plus a K block), which are still organized according to the tree structure. The final structure is still a tree. Cknownfile: The createfromfile method is used to gradually build such a tree when reading the content of the target file. Caichhashtree: findhash can locate the corresponding branches (a pointer to caichhashtree) based on the offset of the read target file and the size of the next part ). If necessary, the branches are automatically created. Therefore, during the multipart operation, the entire caichhashtree is created when the file is read from start to end, and the corresponding hash value of the multipart is also assigned.

At last, we need to pay attention to the hashlist variable in the cknownfile class. That is to say, it also separately retains the hash value of the md4 Algorithm for all blocks in 9728000 bytes. In this way, there are two sets of block verification mechanisms for a file, which can adapt to the network infrastructure in different scenarios-the network infrastructure Infrastructure MFC already has some network infrastructure classes, for example, casyncsocket. However, in the design of eMule, in order to develop network-related code more efficiently, other classes have been built as the infrastructure, and the code of these basic runtime classes has a high value for reuse.

The first is the casyncsocketex class. The features of this class have been described in asyncsocketex. h. It is fully compatible with the casyncsocket class, that is, the casyncsocket in the application is replaced with casyncsocketex, the program can still be the same as the original function, so it is more convenient to use. However, on this basis, it is more efficient, mainly in the message distribution mechanism, that is, it processes socket-related messages more efficiently than the original MFC casyncsocket class.

In addition, the casyncsocketex class supports dividing a socket into several layers by implementing the casyncsocketexlayer class, which makes it easy to implement many network functions, such as setting proxy, or use SSL for encryption.

In addition, the throttledcontrolsocket class and throttledfilesocket class defined in throttledsocket. h define only two interfaces. To enable the speed limit function for any other network socket class, you only need to cache the data in the default sending function (such as send or sendto) instead of sending data, then, when the sendfileandcontroldata or sendcontroldata method in the throttledcontrolsocket or throttledfilesocket interface is implemented, the data is actually sent out, so that the upload speed is limited, which also requires the uploadbandwidththrottler class to cooperate, uploadbandwidththrottler is a subclass of winthread. It runs a thread separately. Next time, we will detail how it controls the global upload speed. Network Infrastructure-Global speed limiter uploadbandwidththrottler is a global upload speed limiter used in eMule. It inherits the cwinthread class and starts to run independently when the class is created. When the class is destructed, the corresponding thread is automatically stopped. The target function of this thread is runproc. To prevent the runproc function from using the this pointer, it uses runinternal to actually complete the work of the working thread. In eMule, another class lastcommonroutefinder has a similar structure. Several socket queues are stored in uploadbandwidththrottler. The processing methods of these queues are slightly different. All queuing classes in the standard Queue (m_standardorder_list) Implement the throttledfilesocket interface. Generally, these classes can transmit file content or control information.

The other four queues are queues that implement the throttledcontrolsocket interface class. The classes in these queues mainly focus on transmission control information.

These four queues have a temporary high priority, a temporary general priority, a formal high priority, and a formal general priority. Unlike addtostandardlist,

Queueforsendingcontrolpacket adds all the sockets to be added to the queue to two temporary queues. Add them to normal temporary queues based on their priorities. In the large loop of runinternal, items in the temporary queue are first moved to the normal queue, and then processed.

Uploadbandwidththrottler uses two critical partitions and two events. Pauseevent is used to pause the entire large loop. Threadendedevent indicates that the entire thread is stopped. Sendlocker is the main critical section used in a large loop, while tempqueuelocker is an additional lock added to two temporary queues, so that you can send data to be sent by the existing set of domain characters in the queue, add the new socket to the queue.

The large loop in runinternal of uploadbandwidththrottler is the daily operation of the worker thread. In this large loop, we have done the following: Calculate the quota, that is, the number of bytes that can be sent in this cycle, arrange scheduling, calculate the amount of time that should be sleep in this cycle, and then perform the corresponding sleep, to speed up. The operation control information queue sends data in this queue. Note that the sockets (m_controlqueuefirst_list and m_controlqueue_list) in the control queue only exit the queue once. The socket in the standard queue does not. After a loop ends, if there is still a quota for unused data sending, some quotas will be saved to the next round. Network Infrastructure-eMule socket cemsocket is a subclass of casyncsocketex and throttledfilesocket. It integrates several functions and can be used as a convenient socket for eMule. For example, you can easily specify a proxy to block the function of creating a new proxy layer in casyncsocketex and adding it to the list. In addition, it can be divided into states, such as whether the current sending control information is in progress.

In cemsocket, we need to carefully examine its sendcontroldata and sendfileandcontroldata methods. As mentioned above, these methods are used in combination with uploadbandwidththrottler to implement the global speed limiting function. Its function should comply with uploadbandwidththrottler's requirements, and send a specified number of bytes when it sends data this time. Therefore, if the other part of the application uses cemsocket to speed up data uploading, the standard send or sendto method should not be called directly, but sendpacket should be called. Here we have another structure packet, which usually contains a complete package in the eMule protocol, such as header data with a protocol, and also has the built-in packpacket and unpackpacket methods, you can perform compression and decompression on your own. Sendpacket puts the packet to be sent into its own queue, which also has two queues, control the information packet queue and standard information packet queue. Add yourself to the uploadbandwidththrottler queue if necessary. We noticed that the sendcontroldata and sendfileandcontroldata methods of cemsocket actually call another overload send method of their own. We also know that this method is called in a large loop in the worker thread of uploadbandwidththrottler, and the content of this send method is also a large loop, but the meaning is clear, if the quota of this packet is not exceeded, the packet in the package queue is obtained and sent out. Similarly, a critical section is used here to ensure that the operations to extract packets from the packet queue for sending and put the packets in the queue are mutually exclusive. Therefore, if we combine it with uploadbandwidththrottler, we can see a two-tier queue, that is, all sockets constitute a sending queue, under the control of uploadbandwidththrottler, the speed limit is ensured, and the packets to be sent by each socket constitute a queue, ensuring that each data transmission meets the requirements of uploadbandwidththrottler.