ROCKETMQ Source Code Analysis

ROCKETMQ Source Code Analysis

ROCKETMQ design Ideas from the Kafka, in the specific design to reflect their own choice and needs, the specific differences can be seen ROCKETMQ and Kafka contrast (18 differences). Next, note some of the exploration of your own reading source.

2. The overall architecture of ROCKETMQ is as follows, you can see the roles that each component plays, and Name Server is responsible for maintaining some global routing information: What broker is currently in, and which broker is in each topic; Broker specifically handles the storage and service of messages, and producers and consumers are the source and destination of the message.
   
   After knowing the basic position of each role, it is time to let the program run, so as to see more real, rocketmq rely on very little, can easily be deployed locally, first set the environment variable Rocketmq_home, which is ROCKETMQ need to work directory. NameServer starts first so that it listens (listening on port 9876) from the broker and specifies nameserver when the broker starts, otherwise it will go to a server to find a list of available nameserver. You can then run the example in Rocketmq-example for a simple test.

4. Name server Dapper, coordinated global
   From the code you can see that the primary is to initialize the service-side communication layer and the thread-executing component, and then start Serverbootstrap boot. With the Jstack command you can see which threads are started.

6. Producer send a message how do you know which broker is sent?
   Each app will normally call Producer.start ()/consumer.start () before sending and receiving messages, and behind the scenes is to create the required instance objects, such as Mqclientinstance; set the scheduled task, If the local topic route info is updated regularly from nameserver, the heartbeat information is sent to all brokers, the size of the thread pool is dynamically adjusted, and so on; The current producer is added to the specified group. The client caches the routing information Topicpublishinfo, and periodically takes topic routing information from nameserver, and each broker establishes a long connection with all nodes in the nameserver cluster. Register topic information regularly to all nameserver. Producer when the message is sent to query the local topicpublishinfotable (a hashmap), if not hit the words will communicate with the nameserver to get the routing information, requestcode=get_ Routeinto_by_topic If the nameserver is also found in the log, then a default TOPIC will be sent for routing queries. The exact process is as follows:
   

8. Producer the specific process of sending a message, take the synchronization mode as an example.
   Get the correspondence address, the sending process is obvious. You can see that the message queue is selected to be sent in a random manner, while maintaining a different broker from the last one sent, preventing hotspots.
   

10. How does the broker receive messages from producer?
    Each producer sends a message with the corresponding broker to establish a long connection, at which time the broker is ready to receive the message, the process is as follows. When the message is received, the commit log file is written (sequential, full, and a new file is created), and then the consume queue file is updated (how the store is anchored to a specific message by topic).
    

12. ROCKETMQ Storage Features
    ROCKETMQ messages are written sequentially to the Commitlog file, and then the consume queue file is used as the index. ROCKETMQ uses a 0 copy Mmap+write to respond to consumer's request, ROCKETMQ claims that most of the requests will be met at the page cache layer, so too many messages will not degrade because of disk reads, and the understanding here is that In the 64bit machine, the virtual storage address space (VM_AREA_STRUCT) is not an issue, so the relevant files will be mapped into memory (there is a regular operation to delete files), even if it is not in memory at the moment, the operating system will be replaced by the fault pages, although the address space is not a problem, However, the number of a process mapping file is limited, so oom may occur here.
    
    Traditional read-write and two 0 copies of the simple pair, for example, is actually the CPU and IO trade-offs.
    
    The above expression can also be seen through the storage directory in the broker:
    

14. How are sequential messages guaranteed?
    It takes the business layer to think for itself which messages should be guaranteed in order, and then to send them to the same queue by rules, because no one knows more about the characteristics of the message order than the business itself. This order is in relative order, local order, because the sender only guarantees that these messages are sent to the same queue on the broker, but there is no guarantee that other producer will be sent to that queue, so you need consumer to do some filtering after pulling the message.

16. ROCKETMQ Brush Disc Implementation
    The final implementation of the brush disk is to use the Mappedbytebuffer.force () in NiO to write data from the map area to disk, and if it is a synchronous brush disk, after the broker writes the message to the Commitlog map area, it waits for the write to complete. Asynchronously, it just wakes up the corresponding thread and does not guarantee the timing of execution, as follows.
    

18. Message filtering
    Similar to the data deduplication technology, can be done at the source side, can also be implemented at the destination, that is, network and storage trade-offs, if the broker side to do message filtering needs to be compared to the consume Queue Tagscode field (hashcode), if the match is transmitted to the consumer, Because it is hashcode, there is a miscarriage of error that requires string-level filtering after consumer receives the message, ensuring accuracy.

Summary: Reading this code also took a few days, although in the distributed reliability has not been carefully studied, mainly to see the key design ideas and processes, there are many things to continue to precipitate and digestion, especially multi-threaded synchronization, asynchronous and so on.

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

ROCKETMQ Source Code Analysis