Optimized--epoll optimization of mosquito (vii)

This article is written by the carefree son, forwarded please mark the original site:

http://blog.csdn.net/houjixin/article/details/46413583

Or

http://houjixin.blog.163.com/blog/static/3562841020155835146428/#

The original mosquito in the mobile internet situation, its performance is not high, the actual operation of a mosquito instance can support 20,000 connection is good; Mosquitto in the case of bad network condition, with the increase of user volume, its CPU consumption will increase greatly, The main CPU is mainly consumed in the following areas:

(1) Defects of poll mechanism;

(2) Defects of Mosquitto internal subscription tree mechanism;

(3) Other information transmission, data structure management defects;

This section will propose appropriate optimization strategies and methods for these defects.

7.1, poll optimization

7.1.1, optimization reason

In the Mosquitto original program, the core processing process is to all the socket for listening processing, this part of the function is mainly used poll to complete, but it is inefficient, especially when the online active users less, performance is worse, this must affect the Mosquitto performance, The low efficiency of epoll is mainly due to the following 3 reasons:

1) poll need to re-register all the sockets that need to be monitored before each listening port;

2) Poll return results, will only modify the socket corresponding to the event of the poll structure, therefore, in the use of all registered sockets corresponding to the poll structure to scan, to determine that those sockets have events.

3) on the internal implementation, the poll needs to query all registered sockets to determine if there are any events occurring.

The above problems of poll are caused by their own implementation, which is difficult to optimize. For poll, Linux implements a more efficient implementation: Epoll, Epoll does not need poll these complex operations, Epoll has the following 3 advantages:

1) Epoll, only need to register the socket being monitored into epoll once, the subsequent epoll will listen to it, and do not need to re-register each time before listening.

2) The Epoll return result contains all sockets that have an event, so during processing, all of these sockets that have an event occurred are scanned without having to scan all registered sockets.

3) in the internal implementation, Epoll does not need to query all registered sockets, it is inside: all the sockets that have events occur themselves to mount the Epoll ready queue, Epoll simply return to the ready queue socket.

Therefore, this optimization first chooses to optimize the poll, mainly uses the Epoll to replace the poll, in order to improve the system execution efficiency.

7.1.2, optimization scheme:

The use of poll in Mosquitto is mainly focused on the loop.c of the file in the function Mosquitto_main_loop, so this modification will replace it with a new function Epoll_mosquitto_main_loop. In addition, in the use of methods, Epoll's listener function epoll_wait return all ready sockets, and Mosquitto program need to know the socket corresponding conetxt to complete the business processing, so in order to support Epoll, need to add a hash table T _fd2context to complete the mapping of the socket to its corresponding conetxt.

The core processing logic of mosquito is mainly in the function Mosquitto_main_loop, the function mainly accomplishes the following functions:

1) Update the information of the system topic;

2) Put all the listening sockets into the poll structure Pollfds;

3) Scan all the context to complete the following tasks:

If the context has a message sent, the message is sent in accordance with the MQTT protocol;

Check if the context is timed out;

The socket of the context is placed in the poll structure Pollfds.

4) Scan all conetxt, if there is a message in the context, update the timestamp of the message;

5) Call poll, poll the poll structure Pollfds all sockets;

6) to deal with the results of poll, the following two tasks need to be completed:

Scan all the context to see if the corresponding socket has an event, and if so, read and write;

Handles all listener sockets and creates a corresponding context if a new connection is entered

7) Perform optional actions such as reloading the configuration file.

During poll's work, the above operations will be executed in a loop, and the core process of the Mosquitto after using Epoll optimization will be changed to:

1) Register the listening port to the Epoll, the step is executed before the loop, and the next 2) step will be put into the loop execution.

2) Update the information of the system topic;

3) Scan all context according to the strategy, complete the following two jobs:

Reclaims the context and places the index of the context into an array of free indexes;

Check timeout, if timeout, then the map of the context and its socket is removed from the hash table t_fd2context;

4) Call Epoll's epoll_wait function to get all the ready sockets;

5) All the ready sockets are processed, and the main tasks are as follows:

If the ready socket is a listening interface, the listener interface is processed, the context is established for each new incoming business socket, and it is registered into the Epoll;

If the socket is not listening, the corresponding context is found in the hash table t_fd2context from the socket to the Cotext, and then the corresponding processing is done, and if not found, the connection to the socket is disconnected.

6) Perform optional actions such as reloading the configuration file.

Because Epoll and poll work in different ways, the above process needs to be modified so that it can adapt to epoll requirements, the main changes include:

1) Socket registration method, each cycle in poll need to re-register the socket into the poll, and Epoll only need to start registration once;

2) return the result processing; poll all registered socket structures need to be scanned to determine if a socket has data processing; Epoll directly returns to the ready socket, so there is no need to traverse the registered socket structure.

3) Add the socket to the corresponding context mapping, because epoll directly back to the ready socket, and mosquitto need to find the corresponding context of the socket in order to handle the upper application, It is therefore necessary to add a hash table to complete the socket-to-context mapping.

4) Modify the function of the message sending section

7.1.3, Concrete implementation

Epoll optimization also uses the original single-threaded structure, and uses a cycle "while" to complete the processing of the task, the cycle is placed in the function Epoll_mosquitto_main_loop, the function and function mosquitto_main_ Loop (the function is the primary business process when using poll), the parameters are exactly the same, and the Epoll_mosquitto_main_loop is called in the function Mosquitto_main_loop. Therefore, the place in the program that originally called Poll's main business handler function Mosquitto_main_loop will be diverted to call Epoll's main business processing function Epoll_mosquitto_main_loop, thus realizing the call to the Epoll function. The process of the system is shown in 5-1.

Figure 7-1 System flowchart after using Epoll

1. Socket Registration

Epoll in use only need to monitor the socket to join once, which differs from poll in the use of methods, in the Mosquitto program, need to epoll monitoring socket including the listener socket and business socket The socket registration process is completed by the function Reg_socket, the registered socket listener type is Epollin, with the default horizontal trigger mode.

1) Listen to the socket, this type of socket is responsible for receiving the client's new connection, such as the default 1883 port in the program socket, the client will use the port to connect to Mosquitto. Before the main loop of function Epoll_mosquitto_main_loop is completed, registration is done only once, and subsequent registration is no longer repeated;

2) Business socket, this type of socket is responsible for completing the transfer of business data between the client and Mosquitto; the business socket will complete the registration when the new connection enters, this process is done in the function Epoll_loop_handle_result.

2. Epoll Event Handling

The Epoll event processing will be done by the function Epoll_loop_handle_result, in which all the ready sockets returned by the Epoll are scanned, and each ready socket is processed in the following manner:

1) If you are listening to a socket, first call the Mqtt3_socket_accept function to process the new connection socket, the process includes: Create a corresponding context for the socket, and then register the socket into the epoll.

2) If the listening port is a business socket, the data on the struct is read, and the data is processed.

Figure 7-2 Epoll Event Processing flow

Optimized--epoll optimization of mosquito (vii)