MySQL Proxy: underlying implementation

Bottom-layer implementation (chassis)

Configfile and Commandline Options]

Glib2 provides the config-file parsing and command-line option parsing functions. It provides methods to expose the option to the caller in the same way, as well as the function to obtain the option from Configfile and Commandline.

The parsing process of all options can be divided into three steps:

1. Extract the basic option on command-line

• -- Help
• -- Version
• -- Defaults-file

2. process the defaults-file

3. process other command-line options and overwrite the same content in the defaults-file.

Plugin Interface]

Chassis provides the basic structure for calling the plugin interface. It is worth noting that it is not specifically used for MySQL, but can be used for any plug-in that meets its interface requirements. The following functions are provided:

Because chassis is not designed only for MySQL, it can be used to load any type of plugin, as long as the plugin provides init and shutdown functions that meet the requirements of chassis.

As far as MySQL Proxy itself is concerned, the plugin loaded is generally:

 
 

  
  
plugin-proxy  
  
  
plugin-admin 
 
 

Threaded IO]

From MySQL Proxy 0.8, thread-based network-io has been added to enable the proxy to linearly expand according to the number of available CPUs and NICs.

To enable the network-threading function, you only need to add the following parameters when starting the proxy:

 
 

  
  
--event-threads={2 * no-of-cores} (default: 0) 
 
 

Every event-thread loops through "event_base_dispatch ()" and executes related functions for network-event or time-event. These threads have only two States: the execution function status and the idle status. If it is in the idle status, it can get the new event to wait from event-queue, and then add it to its own waiting list.

Connection can be "jumped" between multiple event-threads: as long as it is an idle event-thread, you can get wait-for-event request-that is, the specific event-and wait for it. After the event is triggered, the relevant code is executed. At any time, as long as the current connection needs to wait for the event again, that is, the operation corresponding to the previous event has been completed), it will unregister itself from the thread, then, send the wait-for-event request to the global event-queue to obtain the new event.

Until MySQL Proxy version 0.8, script code is executed in a single thread mode: A Global mutex is used to protect the interface operations of the plugin. Because the connection is either in the sending package status or in the status of calling the plugin function, network events will be processed in parallel, parallel connection is not allowed only when multiple connections need to call the same plug-in function.

The chassis_event_thread_loop () function is the main cyclic entity of event-thread, where the event_base_dispatch () function is called. The chassis_event_threads_init_thread () function is used to set the event to be listened to and the corresponding callback.

The following describes a typical scenario where the control flow does not contain the connection pool)

Entities involved: EventRequestQueue, MainThread, WorkerThread1, and WorkerThread2;

 
 

  
  
--- [ label = "Accepting new connection "];   
  
  
 
  
  
    MainThread -> MainThread [ label = "network_mysqld_con_accept()" ];   
  
  
    MainThread -> MainThread [ label = "network_mysqld_con_handle()" ];   
  
  
 
  
  
    MainThread -> EventRequestQueue [ label = "Add wait-for-event request" ];   
  
  
    WorkerThread1 <- EventRequestQueue [ label = "Retrieve Event request" ];   
  
  
    WorkerThread1 -> WorkerThread1 [ label = "event_base_dispatch()" ];   
  
  
    ...;   
  
  
    WorkerThread1 -> WorkerThread1 [ label = "network_mysqld_con_handle()" ];   
  
  
       
  
  
    WorkerThread1 -> EventRequestQueue [ label = "Add wait-for-event request" ];   
  
  
       
  
  
    WorkerThread2 <- EventRequestQueue [ label = "Retrieve Event request" ];   
  
  
    WorkerThread2 -> WorkerThread2 [ label = "event_base_dispatch()" ];   
  
  
    ...;   
  
  
    WorkerThread2 -> WorkerThread2 [ label = "network_mysqld_con_handle()" ];   
  
  
       
  
  
    WorkerThread2 -> EventRequestQueue [ label = "Add wait-for-event request" ];   
  
  
    ...; 
 
 

In the above example, there are two workthreads for processing the event -- event-threads = 2), each thread has its own event_base. Take Proxy plugin as an example. First, set the network_mysqld_con_accept () function to the callback of the socket to be monitored. This function is triggered when a new connection occurs. This callback function is registered on the event_base of the main thread and the event_base of global chassis ). After network_mysqld_con is set, the program enters the state machine processing function network_mysqld_con_handle (), which is still in the main thread.

The state machine enters the starting state: CON_STATE_INIT. In the current code implementation, this state is the first state that the main thread must enter. What MySQL Proxy will do next is either to interact with the client, or to interact with the server, that is, to wait for the socket to read, or to actively establish a connection to the backend server), and the state machine function network_mysqld_con_handle () the struct corresponding to the event waiting for processing will be set to chassis_event_op_t ). Simply put, the event structure is added to the asynchronous queue. Specifically, a file descriptor event is generated by writing a byte to the write file descriptor of the previously created wakeup-pipe. In this way, we can notify all threads that new event requests need to be processed.

The pipe implementation is a replica of the libevent implementation. It establishes a correspondence between various events and the event-handler Based on the file descriptor, and uses the polling method for processing:

In the file chassis-event-thread.c, we can see that through pipe the implementation of the notification to the working thread: There is something to be done in the global event-queue. From the chassis_event_handle () function, we can see that all threads in the idle State have equal opportunities for event processing. Therefore, these threads can "concurrently" pull events from the global event queue, and add it to the list of listening events.

You can add an event to event-queue by calling the chassis_event_add () or chassis_event_add_local () functions. Generally, all events are handled by global event_base. Only when the connection pool is used will the events corresponding to the specific server connection be forcibly shipped to the specific thread to add the current connection to the thread in the connection pool.

If the event is shipped to global event_base, different threads can obtain the event and modify the unprotected connection pool data structure, which may lead to a competitive risk and crash. To make this internal data structure a thread-safe version of 0.9 release, only the minimum thread security is provided.

In typical cases, a thread obtains the request information from the event queue. Theoretically, the thread that sends the wait request may also be the thread that processes the request ), add it to the event_base saved in thread-local-store mode, and receive a notification when the corresponding fd event is triggered.

This process continues until the current connection is disabled by the client or server, or the network error caused by socket shutdown occurs. You cannot process any new requests.

A single thread is sufficient to process any event added to its thread-local event_base. Only when a new blocking I/O operation occurs, in general, that is, when the event_base_dispatch () is blocked again, the event will be "jumped" between different threads for processing, unless otherwise specified. Therefore, theoretically, one thread may process all active socket events, while the other thread remains in the idle state.

However, because the status of waiting for the occurrence of network events is normal, that is, the actual processing speed is very fast), so in terms of probability) the distribution of active connections in all threads must be very uniform, which reduces the pressure on a single thread to process active connections.

It is worth noting that, although not specifically stated in the following description, the main thread will be involved in processing subsequent events after the accept status. This is not an ideal implementation method, because all accept actions must be completed in the main thread. But on the other hand, this problem has not yet become a bottleneck in actual work:

Entities involved: Plugin, MainThread, MainThreadEventBase, EventRequestQueue, WorkerThread1, WorkerThread1EventBase, WorkerThread2, WorkerThread2EventBase;

 
 

  
  
--- [ label = "Accepting new connection "];   
  
  
 
  
  
    Plugin -> MainThread [ label = "network_mysqld_con_accept()" ];   
  
  
    MainThread -> MainThread [ label = "network_mysqld_con_handle()" ];   
  
  
 
  
  
    MainThread -> EventRequestQueue [ label = "Add wait-for-event request" ];   
  
  
    WorkerThread1 <- EventRequestQueue [ label = "Retrieve Event request" ];   
  
  
    WorkerThread1 -> WorkerThread1EventBase [ label = "Wait for event on local event base" ];   
  
  
    ...;   
  
  
    WorkerThread1EventBase >> WorkerThread1 [ label = "Process event" ];   
  
  
       
  
  
    WorkerThread1 -> EventRequestQueue [ label = "Add wait-for-event request" ];   
  
  
       
  
  
    WorkerThread2 <- EventRequestQueue [ label = "Retrieve Event request" ];   
  
  
    WorkerThread2 -> WorkerThread2EventBase [ label = "Wait for event on local event base" ];   
  
  
    ...;   
  
  
    WorkerThread2EventBase >> WorkerThread2 [ label = "Process event" ];   
  
  
       
  
  
    WorkerThread2 -> EventRequestQueue [ label = "Add wait-for-event request" ];   
  
  
    ...;