High-Performance Server program framework

the server is deconstructed into three main modules:

• IO processing unit. Four IO models and two efficient event processing modes.
• Logical Unit. Two efficient concurrency modes.
• Storage unit. (not discussed at this stage)

1. Server Model (1) C/S (client/server) model

The logic of the C/S model is simple. After the server starts, first create one or more listening sockets, and call the bind function to bind it to the port on which the server is interested, and then call the Listen function to wait for the client to connect. After the server is running stably, the client can call the Connect function to initiate a connection to the server. Because the server needs to use an I/O model to monitor the connection request, the server calls the Accept function and assigns a logical unit to the new Connection service because of the asynchronous event that is randomly arrived at the client connection request.

The C/S model is very suitable for resource relative concentration, and its implementation is very simple, but its disadvantage is also obvious: the server is the communication center, when the traffic is too large, may all customers will get very slow response. This problem can be solved by peer.

(2) peer model

The peer model is more compatible with the actual situation of network communication than C/S model. It abandons the server-centric landscape and returns all hosts on the network back to their peers.

The peer model allows each machine to consume services while also providing services, which Yang resources can be sufficient. Free to share. Cloud computing is an example of a peer-to model. Peer disadvantage: The load on the network will increase when there are too many requests between users.

Another problem with peers is that it is difficult to find each other between hosts. So the in-use peer model usually comes with a dedicated discovery server, which usually provides a lookup service and even provides a content server so that every customer can find the resources they need as quickly as possible.

从编程的角度看，P2P模型可以看做C/S模型的扩展：每台主机既是客户端，又是服务器。

2. Server Programming Framework

Module	Single Server program	server cluster
I/O processing module	Handle client connections, read and write network data	As an access server, load balancing is achieved
Logical Unit	Business process or thread	Logical Server
Network storage Unit	Local database, file or cache	Database server
Request queue	Modes of communication between individual units	A permanent TCP connection between the individual servers.

2. I/O Model:

• Blocking IO
• Non-blocking IO
• IO multiplexing//Programs block the IO multiplexing system call, but can listen to multiple IO events at the same time.
• Sigio signal//Signal trigger read/write ready event, user program performs read and write operation, program does not block stage
• The asynchronous io//kernel performs read and write operations and triggers read-write completion events. The program has no blocking stage

3. Two efficient event-handling modes

two efficient event processing modes : The synchronous IO model is typically used to implement the reactor mode, and the asynchronous IO model is used to implement the Proactor mode.
servers typically handle three types of events : IO events, signal events, timed events.

Here 's a holistic overview of two efficient event-handling models : Reactor and Proactor models. Synchronous I/O is typically used to implement reactor mode, and the asynchronous I/O model is used to implement Proactor. However, Proactor can also be simulated with synchronous I/O.

1. Reactor Mode

Reactor is a pattern that requires the main thread to be only responsible for listening to the file description on whether the event occurred, or immediately notifying the worker that the main thread is not doing any other substantive work.
Read and write data, accept new connections, and process client requests are done in the worker thread.
The workflow of the reactor pattern implemented using the synchronous I/O model (epoll_wait as an example) is as follows

• The main thread network registers the read-ready event on the socket in the Epoll kernel event table.

• The main thread calls epoll_wait to wait for data to be read on the socket.

• Epoll_wait notifies the main thread when data is readable on the socket. The main thread puts the socket-readable event into the request queue.

• Sleep on a worker thread in the request queue is awakened, it reads data from the socket, processes the client request, and then registers the write-ready time on the socket in the Epoll kernel event table.

• The main thread waits for the socket to be writable with epoll_wait.

• When the socket is writable, the epoll_wait notifies the main thread. The main thread puts the socket writable event into the request queue.

• Sleep on a worker thread on the request queue is woken up, and it writes to the server on the network socket to process the results of the client request.

The workflow flowchart is as follows:

After the worker thread takes an event from the request queue, it decides how to handle it based on the type class of the time: for the readable event, the operation to read the data and process the request, the write data for the writable event, and therefore no distinction between the so-called "read worker" and "write Worker threads".

2.Proactor Mode

The Proactor mode gives all I/O operations to the main thread and kernel for processing, and the worker thread is only responsible for the business logic. Therefore, the Proactor mode is more consistent with the server framework.

The workflow for Proactor patterns implemented using the asynchronous I/O model (example Aio_read and Aio_write) is as follows:

1) The main thread calls the Aio_read function to register the read completion time on the socket to the kernel and tells the kernel user where to read the buffer, and how to notify the application when the read operation is complete.

2) The main thread continues to handle other logic.

3) When the data on the socket is read into the user buffer, the kernel sends a signal to the application to inform the application that the data is already available.

4) Application pre-defined signal processing function Select a worker thread to process the customer request. After the worker thread processes the client request, the Aio_write function is called to register the write completion time on the socket to the kernel, and tells the kernel where the user writes the buffer, and how to notify the application when the write operation is complete.

5) The main thread continues to handle other logic.

6) When the data of the user buffer is written to the socket, the kernel sends a signal to the application to notify the application that the data has been sent.

7) Application pre-defined signal processing function Select a worker thread to do the aftercare, such as deciding whether to close the socket.

The workflow for Proactor mode is as follows:

, the read-write event on the connection socket is registered with the kernel through aio_read/aio_write, so the kernel will signal to the application to report the read and write events on the socket. Therefore, the epoll_wait call in the main thread can only be used to detect connection request events on the listening socket, not to detect read and write events on the connection socket.

You can also use the synchronous I/O model to simulate the workflow of the Proactor mode.

5. Two efficient concurrency modes

并发编程等待目的是让程序“同时”执行多个任何。如果程序是计算密集型的，并发程序并没有优势，反而由于任何的切换是的效率降低。但如果程序是I/O密集型的，比如经常读写文件，访问数据库等，则情况就不同了。由于I/O操作的速度远远没有CPU计算速度快，所以让程序阻塞与I/O操作将浪费大量的CPU时间。如果程序有多个执行线程，则当前被I/O操作所阻塞的执行线程可主动放弃CPU（或者由操作系统来调度）,并执行权转移到其他线程。这样一来，CPU就可以用来做更加有意义的事情，而不是等待I/O操作完成，因此CPU的利用率显著提升。从实现上来说，并发编程主要有多进程和多线程两种方式，并发模式是值I/O处理单元和多个逻辑单元之间协调完成任何的方式。服务器主要有两种并发编程模式：半同步/半异步模式，领导者/追随者模式。

one, half synchronous/semi-asynchronous mode

在I/O模型中，“同步”和“异步”区分的是内核向应用程序通知的是何种I/O事件（是就绪时间还是完成事件），以及该由谁来完成I/O读写（是应用程序还是内核。）。并发模式中，“同步”指的是程序完全按照代码序列的顺序执行：“异步”指的是程序的执行需要由系统事件驱动。常见的系统时间包括中断、信号等。半同步/半异步模式中，同步线程用于处理客户逻辑。异步线程用于处理I/O事件，异步线程监听客户请求后，就将其封装成请求对象并插入请求队列中。请求队列将通知某个工作在同步模式的工作线程来读取该请求对象。具体选择哪个工作线程来为新的客户请求服务，取决于请求队列的设计。比如简单的轮流选择工作线程的Roud Robin算法，也可以通过条件变量或者信号量来随机选择一个工作线程。工作流程如下所示：

In a server program, there are multiple variants of the semi-synchronous/semi-Asynchronous Pattern if two time processing modes and several I/O modes are considered in conjunction. One variant is called semi-synchronous/semi-reactor mode, as shown in:

, the async thread has only one, which is served by the main thread. It is responsible for listening for all the events on the socket. If a read event occurs on the listener socket, that is, the new connection request is reached, the main thread accepts it to obtain a new connection socket, and then registers the read-write event on the socket in the Epoll kernel event table. If a read-write event occurs on the connection socket, that is, the incoming client request arrives or the data is sent to the client, the main thread inserts the connection socket into the request queue. All worker threads sleep on the request queue, and when there is any coming, they will gain any takeover by the competition. This competitive mechanism allows only idle worker threads to have the opportunity to deal with any new ones.

Disadvantages are as follows:

The main thread and the worker thread share request are aligned. The main thread adds a task to the request queue, or the worker thread pulls the task out of the request queue, requiring lock protection for the request queue, which consumes CPU time in vain.

Each worker thread can only process one customer request at a time. If the customer data is large and the number of worker threads is small, the request queue will accumulate many objects, and the response time of the client is getting slower. If you solve this problem by adding worker threads, the switching of worker threads also consumes a lot of CPU time.

is another relatively efficient semi-synchronous/semi-asynchronous Pattern in which each worker thread can handle multiple client connections at the same time.

, the main thread only manages the listener socket, and the connection socket is managed by the worker thread. When a new connection arrives, the main thread accepts it and sends the newly returned connection socket to a worker thread, and thereafter any I/O operations on the new socket are handled by the selected worker thread, knowing that the client closes the connection. The simplest way for the main thread to distribute a socket to a worker is to write data to the pipeline between it and the worker thread. When the worker thread detects that there is data readable on the pipeline, it analyzes whether a new client connection request arrives. If it is, register the read-write event on the new socket with your own Epoll kernel event table.

II. leader/follower Model

领导者/追随者模式是多个工作线程轮流获得事件源集合，轮流监听、分发并处理事件的一种模式。在任意时间点，程序都仅有一个领导者线程，它负责监听I/O事件。而其他线程则都是追随者，他们休眠在线程池中等待称为新的领导者。当前领导者如果检测到I/O事件，首先要从线程池中推选出新的领导线程，然后处理I/O事件。此时，新的领导者等待新的I/O时间，而原来的领导者则处理I/O事件，二者实现了并发。领导者/追随者模式包含如下几个组件：句柄集（HandleSet）、线程集（ThreadSet）、事件处理器（EventHandler）和具体的事件处理器（ConcreteEventHandler）。其关系如所示：

1. Handle Set

句柄用于表示I/O资源，在Linux下通常就是一个文件描述符。句柄集管理众多句柄，它使用wait_for_event方法来监听这些句柄上的I/O事件，并将其中的就绪时间通知给领导者线程。领导者则调用绑定到Handle上的事件处理器来处理事件。领导者则调用绑定到Handle上的事件处理器来处理事件。领导者将Handle和事件处理器绑定时通过调用句柄集中的register_handle方法实现的。

2. Thread Set

This component is the manager of all worker threads, including the leader thread and follower thread. It is responsible for synchronization between threads and the selection of new leader threads. Threads in thread set the hot Talk time must be in one of the following three states:

Leader: Thread is currently in the lead and is responsible for I/O events on the handle set.

Processing: line is impersonating is handling the event. After the leader detects an I/O event, it can be transferred to the processing state to handle the event and invoke the Promote_new_leader method to elect a new leader. You can also specify other followers to process the time (Event Handoff), where the leader's position is unchanged. When a thread in the processing state finishes processing an event, it becomes the new leader if there is no leader in the current thread set, or he turns directly into a follower.

Follower: The thread is currently in the follower identity, and the new leader may be assigned by the current leader to handle the new task by calling the join method of the thread set and so on.

It is important to note that leader threads elect new leaders and followers to wait for a new leader. The two operations will need to modify the thread set, so the thread set provides a member Synhronizer to synchronize the two operations. To avoid race conditions. ‘

3. Event handlers and specific event handlers

The event handler typically contains one or more callback function handle_event. These callback functions are used to handle the business logic corresponding to the event. The event handler needs to be bound to a handle before it is used, and when an event occurs on that handle, the leader executes the callback function in the event handler that is bound to it. The specific event handler for the derived class. They must re-implement the Handle_event method of the base class to handle specific tasks.

The workflow summary for leader/follower mode is as follows:

Because the leader thread listens to I/O events and processes customer requests, the leader/follower pattern does not require any additional data to be passed between threads, nor does it need to synchronize access to the request queue between threads as in semi-synchronous/semi-reactor mode. However, one obvious disadvantage of a leader/follower is that only one event source collection is supported, so it is not possible to have each worker thread manage multiple client connections independently.

6. Finite state machine

An efficient programming method within a logic unit: finite state machines.

Transitions between states need to be driven internally by the state machine.

7. Other suggestions for improving server performance

Performance is critical to the server, after all customers expect their requests to respond quickly. The primary factor that affects server performance is the hardware resources of the system, such as the number of CPUs, speed, memory size, etc. The following is primarily a software perspective that provides recommendations such as the maximum number of file descriptions that the system allows users to open. Also pay attention to the following aspects: pool, data replication, context switching.

first, the pool

Resources in the pool are pre-statically allocated, and we cannot anticipate how many resources should be allocated. How should the problem be solved? The simplest solution is to allocate enough resources to allocate the necessary resources for each client connection. This usually leads to a waste of resources because customer data at any one time may be far from reaching the maximum number of customers supported by the server.

Another option is to allocate a certain amount of resources in advance, after which the resources are not enough to be allocated dynamically and pooled.

Depending on the resource type, the pool is also divided into multiple, common memory pools, process pools, thread pools, and connection pools.

Memory pools are typically used for the receive cache and send cache of sockets. For some customer requests with limited length, such as HTTP requests, it is reasonable to pre-allocate a receive buffer of sufficient size (such as 5000 bytes). When the length of the client request exceeds the size of the receive buffer, we choose to discard the request or dynamically expand the receive buffer.

Both the process pool and the thread pool are common "tricks" for concurrent programming, and when we need a worker process or worker thread to handle a new incoming customer request, we can get an execution entity directly from the process pool or thread pool without having to call fork or Pthread_ Create functions such as creating processes and threads.

Connection pooling is typically used for internal persistent connections to a server or server cluster. Each logical unit may require frequent access to a local database. The simple approach is that each time the logical unit accesses the database, it initiates a connection to the data, and the connection is released when the access is complete. This approach is inefficient. One approach is to use a connection pool implementation. A connection pool is a set of connections that the server pre-establishes with the database program. When a logical unit needs to access a database, it can get a connected entity directly from the connection pool and use it. After access to the data is completed, the logical unit then returns the connection to the connection pool.

Second, data replication

High-performance servers should avoid unnecessary data replication, especially when data replication occurs between user code and the kernel. For example, the FTP server does not need to read the contents of the target file into the application buffer and call the Send function to send, but can use the "0 copy" function Sendfile to directly send it to the client.

In addition, data replication within user code should also be avoided.

iii. Context Switches and locks

并发程序必须考虑上下文切换的问题，即进程切换或者现场切换导致的系统开销。即使是I/O密集型的服务器，也不应该使用过多的工作线程，否则线程间的切换占用大量的CPU时间，服务器真正用于处理业务逻辑的CPU时间的比重就显得不足了。因此，为每个客户连接都创建一个工作线程的服务器模型是不可取的。半同步/半异步模式是一种比较合理的解决方案，他允许一个线程同时处理多个客户连接。此外，多线程服务器的一个优点是不同线程可以同时运行在不同的CPU上。当线程的数量不大于CPU的数目时，上下文切换就不是问题了。

Another problem that concurrent programs need to consider is the lock protection of shared resources. Locks are often considered to be a factor in inefficient servers, as it introduces code not only to not handle any business logic, but also to access kernel resources. Therefore, if the server has a better solution, you should avoid using locks. If the server must use locks, consider reducing the granularity of locks, such as using read-write locks. Read-write locks do not add additional overhead to the system when all worker threads read only a piece of shared memory. The system must lock the area only if one of its worker threads needs to write this piece of content.

High-Performance Server program framework