[Reprint] Network IO Model

Reprinted from http://blog.csdn.net/zhoudaxia/article/details/8974779

What is the difference between synchronous (synchronous) IO and asynchronous (asynchronous) Io, what is blocking (blocking) IO and non-blocking (non-blocking) IO respectively? The problem is that different people may give different answers, such as wikis, that asynchronous IO and non-blocking io are a thing. This is because different people have different backgrounds, and the context is not the same when discussing this issue. Therefore, in order to better answer this question, I first limit the context of this article.

This article discusses the background of network IO in a Linux environment. The most important references in this paper are Richard Stevens's "Unix®network programming Volume 1, third edition:the Sockets Networking", 6.2 "I/O Models", Stevens in this section details the various IO characteristics and differences, if the English is good enough, recommend direct reading. The style of Stevens is famous, so don't worry about it. The flowchart in this paper is also intercepted from the reference literature.

Stevens compared five IO Model in the article:
* Blocking IO
* nonblocking IO
* IO Multiplexing
* Signal Driven IO
* Asynchronous IO
By signal driven IO is not commonly used in practice, so the main introduction of the remaining four IO Model.
Again, the objects and steps involved in the IO occur. For a network IO (here we read for example), it involves two system objects, one that calls the IO process (or thread), and the other is the system kernel (kernel). When a read operation occurs, it goes through two stages:
1) Wait for data preparation (waiting for the
2) Copy the data from the kernel into the process (Copying the data from the kernel-the process)
It is important to remember these two points, because the difference between these IO models is that there are different situations in both phases.
1. Blocking io (blocking IO)

In Linux, all sockets are blocking by default, and a typical read operation flow is probably this:

Figure 1 Blocking IO

When the user process invokes the RECVFROM system call, Kernel begins the first phase of IO: Preparing the data. For network IO, there are times when the data has not arrived at the beginning (for example, a full UDP packet has not been received), and kernel waits for enough data to arrive. On this side of the user process, the entire process is blocked. When kernel waits until the data is ready, it copies the data from the kernel to the user's memory, and then kernel returns the result, and the user process removes the block state and re-runs it.
Therefore, the blocking IO is characterized by the block of two phases of IO execution (two stages of waiting for data and copying data).

Almost all programmers first come into contact with the network programming from Listen (), send (), recv () and other interfaces started, these interfaces are blocking type. Using these interfaces makes it easy to build a server/client model. Here is a simple "Ask and Answer" server.

Figure 2 Simple one-answer server/client model

We note that most socket interfaces are blocking types. A blocking interface is a system call (typically an IO interface) that does not return a call result and keeps the current thread blocked until the system call obtains a result or a time-out error.
Virtually all IO interfaces (including the socket interface) are blocking, unless specifically specified. This poses a big problem for network programming, such as when calling Send (), the thread will be blocked, during which time the thread will be unable to perform any operations or respond to any network requests.

A simple improvement is to use multithreading (or multiple processes) on the server side. The purpose of multithreading (or multi-process) is to have separate threads (or processes) for each connection, so that blocking of any one connection does not affect other connections. The specific use of multi-process or multi-threading, and does not have a specific pattern. Traditionally, processes are much more expensive than threads, so it is not recommended to use multiple processes if you need to serve more clients at the same time, and if a single service executor needs to consume more CPU resources, such as large-scale or long-time data operations or file access, the process is more secure. Typically, a new thread is created with Pthread_create (), and fork () creates a new process.
We assume that the above-mentioned server/client model requires a higher requirement for the server to provide a single-answer service to multiple clients at the same time. Therefore, the following model is available.

Figure 3 Multi-threaded server model

In the thread/time legend above, the main thread continues to wait for the client's connection request and, if there is a connection, creates a new thread and provides the same question and answer service in the new thread.
Many beginners may not understand why a socket can accept multiple times. In fact, the designer of the socket may have deliberately left a hint for a multi-client scenario, allowing the accept () to return a new socket. The following is the prototype of the Accept interface:
int accept (int s, struct sockaddr *addr, socklen_t *addrlen);
The input parameter s is the socket handle value that is inherited from the socket (), bind (), and listen (). After bind () and listen () are executed, the operating system begins to listen for all connection requests at the specified port and, if there is a request, joins the request queue with the connection request. Calling the Accept () interface extracts the first connection information from the request queue of the socket s, creating a new socket return handle that is similar to S. The new socket handle is the input parameter for subsequent read () and recv (). If the request queue does not currently have a request, the accept () will go into a blocking state until a request enters the queue.
The multi-threaded server model seems to be a perfect solution to the requirements for answering questions and answers for multiple clients, but not really. If you want to respond to hundreds or thousands of connection requests at the same time, no matter how many threads or processes will take up the system resources, reduce the system to the external response efficiency, and the thread and process itself more easily into the suspended animation state.
Many programmers may consider using"thread pool" or "Connection pool"。 The thread pool is designed to reduce the frequency of creating and destroying threads, maintaining a reasonable number of threads, and allowing idle threads to re-assume new execution tasks. Connection pooling maintains a connected cache pool, reusing existing connections as much as possible, and reducing the frequency with which connections are created and closed. Both of these technologies can reduce system overhead and are widely used in many large systems, such as WebSphere, Tomcat, and various databases. However, the thread pool and connection pooling techniques are only to some extent mitigated by the frequent invocation of the IO interface for resource consumption. AndThe so-called "pool" always has its upper limit, when the request greatly exceeds the upper limit, the "pool" composed of the system response to the outside world is not much better than when there is no pool. So using the pool must consider the scale of the response it faces and adjust the size of the pool based on the response scale.
The "thread pool" or "Connection pool" may alleviate some of the stress, but not all of them, in response to the thousands or even thousands of client requests that may appear in the previous example. In short, multithreaded models can easily and efficiently solve small-scale service requests, but in the face of large-scale service requests, multithreading model will encounter bottlenecks, you can use non-blocking interface to try to solve the problem.

2, non-blocking IO (non-blocking io)
Under Linux, you can make it non-blocking by setting the socket. When you perform a read operation on a non-blocking socket, the process looks like this:

Figure 4 Non-blocking IO

As you can see, when the user process issues a read operation, if the data in kernel is not ready, it does not block the user process, but returns an error immediately. From the user process point of view, it initiates a read operation and does not need to wait, but immediately gets a result. When the user process determines that the result is an error, it knows that the data is not ready, so it can send the read operation again. Once the data in the kernel is ready and again receives the system call of the user process, it immediately copies the data to the user's memory and then returns.
therefore, in non-blocking IO, the user process is actually required to constantly proactively ask kernel data ready to be prepared.

The significant difference between a nonblocking interface and a blocking interface is that it returns immediately after being called. Use the following function to set a handle FD to a non-blocking state.
Fcntl (FD, F_SETFL, O_nonblock);
The following gives a model that uses only one thread, but is able to detect data delivery from multiple connections at the same time, and accepts the data.

Figure 5 Using a non-blocking receive data model

In a non-blocking state, the recv () interface returns immediately after being called, and the return value represents a different meaning. As in this example,
* RECV () The return value is greater than 0, indicating that the received data is complete, the return value is the number of bytes received;
* RECV () returns 0, indicating that the connection has been broken normally;
* RECV () returns 1, and errno equals Eagain, indicating that the recv operation has not been completed;
* RECV () returns 1, and errno is not equal to Eagain, indicating that the recv operation encountered a system error errno.
You can see that the server thread can invoke the recv () interface through a loop to implement data reception for all connections within a single thread. But the above model is never recommended. Because the cyclic invocation of recv () will significantly push up CPU occupancy, and in this scenario recv () is more of a test of "Operation Done", the actual operating system provides a more efficient interface for detecting whether "operation is done", such as Select () Multiplexing mode, you can detect multiple connections at once to be active.
3. Multiplexed io (io multiplexing)
The word IO multiplexing may be a bit unfamiliar, but if I say select/epoll, I'll probably get it. Some places also call this IO mode for event-driven IO(driven io). As we all know, the benefit of Select/epoll is that a single process can simultaneously handle multiple network connections of IO. The basic principle of the select/epoll is that the function will constantly poll all sockets that are responsible, and when a socket has data arrives, notifies the user of the process. It's process

Figure 6 Multiplexing IO

    when a user process invokes select, the entire process is blocked, and at the same time, kernel "monitors" all the select-Responsible sockets, When the data in any socket is ready, select returns. This time the user process then invokes the read operation, copying the data from the kernel to the user process.
    This figure is not much different from the blocking IO diagram, in fact it's even worse. Because there are two system calls (select and recvfrom) that need to be used, blocking IO only calls a system call (Recvfrom). However, the advantage of using select is that it can handle multiple connection at the same time. (say: So, if the number of connections processed is not very high, the Web server using Select/epoll does not necessarily perform better than the Web server using multi-threading + blocking IO, and may be more delayed.) The advantage of Select/epoll is not that a single connection can be processed faster, but that it can handle more connections.
     in a multiplexed model, for each socket, it is generally set to non-blocking, but, as shown, the entire user's process is actually always block. only the process is a block of the Select function, not the socket IO. So select () is similar to non-blocking IO.

Most unix/linux support the Select function, which is used to probe the state change of multiple file handles. The prototype for the Select interface is given below:
Fd_zero (int fd, fd_set* FDS)
Fd_set (int fd, fd_set* FDS)
Fd_isset (int fd, fd_set* FDS)
FD_CLR (int fd, fd_set* FDS)
int select (int Nfds, fd_set *readfds, Fd_set *writefds, Fd_set *exceptfds,
struct Timeval *timeout)
Here, the Fd_set type can be simply understood as a queue that marks a handle by bit, for example to mark a handle with a value of 16 in a fd_set, and the 16th bit bit of the fd_set is marked as 1. Specific placement, validation can be achieved using Fd_set, Fd_isset and other macros. In the Select () function, Readfds, Writefds, and Exceptfds are both input and output parameters. If the input Readfds is marked with a number 16th handle, select () detects whether the 16th handle is readable. After select () is returned, you can determine whether the "readable" event occurs by checking if the Readfds has a 16th-number handle. In addition, the user can set timeout time.
The model that receives data from multiple clients in the previous example is re-modeled.

Figure 7 receiving data Model using select ()

The model simply describes the process of receiving data from multiple clients simultaneously using the Select () interface, and because the Select () interface can simultaneously detect multiple handles for read, write, and error states, it is easy to build a server system that provides independent question and answer services to multiple clients. Such as.

Figure 8 Event-driven server model using the Select () interface

It should be noted here that a connect () operation on the client will fire a "readable event" on the server side, so select () can also detect the Connect () behavior from the client.
In the above model, the most critical place is how to dynamically maintain the three parameters of select () Readfds, Writefds, and Exceptfds. As an input parameter, Readfds should mark all the handles of the "readable event" that need to be probed, which will always include the "parent" handle that probes for connect (), and Writefds and Exceptfds should mark all "writable events" and "error events" that need to be probed. Handle (using the Fd_set () tag).
As an output parameter, the handle values of all events captured by select () are saved in Readfds, Writefds, and Exceptfds. The programmer needs to check all the token bits (using the Fd_isset () check) to determine exactly which handles have occurred.
The above model mainly simulates is "a ask a reply" service flow, so if select () found a handle caught "readable event", the server program should do recv () operation in time, and according to the received data ready to send data, and the corresponding handle value added to Writefds, Prepares the next "writable event" of the Select () probe. Similarly, if select () finds a handle that snaps to a writable event, the program should do a send () operation in time and prepare for the next "readable event" probe. Describes one of the execution cycles in the above model.

Figure 9 An execution period for a multiplexed model

This model is characterized by the fact that each execution cycle detects one or a set of events, and a particular event triggers a specific response. We can classify this model as an " event-driven model ".
Compared to other models, the event-driven model using select () executes only single-threaded (process), consumes less resources, consumes too much CPU, and provides services to multiple clients. If you try to build a simple event-driven server program, this model has some reference value.
But the model still has a lot of problems. The first select () interface is not the best option to implement event-driven. Because the Select () interface itself consumes a lot of time to poll each handle when the value of the handle to be probed is large. Many operating systems provide a more efficient interface, such as Linux provides the EPOLL,BSD provides the Kqueue,solaris provides the/dev/poll, .... Interfaces like Epoll are recommended if you need to implement more efficient server programs. Unfortunately, the Epoll interface for different operating systems is a big difference, so using a epoll-like interface to implement a server with better cross-platform capabilities can be difficult.
Secondly, the model is a combination of event detection and event response, which is catastrophic for the entire model once the event response is large. in the following example, a large 1 of the actuator will directly lead to response to event 2 of the execution of the delay is not implemented, and to a large extent, reduce the timeliness of event detection.

Figure 10 Impact of a large execution on an event-driven model using select ()

Fortunately, there are many efficient event-driven libraries that can mask the above difficulties, common event-driven libraries have libevent libraries , and Libev libraries as libevent substitutes. These libraries choose the most appropriate event detection interface based on the characteristics of the operating system, and include techniques such as signaling (signal) to support asynchronous responses, making these libraries an ideal choice for building event-driven models. The following chapter describes how to use the Libev library to replace a select or Epoll interface to achieve an efficient and stable server model.

In fact, starting with 2.6, the Linux kernel also introduces IO operations that support asynchronous responses, such as Aio_read, Aio_write, which is asynchronous IO.

4. Asynchronous IO (asynchronous I/O)
The asynchronous IO in Linux is not used much, and is only introduced from kernel version 2.6. Let's take a look at its process:

Figure 11 Asynchronous IO

After the user process initiates the read operation, you can begin to do other things immediately. On the other hand, from the perspective of kernel, when it receives a asynchronous read, first it returns immediately, so no block is generated for the user process. Then, kernel waits for the data to be ready and then copies the data to the user's memory, and when all this is done, kernel sends a signal to the user process to tell it that the read operation is complete.

Server implemented with asynchronous IO Here is not an example, there will be time to open another article to tell. Asynchronous IO is truly non-blocking, and it does not create any blocking on the request process, so it is critical for high-concurrency network server implementations.
So far, four IO models have been introduced. Now back to the first few questions: what is the difference between blocking and non-blocking, and what is the difference between synchronous IO and asynchronous IO?
First answer the simplest of this: blocking and non-blocking. The difference between the two is clearly explained in the previous introduction. Calling blocking IO will block the corresponding process until the operation is complete, and non-blocking IO will return immediately if the kernel is still preparing the data.
Before explaining the difference between synchronous IO and asynchronous IO, you need to give a definition of both. The definition given by Stevens (in fact, the definition of POSIX) is this:
* A synchronous I/O operation causes the requesting process to being blocked until that I/O operation completes;
* An asynchronous I/O operation does not cause the requesting process to be blocked;
The difference is that synchronous IO will block the process when it does "IO operation". According to this definition, the blocking io,non-blocking Io,io Multiplexing described previously are synchronous IO. One might say that non-blocking io is not block. Here is a very "tricky" place, defined in the "IO operation" refers to the real IO operation, is the example of recvfrom this system call.non-blocking IO does not block the process if the kernel data is not ready when performing recvfrom this system call. But when the data in the kernel is ready, Recvfrom will copy the data from the kernel to the user's memory, and this time the process is blocked, during which time the process is block. The asynchronous IO is not the same, and when the process initiates an IO operation, the direct return is ignored until the kernel sends a signal telling the process that IO is complete. Throughout this process, the process has not been blocked at all.

There is also a less commonly used signal driven IO, which is signal-driven IO. Overall, There are 5 types of IO models summarized in UNP: Blocking IO, non-blocking io,io multiplexing, signal-driven IO, asynchronous IO. The first four types are synchronous IO. Blocking IO Needless to say. Non-blocking IO, the IO request with the O_nonblock class of flag bits, immediately return, IO is not ready to return an error, need to request the process actively polling continuously send IO requests until returned correctly. IO multiplexing is the same as the non-blocking IO nature, but takes advantage of a new select system call, which is the kernel responsible for the polling operation that the request process is supposed to do. It seems that there is more than a non-blocking IO a system call overhead, but because it can support multiple IO, it is more efficient. Signal-driven IO, called sigaltion system call, when the IO data in the kernel is ready to notify the request process with Sigio signal, the request process then reads the data from the kernel into the user space, this step is blocked.
Asynchronous IO, as defined, does not notify the request process because the IO operation is blocked and the IO operation is complete.
Comparison of each IO model:

Figure 12 Comparison of various IO models

As described above, the difference between non-blocking io and asynchronous io is obvious. In non-blocking io, although the process will not be blocked for most of the time, it still requires the process to go to the active check, and when the data is ready, it is also necessary for the process to proactively call Recvfrom to copy the data to the user's memory. and asynchronous Io is completely different. It's like a user process handing over an entire IO operation to someone else (kernel) and then sending a signal notification when someone finishes it. During this time, the user process does not need to check the status of the IO operation, nor does it need to actively copy the data.

Reference documents:
IO-synchronous, asynchronous, blocking, non-blocking: http://blog.csdn.net/historyasamirror/article/details/5778378

Using event-driven models for efficient and stable network server programs: http://www.ibm.com/developerworks/cn/linux/l-cn-edntwk/

[Reprint] Network IO Model