Chapter 4-02

Please indicate the Source:http://blog.csdn.net/gaoxiangnumber1
Welcome to my Github:https://github.com/gaoxiangnumber1
4.6 Multi-Threading and IO
? This book discusses only synchronous Io, including blocking and non-blocking, and does not discuss asynchronous IO (AIO). In the multi-threaded network programming, several questions are: How to handle IO? Can multiple threads read and write the same socket file descriptor at the same time? We know that multitasking with multiple sockets at the same time can often improve efficiency, so can multithreading with the same socket improve efficiency?
? First, the system call to manipulate the file descriptor itself is thread-safe, and we don't have to worry that multiple threads manipulating the file descriptor can cause a process crash or a kernel crash.
? However, it is cumbersome for multiple threads to manipulate the same socket file descriptor at the same time. Here are some things to consider:
1. If one thread is blocking the read (2) of a socket, and the other thread is close (2) the socket.
2. If a thread is blocking the Accept (2) One of the listening sockets, and the other thread is close (2) the socket.
3. Worse, one thread is preparing to read (2) a socket, while another thread is close (2) to the socket, and the third thread happens to open (2) Another file descriptor whose FD number is exactly the same as the previous socket. The logic of this procedure is confusing (§4.7).
The above cases reflect a problem in the design of the programming logic.
It is not always good to assume that you do not consider closing the file descriptor, only read and write. Because the socket read-write feature is not guaranteed integrity, read 100 bytes may only return 20 bytes, the write operation is the same.
1. If two threads read the same TCP socket at the same time, and each of the two threads receives a portion of the data, how do you spell the data into a complete message? How do I know which part of the data arrives first?
2. If two threads write the same TCP socket at the same time, each thread emits only half a message, and the receiver receives the data.
3. If you assign a lock to each TCP socket, so that only one thread can read or write this socket at the same time, it solves the problem, but it is not as straightforward as always letting the same thread operate the socket directly.
4. For non-blocking IO, the situation is the same, and the integrity and atomicity of sending and receiving messages is almost impossible to secure with locks because it blocks other IO threads.
In this view, theoretically only read and write can be divided into two threads, because the TCP socket is bidirectional IO. Is the question worth breaking read and write into two threads?
? The above is all about network IO, so can multiple threads speed up disk IO? The first step is to avoid lseek (2)/Read (2) race condition (§4.2). After doing this, using multiple threads to read or write the same file does not speed up.
? Not only that, multiple threads that read or write on the same disk will not necessarily speed up the files. Because each disk has an operations queue, multiple threads read and write requests to the kernel that are queued for execution. Multithreaded reading of these hot data can be faster than a single thread only if the kernel caches most of the data.
? One idea for multithreaded disk IO is that each disk is equipped with one thread, and all IO for this disk is moved to the same thread, which may prevent or reduce lock contention in the kernel.
? I think it is clear that the program should be written in a "clearly correct" way, and that a file is read and written by only one thread in a process.
For the sake of simplicity, multithreaded procedures should follow the principle that each file descriptor is operated by only one thread, making it easy to solve the ordering problem of message sending and receiving, and also avoiding the various race condition of closing file descriptors. A thread can manipulate multiple file descriptors, but one thread cannot manipulate file descriptors owned by other threads.
Epoll also follows the same principle. The Linux documentation does not state: When a line is impersonating blocks on epoll_wait (), another thread goes to this epoll fd to add a new monitoring FD to what happens. Will the event on the new FD be returned in this epoll_wait () call? For the sake of safety, we should put the operation of the same Epoll FD (add, delete, modify, wait) on the same thread, which is why we need to Muduo::eventloop::wakeup (). The general program does not directly use Epoll, read, write, these underlying operations are done by the network library.
There are two exceptions to this rule: for disk files, multiple threads can simultaneously invoke Pread (2)/pwrite (2) to read and write the same file when necessary, and for UDP, because the protocol itself guarantees the atomicity of the message, Under appropriate conditions (such as the message being independent of each other) multiple threads can read and write to the same UDP file descriptor at the same time.
4.7 Wrapping file descriptors with Raii
? The file descriptor for Linux is a small integer, and when the program is just started, 0 is the standard input, 1 is the standard output, and 2 is the standard error. At this point, if we open a new file, its file descriptor will be 3, because the POSIX standard requires that each new open file (including socket) must use the current minimum available file description symbol code.
? POSIX this way of assigning file descriptors can easily cause a string of words. For example, a thread is preparing to read (2) a socket, and the second thread almost simultaneously close (2) the socket, and the third thread exactly open (2) Another file descriptor whose number is exactly the same as the previous socket (because the smaller number is occupied). At this point the first thread may read data that does not belong to it, and it also destroys the functionality of the third thread, because the first thread reads the data (the data for the TCP connection can only be read once and the disk file moves the current location). In another case, a thread receives a more time-consuming request from FD = 8, starts processing the request, and remembers to send the response result to FD = 8. However, during the processing, FD = 8 is disconnected, closed, and a new connection arrives, which happens to use the same FD = 8. When the thread completes the response calculation and sends the result to FD = 8 o'clock, the receiver has changed and the consequences are unknown.
In a single-threaded program, you can avoid cross-talk by some sort of global table, which is not efficient in multithreaded programs because it means that the global table is locked for each read and write.
? Use RAII to solve this problem in C + +. Wraps the file descriptor with the socket object, and all read and write operations on the file descriptor are done through this object, closing the file descriptor in the object's destructor. As long as the socket object is still alive, there will be no other socket object that has the same file descriptor as it, and it is not possible to string words. The rest is to do multi-threaded object life management, see Chapter 1th.
Why should the service-side program not turn off standard output (FD = 1) and standard error (FD = 2)?
Because some third-party libraries print error messages to stdout or stderr in special cases, if our program turns off standard output (FD = 1) and standard error (FD = 2), these two file descriptors may be occupied by the network connection, resulting in the other party receiving inexplicable data. The correct approach is to redirect stdout or stderr to disk files (preferably not/dev/null) so that we do not lose critical diagnostic information. This should be done by the process that started the service program 29, which is transparent to the service program itself.

? A feature of modern C + + is the progress of object life management, i.e. no manual delete objects are required. In network programming, some objects are long-lived (such as TCPServer), and some objects are ephemeral (such as tcpconnection). Long-lived objects tend to have the same length of life as the entire program, and the objects on the stack that use global objects (or SCOPED_PTR) or make main () are the same. For short-lived objects, their lifetimes may not be entirely controlled by us, such as the other client disconnects a TCP socket, it corresponds to the service side of the process of the Tcpconnection object (must be a heap object, it is not possible to be a stack object) Life also came to an end. But at this point we cannot delete the object immediately, as other places may still hold its reference, and a hasty delete will cause a null-dangling pointer. You can safely destroy an object only if it is not held elsewhere, which uses a reference count. See Chapter 1th.
In non-blocking network programming, we face a scenario where a request is received from a TCP connection, the program starts processing the request, the processing takes a certain amount of time, and in order to avoid blocking other request, the program remembers the TCP connection that sent the request. The request is processed in a pool of threads, and after processing, the response is sent back to TCP connection a. However, during request processing, the client disconnects from TCP connection A and the other client just creates a new connection B. Instead of remembering the file descriptor of TCP connection A, our program should hold the Tcpconnection object that encapsulates the socket, ensuring that the file descriptor for TCP connection A is not closed during request processing. Or holding a weak reference to the Tcpconnection object (weak_ptr) to know if the socket connection was closed during request processing, and whether FD = 8 's file descriptor was a "past" or "life".
Otherwise, the old TCP connection a disconnects, the Tcpconnection object is destroyed, the old file descriptor (RAII) is closed, and the socket file descriptor of the new connection B may be equal to the previously disconnected TCP connection (POSIX requires that the current smallest available integer be picked up each time a new file descriptor is created). When the program finishes processing the request for the old connection, it is possible to send the response to the new TCP connection B, resulting in a serial conversation.
To prevent access to invalid objects or to network strings, Muduo uses shared_ptr to manage the lifetime of the tcpconnection. This is the only object that manages lifetimes with reference counting.
4.8 Raii and fork ()
When writing C + +, we always try to ensure that the construction and destruction of objects are in pairs, otherwise there will almost certainly be a memory leak. This is not difficult to do (§1.7). Using this feature, we can use objects to package resources and unify resource management and Object Life Management (RAII).
? However, if the program will fork (), this assumption will be destroyed. Consider the following example, the Foo object was constructed once, but it was refactored two times.

 #include <unistd.h>   #include <stdio.h>  class  Foo{public : Foo () {printf  (" ctor\n "); } void  doit () {printf  (" doit () \ n ");    } ~foo () {printf  ( "dtor\n" ); }};    int  Main () {foo foo; //call constructor  Fork (); //fork for two processes  foo.doit ();    //use foo  in parent-child processes //destructor is called two times, and the parent and child processes are } Output:ctordoit () dtordoit () dtor  

? If the Foo class encapsulates a resource that is not inherited by the quilt process, then foo::d oit () function is not in the child process. And we have no way to automatically prevent this, you can not always request a resource to call once Pthread_atfork () it? After
? Fork (), the child process inherits almost all of the state of the parent process, but there are a few exceptions. The child process inherits the address space and file descriptors, so the Raii class for managing dynamic memory and file descriptors works correctly. However, the child process does not inherit:
1. The memory lock of the parent process, Mlock (2), Mlockall (2).
2. The file lock of the parent process, FCNTL (2).
3. Some timers of the parent process, Setitimer (2), Alarm (2), Timer_create (2), and so on.
4. Other, see Man 2 fork.
? Usually we use the RAII technique to manage the resources above (locking unlock, create a destroy timer, and so on), but the sub-processes that come out of the fork () do not necessarily work properly because the resources are freed at fork (). For example, using the RAII technique to encapsulate Timer_create ()/timer_delete (), the destructor call Timer_delete () in a child process may go wrong because an attempt was made to free a nonexistent resource. Or worse, release the timer that other objects hold (if it happens to be a new timer_t).
? As a result, when writing a service-side program, "whether to allow fork ()" is an issue that should be considered at the outset, and the use of fork () in a program that is not ready for fork (), will encounter unpredictable problems.
4.9 Multi-Threading with fork ()
? The collaboration between multi-threading and fork () 30 is poor. This is a historical burden on the POSIX family of operating systems. Because the program is single-threaded for a long time, fork () works fine. When multithreading was introduced in the early 1990s, the application of fork () was reduced to a large extent.

Fork () generally cannot call 31 32 in a multithreaded program because the fork () of Linux only clones the thread of control of the current threads and does not clone other threads. After fork (), other threads disappear except for the current thread. In other words, you cannot fork () a Cheng Zi process as much as the parent process. Linux does not have Forkall () such a system call, Forkall () is actually very difficult to do, because other threads may be waiting on the condition variable, may block on the system call, may wait for the mutex to enter the critical section, but also in the dense calculation, These are not good enough to move to the sub-process.

? fork () Then there is only one thread in the child process, and the other threads are gone, which creates a dangerous situation. Other threads may be in the critical section, hold a lock, and suddenly die, and never have a chance to unlock it. If a child process attempts to lock the same mutex again, it will immediately deadlock.
After fork (), the child process is equivalent to being in signal handler, and you cannot call the thread-safe function (unless it is reentrant) and can only invoke the function of asynchronous signal security (ASYNC-SIGNAL-SAFE). For example, after fork (), the child process cannot be called:
1.malloc (3). Because malloc () is almost certainly locked when it accesses the global state.
2. Any function that may allocate or free memory, including new, Map::insert (), snprintf33, and so on.
3. Any pthreads function. You cannot use Pthread_cond_signal () to notify the parent process, and you can only synchronize 34 by reading and writing pipe (2).
4.PRINTF () series functions, because other threads may hold stdout/stderr locks exactly.
5. Any function other than the "signal security" function explicitly listed in Man 7 signal.

? The only safe way to do this is to call EXEC () immediately after the fork () to execute another program that completely partitions the child process from the parent process.
4.1 Multi-Threading with signal
? Linux/unix signal (signal) and multi-threaded is not 35. In the single-threaded era, writing signal processing functions (signal handler) is a tricky task, as signal interrupts the running thread of control, in signal Only function 36 of Async-signal-safe can be called in handler, the so-called "reentrant (reentrant)" function. Not every thread-safe function is reentrant, see §4.9 example.

In addition, if the global data needs to be modified in signal handler, the modified variable must be 38 of the sig_atomic_t type. Otherwise the interrupted function may not immediately see signal handler modified data after resuming execution, because the compiler might assume that the variable will not be altered elsewhere, thus optimizing memory access.

In the age of multithreading, the semantics of signal are more complex. The signal is divided into two categories: Send to a thread (SIGSEGV), sent to any thread in the process (SIGTERM), but also consider masking mask to the signal and so on. In particular, no pthreads functions can be called in signal handler, and other threads cannot be notified through condition variable.
? In multithreaded programs, the first principle of using signal is not to use signal39. Including
1. Do not use signal as a means of IPC, including not using signals such as SIGUSR1 to trigger the behavior of the server. If you really need to, you can use the §9.5 introduced to increase the listening port way to achieve bidirectional, remote access to the process control.
2. Do not use timing functions based on the signal implementation, including Alarm/ualarm/setitimer/timer_create, Sleep/usleep, and so on.
3. Do not actively handle various abnormal signals (SIGTERM, SIGINT, etc.), using only the default semantics: End process. One exception: Sigpipe, a server program typically ignores this signal 40, otherwise the program terminates unexpectedly if the other person disconnects and the machine continues to write.
4. In the absence of alternative alternatives (for example, to process the SIGCHLD signal), convert the asynchronous signal to a synchronized file descriptor event. The traditional approach is to write a byte in the signal handler to a specific pipe (2), read from the pipe in the main program, and into the unified IO Event processing framework. The modern Linux approach uses SIGNALFD (2) to convert signals directly to file descriptor events, thus avoiding the use of signal handler41 fundamentally.

The revelation of new system call of 4.11 Linux
? This section originates from blog 42, omitting SIGNALFD, TIMERFD, EVENTFD, and so on.

? Roughly from the Linux kernel 2.6.27, any syscall that creates a file descriptor generally adds an additional flags parameter that can be directly specified O_nonblock and fd_cloexec, for example:
accept4-2.6.28, eventfd2-2.6.27, inotify_init1-2.6.27, pipe2-2.6.27, signalfd4-2.6.27, timerfd_create-2.6.25
The above 6 syscall, except the last one is the new function of 2.6.25, the rest is to enhance the original call, the number tail number removed is the original syscall.
? The function of O_nonblock is to turn on non-blocking IO, and the file descriptor is blocked by default. These system calls that create file descriptors can set the O_nonblock option directly.
In addition, the following new system call can turn on the fd_cloexec option when creating a file description:
dup3-2.6.27, epoll_create1-2.6.27, socket-2.6.27
? The Fd_cloexec function is to let the program exec (), the process will automatically close this file descriptor. The file description is inherited by the quilt process by default (this is a typical IPC of traditional UNIX, such as pipe (2) One-way communication between parent and child processes).
The above 8 new Syscall allow direct designation of FD_CLOEXEC, stating that the main purpose of fork () is no longer to create worker process and maintain communication through shared file descriptors and parent processes, but rather to create clean processes (after fork (), exec () immediately after ), with little involvement with the parent process. To avoid the race condition of file descriptor leaks between fork () + exec (), this introduces fd_cloexec parameters on almost all system calls that can create new file descriptors, see Secure file descriptor handling 43.
? The above two flags show that the mainstream model of Linux server development is being transformed from the fork () + Worker processes model to the recommended multithreaded model in the 3rd chapter. The use of fork () decreases, and in the future, only programs that are specifically responsible for initiating other processes will call fork (), whereas a generic Web server program will no longer fork () out of the child process. One reason is that fork () is generally not callable in multithreaded programs (§4.9).
Summary
? This chapter discusses only the technical aspects of multithreaded programming, and does not discuss design aspects, especially how to plan the number and use of threads for a multithreaded service program. I personally follow the principles of writing multithreaded C + + programs as follows:
1. Threads are valuable, and a program can use several or more than 10 threads. A machine should not run hundreds of, thousands of user threads at the same time, which greatly increases the burden on the kernel scheduler and reduces overall performance.
2. The creation and destruction of threads is a cost, and a program is best to create the required threads at the outset and use them repeatedly. Do not repeatedly create and destroy threads during run time, and if this is necessary, it is best to reduce the frequency to 1 minutes and 1 times (or less).
3. Each thread should have clear responsibilities such as IO threads (running Eventloop::loop (), processing IO events), compute threads (in ThreadPool, responsible for calculations), and so on (§3.5.3).
4. The interaction between threads should be as simple as possible, and ideally, the threads would interact only with message passing (for example, Blockingqueue). If you must use a lock, it is best to avoid a thread that holds two or more locks at the same time, which completely prevents deadlocks.
5. Consider in advance whether a mutable shared object will be exposed to which threads, whether each thread reads or writes, and whether or not read and write are possible concurrently.
Please indicate the Source:http://blog.csdn.net/gaoxiangnumber1
Welcome to my Github:https://github.com/gaoxiangnumber1

Chapter 4-02