Linux multi-thread socket programming experience

Some experiences have been found in ningx (someone used it to maintain 30 thousand processing connections at the same time). These are not only servers that use models such as web, it is also suitable for all others, such as games and ftp servers.
The key issue for maintaining a high number of online users is that the event processing model requires m: n, m far <n, m is socket, and n is the connection socket, no matter how many CPUs you have (such as 2 CPUs and 4 cores), the number of threads that the computer can process at the same time is very small. The key to socket lies in the network and other IO delays, if the thread is congested on a network I/O, all the basic optimization algorithms are useless, because the poor algorithm can only be considered for a short time, but the network I/O latency is hard to explain, A Senior said a good sentence:The multi-server network collaboration program always remembers that reducing one route is more cost-effective than optimizing one algorithm by 100..
Nginx strictly separates the processing programs (this is why the Flower web server I am working on is also based on this principle). When epoll and other events are notified, nginx maintains two linked lists globally, A read connection table, a write linked list, epoll, and other system network notifications notify a socket that the status is saved in the corresponding variable after it is operable, and then the m threads are used to poll and execute the task, in this way, the thread does not have to wait for network io congestion. A normal read socket data needs to be cyclically until the data is read or an error is generated, I maintain a unified socket fd table globally, which is determined by the linux features, because the linux socket fd starts from a very small value, in addition, there will be no duplicates within the same time (it is estimated that there is a thread lock inside). All fd itself is an important identifier, and a large fd will be reused from an early stage after it is released, winsows is different. It is a long pointer. All my global status tables need a logo that uses fd as the index. It is a general number that maintains the number of online users at the same time. I set a 20000 status table. The memory for the current stack is very small and will not be used up.
Then, I set two independent State linked lists for read and write, which are processed by m threads. The difference is that the two tables have only one fd entry, the rest of the data is saved in the global state table of heap allocation. I set many Status values for the global state table, such as read write readed writed needRepetRead needrepetWrite restart, in this way, I can use the new State linked list to maintain other processes. The status of the global table is different when the locks are different, thus achieving the collaboration of multiple processing programs, the mutex lock only exists in each processing link. If the read link is slow, the number of read threads can be increased. If the write speed is slow, the write speed can be increased, if the processing process is slow, you can increase the processing speed to facilitate later debugging and optimization of the system. However, the processing of Global tables is not affected in every step, it won't be unable to perform the global state because a thread is blocked or dead.

When the initial model is also using sendfile + epoll, my server has slightly exceeded nginx after testing. The following section further standardizes and optimizes the expansion.