Bug that plagued me for almost a month

Finally solved yesterday...

1. Background:

As the number of users increases, UDP packet loss increases. In fact, the packet volume is not large. UDP packets of no more than packets per second during peak periods should not be lost.

The previous generation's bottom-layer network code is single-threaded, so they decided to separate network threads to improve efficiency.

2. I have learned about the operating system through multithreading policies, and I should know the examples of producers and consumers. You only need to know the example of. In this case, the lockless ring buffer can be used to avoid the price of mutex (although in fact the price of mutex is not large ). For each socket, from the logic thread perspective (that is, the thread that runs main), there are two corresponding buffers: recvbuffer and sendbuffer. each buffer has a read cursor and a write cursor. The producer modifies the write cursor, and the consumer modifies the read cursor. The buffer between the two cursor is valid data. For recvbuffer, the logical thread is the consumer and the network thread is the producer. For sendbuffer, the opposite is true. 3. The structure of the circular buffer is very simple ,. Note: When the buffer is set to the power of 2, the cursor is calculated as follows: Pos = (Pos + offset) & (m_maxsize-1 ); if m_maxsize is not a power of 2, only the % operator can be used. operation: do not go into details. Note that an element must be empty in the ring buffer to distinguish between full and empty. Null condition: readpos = writepos; full condition: (writepos + 1) & (m_maxsize-1) = readpos; if no element location is left blank, it cannot be full or empty. when a buffer wrap occurs, the blue is valid data: in this case, we cannot use the sendto/recvfrom or Recv/send function because it can only process consecutive buffers. Therefore, writev/readv, recvmsg/sendmsg should be used to support iovec, so-called clustered write/distributed read. The recvmsg/sendmsg is selected, and the reason is described later. If the Recv/send function is the true subset of sendto/recvfrom, sendto/recvfrom is the true subset of recvmsg/sendmsg. 4. How to lock-free read/write cursors is only an int variable. If there is only one writer, interlocked functions can be used to ensure read and write security. In x86 series CPUs, this series of functions maintain a hardware signal on the bus to prevent other CPUs from accessing the same memory address. (This sentence is excerpted from the core programming language of Windows.) Linux has three functions, including _ sync_add_and_fetch. For more information, see. 5. Some people may doubt the performance and performance of multithreading. All are worrying. Multiple. Even if it is a single CPU, it cannot be the same as the single-threaded network library. There is also a copy between the buffer; it is clearly pointed out that the user space is 0 copies. (Recvmsg this copy from the kernel to the user space is not counted, because it is absolutely unavoidable, unless the Windows iocp is used) 6. introduction of UDP packet sending and receiving (not about TCP, TCP connection-oriented, very simple to send and receive, and almost no department) previously only write TCP, this time due to packet loss problems, it took two or three days to add UDP support temporarily, but the test took at least two weeks to ensure stability. The simplicity of sending and receiving packets in a single thread: We recvfrom a package, process the package, return the package, only need to return the address obtained by recvfrom, or a specified address. The same is true for sending packets. A package corresponds to an address, which is easy. The complexity of sending and receiving packets in multiple threads: (1) network threads receive many packets and the logic threads process them one by one. The source address corresponding to these packages cannot be obtained from recvfrom. Therefore, each time a network thread receives a packet, it not only needs to put the packet into the receiving buffer, but also needs to write the address at the end of the packet. This address must be transparent to the application. If the application does not specify an address, it directly calls sendpacket (my_packet ), then the network thread automatically uses the package source address as the destination address (this is also a back package ). (2) The received package must be indicated with a length of 0 to be copied and robust. (what if the logic thread fails to unpack the package and cannot adjust the read cursor correctly ?) Therefore, the message length must be provided. The network library must adjust the read cursor after each packet is parsed. This length is transparent to the application, so you do not have to worry about it. Recvmsg is a function that supports obtaining the peer address and discrete reading. Therefore, it is used. (3) You must specify an address for each package. For broadcast, you need to specify the number of addresses, followed by the address string, followed by the package length and Package content. The format is as follows: naddr + addr_1 + addr_2 +... + addr_n + npacketlen + packetcontent. The above design makes circularbuffer hard to write and requires caution. Because we cannot adjust the write/read cursor every time we write/read data, we need to ensure logical integrity. When I first made this mistake, I ran the program for a day and suddenly collapsed. sendmsg is a function that supports both filling in the address of the other party and writing together, so I chose it. 7. Be careful when sending packets. UDP is not the same as TCP. If there is no traffic control, you cannot send the packets so they will lose packets. But I am not very familiar with whether UDP sendto will return eagain. (Supplement: Yes, you can register epoll_out.) If this check is performed, stop sendto and change to epoll_wait. Some manual restrictions are also imposed, such as sending a message, if the total number of packets reaches 5000, or MB, it is also converted to epoll_wait. There will be severe packet loss. 8. We recommend that you learn Mr Chen's muduo (TCP only ). Now let's talk about bugs.The double-threaded program makes the packet loss rate almost zero, but the program may be suspended for a few days and has to be recalled. Check the core file and find that there is an Assert statement when the logic thread parses the message. It is assumed that the received message length is correct. (If there is no bug in the network library, this assert must be true) but it is obvious that the length information obtained is a large number. Check the hexadecimal notation and find that the value comes from the message body; that is to say, when a network thread receives messages, it will disrupt the receiving buffer? First, we suspect that competition may occur in some places. But I checked it several times and confirmed it was not. Then various critical tests were performed on circularbuffer and passed. The doubts were transferred to the atomic series functions. Because I directly read the int, and there is no atomic_get; I think this is true. Reading must be atomic and can read the old value, but will never read the random value. If the cause is not found, the core file cannot be identified, and the problem is blocked. When I read the code yesterday, I suddenly found some doubts:

Bool datagramsocket: onreadable () {While (true) {buffersequence buffers; m_recvbuf.getspace (buffers, sizeof (INT )); // pre-Write 4-byte length information if (0 = buffers. count) {err <"Recv Buf is not enough, logical thread is too slow"; return true;} sockaddr_in ADDR; msghdr MSG; MSG. msg_name = & ADDR; MSG. msg_namelen = sizeof (ADDR); MSG. msg_iov = buffers. buffers; MSG. msg_iovlen = buffers. count; MSG. msg_control = 0; MSG. msg_controllen = 0; const int nbytes =: recvmsg (m_localsock, & MSG, 0 ); if (-1 = nbytes & (eagain = errno | ewouldblock = errno) {return true;} If (nbytes <0) {err <_ pretty_function _ <"udp fd" <m_localsock <"onread error:" <nbytes <", errno =" <errno; return true;} else {m_recvbuf.pushdataat (& nbytes, sizeof (nbytes), 0); m_recvbuf.pushdataat (& ADDR, sizeof (ADDR), sizeof (nbytes) + nbytes ); m_recvbuf.adjustwriteptr (sizeof (nbytes) + nbytes + sizeof (ADDR);} return true ;}

This function is triggered when epoll_in occurs.

Because the package receiving volume is very large and the package size is too large, the receiving buffer is full during peak hours. The above code has multiple vulnerabilities: 1. Misunderstanding of recvmsg. When the provided buffer is smaller than the size of a UDP datagram, The UDP packet is truncated, And the recvmsg still returns the number of bytes after being truncated. I always thought it was a-1 response and didn't process it. 2. Ignore the pushdataat return value of the ring buffer. If push fails, false is returned. Do not adjust the write cursor, which may lead to disorder in the buffer zone. This is the root cause of the bug! I also remember that the previous three cores share a common feature: writepos is slightly larger than readpos. In fact, the buffer overflow! The reason why the buffer is not determined to be full is that I thought the 16 Mb is enough... Hoho is too common. Do not ignore the return values of non-void functions in the future... With the modified Code:

Bool datagramsocket: onreadable () {While (true) {buffersequence buffers; m_recvbuf.getspace (buffers, sizeof (INT )); // pre-Write 4-byte length information if (0 = buffers. count) {err <"Recv Buf is not enough, logical thread is too slow"; return true;} sockaddr_in ADDR; msghdr MSG; MSG. msg_name = & ADDR; MSG. msg_namelen = sizeof (ADDR); MSG. msg_iov = buffers. buffers; MSG. msg_iovlen = buffers. count; MSG. msg_control = 0; MS G. msg_controllen = 0; const int nbytes =: recvmsg (m_localsock, & MSG, 0 ); if (-1 = nbytes & (eagain = errno | ewouldblock = errno) {return true;} If (nbytes <0) {err <_ pretty_function _ <"udp fd" <m_localsock <"onread error:" <nbytes <", errno =" <errno; return true;} else if (MSG. msg_flags & msg_trunc) {err <"MSG truncated! Only get bytes "<nbytes;} else {If (! M_recvbuf.pushdataat (& nbytes, sizeof (nbytes), 0) |

                !m_recvBuf.PushDataAt(&addr, sizeof(addr), sizeof(nBytes) + nBytes))            {                ERR << "Recv buffer is overflow!";                return true;            }             m_recvBuf.AdjustWritePtr(sizeof(nBytes) + nBytes + sizeof(addr));        }    }    return  true;}

Previous Article: Android multi-point touch (with some code)

Next article: Key Points of counting smart pointers (shared_ptr)