Summary of key issues in network programming and key issues in Network Programming

Summary of key issues in network programming and key issues in Network Programming
Summary of key issues in Network Programming

Summarize the key issues in network programming, including connection establishment, connection disconnection, message arrival, and message sending;

Establish a connection

This includes receiving (accept) new connections from the server and successfully initiating (connect) connections from the client.
Accept accepting connections will be discussed at the end of this Article. Here we will talk about the key points of connect;
Note the following before creating a non-blocking connection:
After the return result of connect/select is returned, the connection may not exist. You need to confirm again whether the connection is successful;

Steps:

Fcntl (sockfd, F_SETFL, flags | O_NONBLOCK); error = 0; if (n = connect (sockfd, saptr, salen) <0) if (errno! = EINPROGRESS) return (-1);/* Do whatever we want while the connect is taking place. */if (n = 0) goto done;/* connect completed immediately */if (n = Select (sockfd + 1, & rset, & wset, NULL, nsec? & Tval: NULL) = 0) {close (sockfd);/* timeout */errno = ETIMEDOUT; return (-1);} if (FD_ISSET (sockfd, & rset) | FD_ISSET (sockfd, & wset) {len = sizeof (error); // check whether the connection is successful twice if (getsockopt (sockfd, SOL_SOCKET, SO_ERROR, & error, & len) <0) return (-1);/* Solaris pending error */} else err_quit ("select error: sockfd not set ");

Disconnected

Including active disconnection (close or shutdown) and passive disconnection (read returns 0 ).

When I plan to disable the network connection, how can I know that the other party has sent the data and has not received it?
Solve the problem at the TCP layer: only half-shutdown () is used for active shutdown. In this way, the write end is disabled on the server side and the data can be read normally; after the client receives the signal of shutdown (read returns 0), it will call shutdown to close the entire connection;
Solution at the application level: the two parties negotiate through a tag and no longer read or write data after the tag, so that the connection can be completely closed;

Note the following when closing the connection:
Whether there is any unsent data. You must ensure that all data in the application buffer is sent before closing the buffer;
We don't need to consider the TCP cache, because when we call shutdown or close, the TCP implementation will send the data in the TCP sending buffer, then send the FIN packet (or combine it into a message );

Message arrival

Message arrival is the most important event. The processing of it determines the network programming style: blocking or non-blocking, packet processing, and how to design the application layer buffer;

Handling Subcontracting

The so-called subcontracting means how to differentiate messages in byte streams;
Common subcontracting methods include:

Note byte alignment for byte order conversion

If binary data is transmitted, the core dump may be directly forced in the buffer area of the byte stream. Because some system access addresses need to be byte aligned, the binary type (such as integer) cannot be accessed at any address. A reasonable method is to copy it to a local variable and then convert the byte order:

int32_t peekInt32() const{    assert(readableBytes() >= sizeof(int32_t));    int32_t be32 = 0;    ::memcpy(&be32,readerIndex_, sizeof(be32) );    return be32toh(be32);}

Implementation of the application layer cache Zone

Note the following before processing data:
When the socket read event arrives, all data must be read at a time. Otherwise, there will always be a readable event, resulting in busy-loop; of course, the read data must be stored in an application-layer buffer;
Because the cache area of the application is limited, you can set a size by default, for example, 2 kb, or do not set the initial size and how much to allocate; in muduo, vector is used as the cache area, which can be dynamically increased;

Tips for using muduo buffer:
Buffe adopts a Data Structure with automatic growth of vector;
When calling from the system kernel, a buffer must be large enough at the application layer. It is best to leave the system recv buffer blank at a time, so that everything can be done at a system call;
Considering the possibility of many concurrent requests in the application buffer zone, it is a serious waste to allocate a large buffer zone for each connection at a time. Chen Shuo recommends using readv to read data into the two addresses at a time, first, fill in the first address. If there is more data, write it to the temporary buffer and append it to the application buffer;

When reading data, use readv, and use a large enough extra space (64 KB). In this way, one read operation is sufficient to empty the buffer area in the socket (generally no more than 64 KB, if you really want to set a large cache area for tcp buffer, you need to adjust the system parameters). If there is not much data, the internal buffer may be installed without additional operations. Otherwise, more data is read to the external cache area and then appended to the internal cache area:

Ssize_t Buffer: readFd (int fd, int * savedErrno) {// saved an ioctl ()/FIONREAD call to tell how much to read char extrabuf [65536]; struct iovec vec [2]; const size_t writable = writableBytes (); vec [0]. iov_base = begin () + writerIndex _; vec [0]. iov_len = writable; vec [1]. iov_base = extrabuf; vec [1]. iov_len = sizeof extrabuf; // when there is enough space in this buffer, don't read into extrabuf. // when Extrabuf is used, we read 128k-1 bytes at most. const int iovcnt = (writable <sizeof extrabuf )? 2: 1; // only one system call: The implementation here is clever const ssize_t n = sockets: readv (fd, vec, iovcnt); if (n <0) {* savedErrno = errno;} else if (implicit_cast <size_t> (n) <= writable) {writerIndex _ + = n;} else {writerIndex _ = buffer _. size (); append (extrabuf, n-writable);} // if (n = writable + sizeof extrabuf) // {// goto line_30; //} return n ;}

Send message

In network programming, data transmission is harder to process than data reception;
To receive data, you only need to peek enough data to receive it from the application buffer and then process it. However, when sending data, you also need to consider the slow reception of the other party, as a result, the tcp sending buffer is accumulated, and the application buffer is accumulated;

For example, a client only sends messages to the echo server but does not intentionally receive them;
If the client only sends but never receives the packet, the packet sent in the past will first cause the tcp receiving buffer of the client to be full, and then the server will be notified through the ack packet, the sliding window here is 0, so no more messages can be sent. The subsequent messages sent by the client will fill up the TCP sending buffer on the server, and then accumulate the sending buffer on the application layer (because it is not blocked ), in the end, the application cache on the server is full or the memory is full;

When data needs to be sent, write () is directly called for sending first. If the sending fails, or not all data has been sent, the data will be added to the sending cache and sent after the write event arrives;
When you directly call write () to send data, you must first add the data to be sent to the cache, and then send the data to the cache, you cannot directly send this data (because there may be data left in the cache area that has not been completely sent)

Void TcpConnection: handleWrite () {loop _-> assertInLoopThread (); if (channel _-> isWriting () {// note that only one write is called here, the write request is not repeatedly called until the EAGAIN error occurs. // The cause is that if the full data is not sent during the first call, the second call is almost certainly an EAGAIN error, // Therefore, a system call is reduced here, which does not affect the correctness, but can reduce the system latency ssize_t n = sockets: write (channel _-> fd (), outputBuffer _. peek (), outputBuffer _. readableBytes (); if (n> 0) {outputBuffer _. retrieve (n); if (outputBuffer _. readableBytes () = 0) {// if the sending cache is empty, do not pay attention to write events. Avoid busy loop channel _-> disableWriting (); // if there is still a callback after the write is completed, add the callback queue to be executed if (writeCompleteCallback _) {loop _-> queueInLoop (boost: bind (writeCompleteCallback _, shared_from_this ();} // if shutdownInLoop is being disabled at this time, call shutdownInLoop to continue executing the close process if (state _ = kDisconnecting) {shutdownInLoop ();}}} else {LOG_SYSERR <"TcpConnection: handleWrite"; // if (state _ = kDisconnecting) // {// shutdownInLoop (); // }}} else {LOG_TRACE <"Connection fd =" <channel _-> fd () <"is down, no more writing ";}}

Message sent

For low-traffic services, you don't have to worry about this event. In addition, "sent" refers to writing data into the buffer zone of the operating system, in the future, the TCP stack will be responsible for data transmission and retransmission, which does not mean that the other party has received the data.

Can I use IO multiplexing with blocking sockets?

Generally, this function is used with a non-blocking socket. If I/O is blocked, the current thread may be blocked in the read/write event, resulting in the inability to continue to process ready events;
Network Programming for beginners may have this idea. After the select statement is returned, if it is a read event, the tcp read buffer must have data at this time, even if the block socket is used for read, it should not be blocked either. However, if a vertex is ignored in this way, the buffer zone does have data, but the data that is likely to arrive does not meet the data size you want to read. In this way, the read call will still be blocked, it is not returned until there is enough data;
Therefore, data reading is not allowed. accept () is always acceptable. Connection events are returned. New users are generally allowed to access the data. At this time, blocked accept () should always be able to return data; however, in some cases, the other party may be disconnected from the connection and send an RST request to the server. As a result, the server has removed the ready connection request, in this scenario, select returns, but accept cannot obtain a new connection, resulting in blocking until the next connection request arrives. (For an example, see UNIX Network Programming Volume 1: section 16.6 of Socket network API non-blocking accept ())
Therefore, IO multiplexing must be used with non-blocking IO at any time;

Zero copy implementation

For the implementation of the kernel layer, the underlying calling is the sendFile () method called by the system;
Zerocopy technology eliminates the need to copy the read buffer of the operating system to the buffer of the program, and directly copy the read buffer to the socket buffer from the buffer of the program;
void swap(Buffer& rhs){ buffer_.swap(rhs.buffer_); // std::vector<char> buffer_; std::swap(readerIndex_, rhs.readerIndex_); std::swap(writerIndex_, rhs.writerIndex_);}Epoll use LT

Epoll is used instead of ET for the following reasons:

It is generally believed that the edge-trigger mode has the advantage of reducing epoll-related system calls, which is not false. However, it is not only called by epoll-related systems in network service programs. To avoid starvation, in edge-trigger mode, you need to perform read/write loop processing on your own. What is the overall performance benefit of the added system calls and the reduced epoll system calls? Only the actual measurements are known and cannot be generalized. To reduce the complexity of processing logic, most of the commonly used event processing libraries adopt the level-trigger mode (such as libevent, boost: asio, muduo, etc)

Reference

UNIX Network Programming Volume 1: Socket network API
Multi-thread Server programming in Linux: using the muduo network library

Posted by: Large CC | 31DEC, 2015
Blog: blog.me115.com [subscription]
Github: Large CC