The path of Python-sticky bag for network programming

• The path of Python-sticky bag for network programming
• Sticky bag??
• Workaround??
• Low method??
• Make a header??

Sticky bag??

From the previous article <python-socket implementation of remote execution command > The problem arises from the problem of sticky bag, what is the sticky bag?

First of all, why is it that 粘包现象只出现在TCP中 only in TCP will occur in the sticky packet phenomenon, the first to explain in detail TCP and UDP bar

Tcp

TCP (Transprot Control Protocol, Transmission Protocol) is connection-oriented, stream-oriented and provides high reliability services. Send and receive each end of the socket (one-to-one mode), so the sending side in order to be sent to the receiving end of the package, more effective send to the other side, using the optimization method (Nagle algorithm), 将多次间隔较小且数据量小的数据 , 合并成一个大的数据块 , 然后进行封包 . must provide a scientific unpacking mechanism, in order to make a reasonable resolution, so that the flow-oriented communication is non- Protection boundaries.

Udp

UDP (User Datagram Protocol, Subscriber Datagram Protocol) is non-connected, message-oriented, providing efficient service. Without the use of block merge optimization algorithm, because UDP support is a one-to-many mode, so the receiver Skbuff (socket buffer) adopted a chain structure to record each incoming UDP packet, in each UDP packet has a message header (message source address, port and other information), so that the receiver end to Said, it is easy to distinguish between processing. Communication that is oriented is a message-protected boundary.

Difference

TCP is based on data flow, so send and receive messages can not be empty, which requires the client and the server to add a null message processing mechanism to prevent the program stuck, and UDP is based on the datagram, even if the sending and receiving empty content, is not an empty message, the UDP protocol will automatically help you encapsulate the message header

Reasons for the occurrence of sticky-bag phenomenon

There are two types of sticky bags

2. The sticky packets caused by the sender

   In this case the sticky packets caused by the TCP protocol itself, TCP in order to improve the efficiency of transmission, the sender is often to collect enough data to send a TCP segment   (the time interval is also sent, The time interval is very short)   If there are few data to be sent several times in a row, TCP will typically synthesize the data to a TCP segment based on the optimization algorithm and send it one at a time, so a few times the data is stuck to the receiver in a package

   as follows:

    # sender sends send for the first time (b "I ' m") # immediately second time, no more than interval send (b "Lyon")-------------# Receive the data = recv (1024) # received two times stuck together with the print (Data.decode ()) # print Result: I ' m Lyon          

4. Adhesive packets caused by the receiving party

   This situation caused by the packet is caused by the receiver is not in time to receive buffer packets, such as the sender sent 10 bytes of data at a time, and the receiver only received 2 bytes, then the remaining 8 bytes of data will be in the buffer waiting to receive, and at this time the sender sends 2 bytes of data, After a while, the receiver receives 20 bytes (greater than the remaining 10 bytes), and the remaining data in the buffer is glued to a packet after the second data is sent, resulting in a sticky packet

   As follows:

   # 发送4字节内容send(b"I‘m ")# 接收1字节,缓冲区还有3字节data1 = recv(1)print("data1:",data1)# 发送4字节内容,粘到缓冲区中剩余的3字节后面send(b"Lyon")# 接收7字节,接收完毕data2 = recv(7)print("data2:",data2)‘‘‘打印结果:data1:Idata2:‘m Lyon‘‘‘

So, the so-called sticky packet problem is mainly because the receiver does not know the boundary between the message, do not know how many bytes of data extracted at a time caused by

Workaround??

Since the packets are sticky because the receiver doesn't know the message bounds, we create the boundaries ourselves.

Low method??

We just need to subprocess_server.py make a little bit of a change in the last article subprocess_client.py .

subprocess_server_development.py

Import socketImport Subprocesssock = Socket.socket () sock.setsockopt (socket. Sol_socket, SOCKET. SO_REUSEADDR,1) Sock.bind ((' 127.0.0.1 ',8080)) Sock.listen (5)WhileTrue:print ("Waitting for Connection ...") conn, addr = Sock.accept () print ("{}successful connection ...". Format (addr))WhileTrue:# receive instruction cmd = CONN.RECV (1024)IfNot Cmd:print ( "Client is disconnected ...") break print ( "the command is {}". Format (Cmd.decode ())) # get execution result data = subprocess. Popen (Cmd.decode (), Shell=true, stdout=subprocess. PIPE, Stdin=subprocess. PIPE, Stderr=subprocess. PIPE) # get error handle err = Data.stderr.read () if err:res = Err else:res = Data.stdout.read () # send data length conn.send (str (len)). Encode ( Span class= "hljs-string" > ' utf-8 ')) # prevents data from being stuck with two times ready = CONN.RECV (1024) if ready = b ' OK ': # sendall Continuous Call send complete send Conn.sendall (res) conn.close () sock.close ()     

subprocess_client_development.py

Import Socketsock = Socket.socket () sock.connect ((' 127.0.0.1 ',8080))WhileTrue:cmd = input ("Please input the command:"). Strip ()Ifnot cmd:print ( "Can ' t Empty ...")  Continue elif cmd =  ' exit ': break Span class= "hljs-comment" ># send instruction Sock.send (Cmd.encode ( ' utf-8 ')) # gets the length of the data = SOCK.RECV (1024). Decode (  ' utf-8 ') # Send flag Sock.send (b ' OK ') recvsize = 0 data = b "# loop receive while recvsize < int (length): RecvData = sock.recv (1024) recvsize + = Len (recvdata) Data + = RecvData print ( Data.decode ( ' GBK ')) sock.close ()        

In this way, we need to send the data size in advance, which will undoubtedly amplify the performance loss caused by network latency.

Make a header??

Since we need to send the size of the past, then we can add a custom fixed-length header, the header of the replacement data size and other information, and then send the past directly, the other side as long as the receipt of the first from the header, then fetch the data

So we just need to fix the length of the header, we can use the struct module to make the header, just a little modification to the above method

subprocess_struct_server.py

Import Socket,structImport Subprocesssock = Socket.socket () sock.setsockopt (socket. Sol_socket, SOCKET. SO_REUSEADDR,1) Sock.bind ((' 127.0.0.1 ',8080)) Sock.listen (5)WhileTrue:print ("Waitting for Connection ...") conn, addr = Sock.accept () print ( "{}successful connection ...". Format (addr)) while true:cmd = Conn.recv (1024) Span class= "Hljs-keyword" >if not cmd:print ( "Client is Disconnected ") break print (" the command is {} ". Format ( Cmd.decode ())) data = subprocess. Popen (Cmd.decode (), Shell=true, stdout=subprocess. PIPE, Stdin=subprocess. PIPE, Stderr=subprocess. PIPE) Err = Data.stderr.read () if err:res = Err else:res = Data.st Dout.read () # make 4-bit fixed header and send Conn.send (Struct.pack ( ' I ', Len (res) )) # Direct loop send Conn.sendall (res) conn.close () sock.close ()     

subprocess_struct_client.py

Import Socket,structsock = Socket.socket () sock.connect ((' 127.0.0.1 ',8080))WhileTrue:cmd = input (if not cmd:print ( "Can ' t Empty ...")  Continue elif cmd =  ' exit ': break Sock.send (Cmd.encode ( ' utf-8 ')) res = SOCK.RECV (4) # unlock header take out data length = Struct.unpack ( ' I ', res) [0] Recvsize = 0 data = b "# Loop receive while recvsize < Length:data + = SOCK.RECV (1024) recvsize + = Len (data ) Print (Data.decode (      

Python path-sticky packet for network programming