Python--socket Sticky Bag

Socket Sticky Pack 1 What is a sticky bag

Note: Only TCP has sticky packet phenomenon, UDP never sticky packet, first need to master a Socket transceiver message principle,

The so-called sticky packet problem is mainly because the receiver does not know the boundary between the message, do not know how many bytes of data extracted at once.

The sender can be a K-K to send the data, and the receiving side of the application can be two K two k to take the data, of course, it is possible to take 3 K or 6K data at a time, or only a few bytes of data at a time, that is, the application sees the data is a whole, or a stream (stream), The number of bytes of a message is not visible to the application, so the TCP protocol is a stream-oriented protocol, which is also the cause of the sticky packet problem.

The TCP protocol layer renders the data segment that makes up the entire message before it is rendered in the kernel buffer.

For example, the TCP-based socket client uploads a file to the server, sending the file content is sent in accordance with a paragraph of the stream of bytes, in the receiving party, at all, do not know where the file's stream of bytes started, where to end.

In addition, the packet caused by the sender is caused by the TCP protocol itself, TCP to improve transmission efficiency, the sender often to collect enough data before sending a TCP segment. If there are few data to send in a few consecutive times, TCP will usually send the data to a TCP segment based on the optimization algorithm, and the receiver receives the sticky packet data.

1.TCP (Transmission Control Protocol) is a connection-oriented, stream-oriented, providing high-reliability services, both ends of the transceiver (client and server) have one by one pairs of sockets, so the sending side in order to send multiple packets to the receiving end of the package, more efficient to send to each other, using the optimization method (Nagle algorithm), Data that is small in number and smaller in size is combined into a large chunk of data and then encapsulated. In this way, the receiving side, it is difficult to distinguish, must provide a scientific unpacking mechanism, that is, flow-oriented communication is no message protection boundary.

2.TCP is based on data flow, so send and receive messages can not be empty, which requires both the client and the server to add a null message processing mechanism to place the program card master.

TCP protocol data will not be lost, no packet received, the next receive, will continue to receive the last time, the end will always receive an ACK to clear the buffer content, the data is reliable, but sticky packet.

In both cases, the sticky pack will occur:

The sending side need to wait for the buffer full to send out, resulting in sticky packets (send data time interval is very short, data volume is small, to join together to produce sticky packets)

#服务端import Socket,subprocesss=socket.socket (Socket.af_inet,socket. Sock_stream) S.bind (("127.0.0.1", 8000)) S.listen (5) conn,addr=s.accept () data1=conn.recv (1024x768) DATA2=CONN.RECV (1024 Print ("First package", data1) print ("Second package", data2) Conn.close () S.close () execution Results first package B ' HELLOWORLDSB ' second packet B ' #客户端import socket, Subprocesss=socket.socket (Socket.af_inet,socket. Sock_stream) S.connect (("127.0.0.1", 8000)) S.send ("HelloWorld". Encode ("Utf-8")) S.send ("SB". Encode ("Utf-8")) S.close ()

Solve Sticky pack problem 1:low. The low method adds a time delay to the client, which solves the problem temporarily.

#服务端import Socket,subprocesss=socket.socket (Socket.af_inet,socket. Sock_stream) S.bind (("127.0.0.1", 8000)) S.listen (5) conn,addr=s.accept () data1=conn.recv (1024x768) DATA2=CONN.RECV (1024 Print ("First package", data1) print ("Second package", data2) Conn.close () S.close () execution Results first package B ' HelloWorld ' second package B ' SB ' #客户端import socket, Subprocess,times=socket.socket (Socket.af_inet,socket. Sock_stream) S.connect (("127.0.0.1", 8000)) S.send ("HelloWorld". Encode ("Utf-8")) Time.sleep (3) s.send ("SB". Encode (" Utf-8 ")) S.close ()

　　

The receiver does not receive the buffer in time packets, resulting in multiple packets received (the client sent a piece of data, the server only received a small portion of the service end of the next time to collect the last data from the buffer, resulting in sticky packets)

#服务端import Socket,subprocess,times=socket.socket (Socket.af_inet,socket. Sock_stream) S.bind (("127.0.0.1", 8000)) S.listen (5) conn,addr=s.accept () data1=conn.recv (1)   #第一次收了个 "H" # Time.sleep (5) data2=conn.recv (1024x768) #第二次收了 "Elloworld" Print ("First package", data1) print ("Second package", data2) Conn.close () S.close () #客户端import Socket,subprocess,times=socket.socket (Socket.af_inet,socket. Sock_stream) S.connect (("127.0.0.1", 8000)) S.send ("HelloWorld". Encode ("Utf-8")) Time.sleep (3) s.send ("SB". Encode (" Utf-8 ")) S.close ()

Solve Sticky pack Problem 2: Reduce a low by less than Method 1　

The root of the problem is that the receiver does not know the length of the stream to be transmitted by the sender, so the way to solve the sticky packet is around, how to let the sending side before sending the data, the total size of the bytes will be sent to the receiving end to know, and then receive the end of a dead loop to receive all the data

Workaround for low version

Analog Ethernet Protocol Encapsulation header:

Header features: fixed length

Contains descriptive information about the data that will be sent

#服务端import Socket,subprocess,structs=socket.socket (Socket.af_inet,socket.            Sock_stream) S.bind (("127.0.0.1", 8000)) S.listen (5) while true:conn,addr=s.accept () while True:try: MSG=CONN.RECV (1024x768) res=subprocess. Popen (Msg.decode ("Utf-8"), Shell=true, STDOUT=SUBPROCESS.P IPE, Stderr=subprocess. PIPE) Out_res=res.stdout.read () Err_res=res.stderr.read () data_size= (Len (out_res) +len (err _res)) #发送报头 Conn.send (Struct.pack ("I", data_size)) #发送真实数据部分 Conn.send (out_res ) Conn.send (err_res) except Exception:break conn.close () s.close () #客户端 # Sticky Pack Own Package Header Import Soc Ket,structs=socket.socket (Socket.af_inet,socket. Sock_stream) S.connect (("127.0.0.1", 8000)) while True:msg=input ("Please enter command:"). Strip () if not msg:continue s.send (bytes    (msg,encoding= "Utf-8")) #收报头 BAOTOU=S.RECV (4) data_size=struct.unpack ("I", Baotou) [0] #收收据 recv_size=0 recv_data=b "" While Recv_size <data_size: DATA=S.RECV (1024x768) recv_size+=len (data) recv_data+=data print (Recv_data.decode ("GBK")) S.close ()

Add a custom fixed-length header to the byte stream, including the byte stream length in the header, and then send to the peer at a time, and the peer receives the fixed-length header from the cache before fetching the real data

3.struct Module

The module can turn a type, such as a number, into a fixed-length bytes

>>> res=struct.pack (' i ', 1111111111111) #打包成固定长度的bytes

>>> struct.unpack ("I", Res) #解包

2 How the Great God solves the sticky bag

We can make a dictionary of the header, the dictionary contains the details of the real data to be sent, then JSON serialization, and then use struck to package the serialized length of the data into 4 bytes (4 of them enough)

When sending:

First outgoing header length

Re-encode the header contents and send

Finally send the real content

When receiving:

Initiator header length, removed with struct

The header content is charged according to the length taken, then decoded and deserialized

Extract the details of the data to be fetched from the deserialized results and then fetch the actual data content

#服务端import Socket,subprocess,struct,jsons=socket.socket (Socket.af_inet,socket.            Sock_stream) S.bind (("127.0.0.1", 8000)) S.listen (5) while true:conn,addr=s.accept () while True:try: MSG=CONN.RECV (1024x768) res=subprocess. Popen (Msg.decode ("Utf-8"), Shell=true, STDOUT=SUBPROCESS.P IPE, Stderr=subprocess. PIPE) Out_res=res.stdout.read () Err_res=res.stderr.read () Data_size=len (out_res) +len (err_ RES) head_dic={"Data_size":d ata_size} head_json=json.dumps (Head_dic) Head_bytes=head_json . Encode ("Utf-8") #报头 #part1: The length of the first header Head_len=len (head_bytes) conn.send (Struct.pack ("I", hea D_len)) #part2: Resend Header Conn.send (head_bytes) #part3: Last Send data True part Conn.send (err_res ) Conn.send (out_res) except Exception:break conn.close() S.close () #客户端import socket,struct,jsons=socket.socket (socket.af_inet,socket. Sock_stream) S.connect (("127.0.0.1", 8000)) while True:msg=input ("Please enter command:"). Strip () if not msg:continue s.send (bytes    (msg,encoding= "Utf-8")) #part1: The length of the first header Head_struct=s.recv (4) head_len=struct.unpack ("I", head_struct) [0] #part2: Re-receiving header Head_bytes=s.recv (    Head_len) Head_json=head_bytes.decode ("Utf-8") head_dic=json.loads (Head_json) data_size=head_dic["Data_size"] #part3: Collect Data recv_size=0 recv_data=b "" While Recv_size < Data_size:data=s.recv (1024x768) recv_size+=l En (data) recv_data+=data print (Recv_data.decode ("GBK")) S.close ()

　　

Python--socket Sticky Bag