Python/socket programming of sticky bag

Python/socket programming of sticky bag Sticky Bag:

Only TCP has a urine packet phenomenon, UDP will never sticky packets.

First you need to master a socket to send and receive messages the principle

The sender can be 1k,1k send data and the receiving end of the application can 2k,2k extract data, of course, it may be 3k or more K extract data, that is, the application is not visible, so the TCP protocol is the interface to the flow of the Protocol, This is also prone to sticky packets for the reason that UDP is a smile-oriented protocol, each UDP segment is a message, the application must extract data in the message unit, can not extract any byte of data at once, which is very similar to TCP. How do you define a message? Think the other one-time Write/send data for a message, the need to kill is when the other side send a message, no matter how fragmented Dingcheng, the TCP protocol layer will constitute the entire message of the data segment after the sorting is complete before rendering in the kernel buffer.

For example, the TCP-based socket client to the server to upload files, sent when the content of the file is sent in accordance with a paragraph of the byte stream, the receiver seems more stupid do not know where the text stream from the beginning, where the end.

The so-called sticky packet problem is mainly because the receiver does not know the boundary between the message, do not know how many bytes of data extracted at once caused by

The sticky packets caused by the sender are caused by the TCP protocol itself, and TCP is often needed to collect enough data to send a TCP segment to improve the transmission efficiency. such as the number of times you need to send the data are very few, usually TCP will be based on the optimization algorithm to synthesize the data of a TCP segment once sent out, so that the receiver received the sticky packet data

1. TCP (Transport Control Protocol, transmission Protocol) is connection-oriented, stream-oriented and provides high reliability services. Both ends of the transceiver (client and server side) have one by one pairs of sockets, so the sending side in order to send multiple packets to the receiver, more efficient to the other side, the use of the optimization method (Nagle algorithm), the multiple interval small and small data volume data, combined into a large block of data, and then to the packet. In this way, the receiving end, it is difficult to distinguish out, must provide a scientific unpacking mechanism. That is, stream-oriented communication is a non-message-protected boundary.
2. UDP (User Datagram Protocol, Subscriber Datagram Protocol) is non-connected, message-oriented, providing efficient service. The Block merging optimization algorithm is not used, because UDP supports a one-to-many pattern, so the receiver Skbuff (socket buffer) uses a chain structure to record each incoming UDP packet, in each UDP packet there is a message header (message source address, port and other information), so for the receiving end , it is easy to distinguish between the processing. that is, message-oriented communication is a message-protected boundary.
3. TCP is based on data flow, so send and receive messages can not be empty, which requires the client and the server to add a null message processing mechanism to prevent the program stuck, and UDP is based on the datagram, even if you enter the empty content (direct carriage), it is not an empty message, the UDP protocol will help you encapsulate the message header, the experiment slightly

   UDP Recvfrom is blocked, a recvfrom (x) must be to a sendinto (y), after the X-byte data is completed, if the y>x data is lost, which means that UDP is not sticky packets, but will lose data, unreliable

   TCP protocol data is not lost, no packets are received, the next time it is received, it continues to receive the last time, and the buffer content is always cleared when the ACK is received by the client. The data is reliable, but it will stick to the package.

   In both cases, the sticky pack will occur:

1. The sending side needs to wait for the local buffer full after the send out, resulting in sticky packets (send data time interval is very small, the data is very low, rendezvous in one, producing sticky bag)

2. The receiver does not receive the buffer in time, resulting in multiple packets accepted (the client sends a piece of data, the server only received a small portion of the service end of the next time or from the buffer to take the last remaining data, the resulting sticky packet)

Examples of sticky packs:

Service Side Import Socket Import Subprocessdin=socket.socket (socket.af_inet,socket. SOCK_STREAM) Ip_port= ('127.0.0.1', 8080) din.bind (ip_port) Din.listen (5 ) Conn,deer=din.accept () data1=conn.recv (1024x768) data2=conn.recv (1024x768) Print (data1) Print (DATA2)

ClientImportSocketImportSubprocessdin=Socket.socket (Socket.af_inet,socket. SOCK_STREAM) Ip_port=('127.0.0.1', 8080) Din.connect (ip_port) din.send ('HelloWorld'. Encode ('Utf-8')) Din.send ('SB'. Encode ('Utf-8'))

The method of solving the sticky package by low ratio:

Add a time sleep below the client send to avoid the sticky packet phenomenon. When the server receives the time to sleep, in order to effectively avoid the sticky bag situation

#ClientImportSocketImport TimeImportSubprocessdin=Socket.socket (Socket.af_inet,socket. SOCK_STREAM) Ip_port=('127.0.0.1', 8080) Din.connect (ip_port) din.send ('HelloWorld'. Encode ('Utf-8')) Time.sleep (3) Din.send ('SB'. Encode ('Utf-8'))

#Service SideImportSocketImport TimeImportSubprocessdin=Socket.socket (Socket.af_inet,socket. SOCK_STREAM) Ip_port=('127.0.0.1', 8080) Din.bind (ip_port) Din.listen (5) Conn,deer=din.accept () data1=CONN.RECV (1024) Time.sleep (4) Data2=CONN.RECV (1024)Print(data1)Print(DATA2)

How to solve the sticky package aloud:

Add a custom fixed-length header to the byte stream, the header contains the byte-stream length, and then send to the peer, then the peer receives the fixed-length header from the cache before fetching the true data.

Using a struct module

The module can turn a type, such as a number, into a fixed-length bytes byte

struct module The module can convert a type, such as a number, into a fixed-length bytes

>>> res=struct.pack (' i ', 1111111111111) #打包成固定长度的bytes

>>> struct.unpack ("I", Res) #解包

Take the form of a custom header:

#Service SideImportSocket#Import ModuleImportstruct#Import ModuleImportSubprocess#Import ModuleDin=socket.socket (Socket.af_inet,socket. SOCK_STREAM)#transmission in the form of data streams based on the network familyIp= ('127.0.0.1', 8080)#identifies the server-side unique addressDin.bind (IP)#Binding AddressDin.listen (5)#set maximum number of link buffer pools whiletrue:conn,addr=din.accept ()#wait for link to enter    Print('------>', addr) whileTrue:Try:#Start catching exceptionsCMD=CONN.RECV (1024)#assigns the received content to the variable cmdRes=subprocess. Popen (Cmd.decode ('Utf-8'), shell=true,stdout=subprocess. Pipe,stderr=subprocess. PIPE)#set up a pipeline, write it as a shell, and judge if the correct command is placed in a pipe of standard output, or put it in the wrong output pipeDui=res.stdout.read ()#reads the contents of the standard output pipeline to DUICuo=res.stderr.read ()#reads the contents of the error output pipeline to CuOData_bat=len (DUI) +len (CuO)#Add the content length of DUI and CuO to Data_batConn.send (Struct.pack ('I', Data_bat))#send to client via module emulation headerConn.send (DUI)#Send the DUI to the clientConn.send (CuO)#send the CuO to the client        exceptException:#End Catch Exception             Break             #JumpConn.close ()#Break LinkDin.close ()#Turn off links that are based on network family transfers

#ClientImportSocketImportStructdin=Socket.socket (Socket.af_inet,socket. SOCK_STREAM) Ip_port=('127.0.0.1', 8080) Din.connect (ip_port) whileTrue:cmd=input ('Please enter a command:'). Strip ()if  notCmd:Continuedin.send (bytes (cmd,encoding='Utf-8')) Baotou=DIN.RECV (4) Data_size=struct.unpack ('I', Baotou) [0] Rec_size=0 Rec_data=b"'     whileRec_size <Data_size:data=DIN.RECV (1024) Rec_size+=len (data) Rec_data+=DataPrint(Rec_data.decode ('GBK') ) Din.close ()

The form of a cool header:

Service side
ImportstructImportSocketImportJSONImportSubprocessdin=Socket.socket (Socket.af_inet,socket. SOCK_STREAM) IP=('127.0.0.1', 8080) din.bind (IP) din.listen (5) whiletrue:conn,addr=din.accept ()Print('------->>>')     whileTrue:Try: Dr=CONN.RECV (1024) Res=subprocess. Popen (Dr.decode ('Utf-8'), shell=true,stdout=subprocess. Pipe,stderr=subprocess. PIPE) DUI=res.stdout.read () CuO=res.stderr.read () Data_lik=len (DUI) +Len (cuo) Head_dic={'Data_lik':d Ata_lik} head_json=json.dumps (head_dic) head_bytes=head_json.encode ('Utf-8') Head_len=Len (head_bytes)#Send Header LengthConn.send (Struct.pack ('I', Head_len)) #Send Headerconn.send (head_bytes)#Send real Dataconn.send (DUI) conn.send (CuO)exceptException: Breakconn.close () din.close ( )

Client
ImportstructImportSocketImportJsondin=Socket.socket (Socket.af_inet,socket. SOCK_STREAM) IP=('127.0.0.1', 8080) din.connect (IP) whileTrue:run=input ('Please enter a command:')    if  notRunContinuedin.send (bytes (run,encoding='Utf-8')) Data=DIN.RECV (4) Head_len=struct.unpack ('I', data) [0] Head_bytes=din.recv (head_len) Head_json=head_bytes.decode ('Utf-8') Head_dic=json.loads (Head_json) Data_lik=head_dic['Data_lik'] Recv_size=0 Recv_data=b"'     whileRecv_size <Data_lik:data=DIN.RECV (1024) Recv_size+=len (data) Recv_data+=DataPrint(Recv_data.decode ('GBK') ) Din.close ()

Python/socket programming of sticky bag