10.python Network programming (solving sticky pack problems Part 2)

I. When will the phenomenon of sticky-pack occur.

Sticky packets can only occur if the TCP protocol is used! The UDP protocol will never be!

650) this.width=650; "Src=" http://images2015.cnblogs.com/blog/1036857/201612/ 1036857-20161210123107304-1582863963.png "/>

The sending side can 1k1k the data sent out, the receiving side, can 2k2k to receive the data, may receive 3k at a time, also may receive 6k 1 times.

The TCP protocol is a stream-oriented protocol, which is also the cause of sticky packet problems. And UDP is a message-oriented protocol, each UDP segment is a message, the application must be in the message to extract data, not one time to extract arbitrary bytes of data, which is very different from TCP. How do you define a message? Can think of the other one-time Write/send data for a message, it is necessary to understand that when the other side send a message, regardless of the underlying how fragmented shards, the TCP protocol layer will constitute the entire message of the data segment is completed before rendering in the kernel buffer.

For example, the TCP-based socket client to the server to upload files, the content of the file is sent in accordance with a paragraph of the stream of bytes sent, in the receiver saw, do not know where the file's byte stream from where to start, where the end!!!

Actually, it's a little more simple.

The so-called sticky packet problem is mainly because the receiver does not know the boundary between the message, do not know how many bytes of data extracted at once.

！！ In addition, the packet caused by the sender is caused by the TCP protocol itself, TCP to improve transmission efficiency, the sender often to collect enough data before sending a TCP segment. If you need to send data for a few times in a row, TCP will usually send the data to a TCP segment based on the optimization algorithm, so that the receiving party receives the sticky packet data.!

TCP (Transport Control Protocol, transmission Protocol) is connection-oriented, stream-oriented and provides high reliability services. Both ends of the transceiver (client and server side) have one by one pairs of sockets, so the sending side in order to send multiple packets to the receiver, more efficient to the other side, the use of the optimization method (Nagle algorithm), the multiple interval small and small data volume data, combined into a large block of data, and then to the packet. In this way, the receiving end, it is difficult to distinguish out, must provide a scientific unpacking mechanism. That is, stream-oriented communication is a non-message-protected boundary.

UDP (User Datagram Protocol, Subscriber Datagram Protocol) is non-connected, message-oriented, providing efficient service. The Block merging optimization algorithm is not used, because UDP supports a one-to-many pattern, so the receiver Skbuff (socket buffer) uses a chain structure to record each incoming UDP packet, in each UDP packet there is a message header (message source address, port and other information), so for the receiving end , it is easy to distinguish between the processing. That is, message-oriented communication is a message-protected boundary.

UDP Recvfrom is blocked, a recvfrom (x) must be to a sendinto (y), after the X-byte data is completed, if the y>x data is lost, which means that UDP is not sticky packets, but will lose data, unreliable

TCP protocol data is not lost, no packets are received, the next time it is received, it continues to receive the last time, and the buffer content is always cleared when the ACK is received by the client. The data is reliable, but it will stick to the package.

Second, in detail in the TCP communication under the two cases of adhesive packet:

The sending side need to wait for the buffer full to send out, resulting in sticky packets (send data time interval is very short, the data is very small, to join together to produce sticky packets).

In another case, when the length of the transmit buffer is greater than the MTU of the NIC, TCP will split the sent data into several data packets sent out.

Supplemental Explanation: Why is TCP connection reliable?

TCP at the time of data transmission, the sending side first sends the data to its own cache, then the Protocol control to send the data in the cache to the peer, returns a ack=1 to the end, the sending side cleans up the data in the cache, returns the ack=0 to the end, then sends the data again, so TCP is reliable

While UDP sends data, the peer does not return confirmation information, so it is unreliable.

Third, how to solve the problem of sticky bag?

We can solve the sticky packet problem by customizing the header.

Add a custom fixed-length header to the byte stream, the header contains the byte stream length, and then send to the peer at a time, and the peer receives the fixed-length header from the cache before fetching the real data.

When we say the custom header, we have to mention a module called struct, the main function of which is to turn a type, such as a number, into a fixed-length bytes. (Why use this module?) This module is used to fix the length of the header, and when the header length is fixed, it is possible to distinguish between the header and the real data in the data stream. ）

For detailed usage of the struct module, please refer to:

Http://www.cnblogs.com/coser/archive/2011/12/17/2291160.html

Make a header about the process demo:

Import Json,struct

#假设通过客户端上传1T: 1073741824000 of files A.txt

#为避免粘包, you must customize the header

header={' file_size ': 1073741824000, ' file_name ': '/a/b/c/d/e/a.txt ', ' MD5 ': ' 8f6fbf8347faa4924a76856701edb0f3 '} # 1T data, file path and MD5 values

#为了该报头能传送, need to serialize and convert to bytes

Head_bytes=bytes (Json.dumps (header), encoding= ' Utf-8 ') #序列化并转成bytes, for transmission

#为了让客户端知道报头的长度, use struck to convert the number of the header length to a fixed length: 4 bytes

Head_len_bytes=struct.pack (' I ', Len (head_bytes)) #这4个字节里只包含了一个数字, the number is the length of the header

#客户端开始发送

Conn.send (head_len_bytes) #先发报头的长度, 4 x bytes

Conn.send (head_bytes) #再发报头的字节格式

Conn.sendall (file contents) #然后发真实内容的字节格式

#服务端开始接收

HEAD_LEN_BYTES=S.RECV (4) #先收报头4个bytes, gets the byte format of the header length

X=struct.unpack (' i ', head_len_bytes) [0] #提取报头的长度

HEAD_BYTES=S.RECV (x) #按照报头长度x, receive the bytes format of the header

Header=json.loads (Json.dumps (header)) #提取报头

#最后根据报头的内容提取真实的数据, like

Real_data_len=s.recv (header[' file_size ')

S.RECV (Real_data_len)

The solution of sticky bag:

We can make a dictionary of the header, the dictionary contains the details of the real data to be sent, then JSON serialization, and then use struck to package the serialized length of the data into 4 bytes (4 of them enough)

When sending:

The header length is first sent.

Re-encode the header content and send it.

Finally send the real content.

When receiving:

First, the length of the header is taken out by the struct.

The header content is charged according to the length taken, then decoded and deserialized.

Extract the details of the data to be fetched from the deserialized results and then fetch the actual data content.

The following is an example of the python2.7 version as a demonstration, a file upload function, the client can send the file to the server.

Service-Side code:

#!/usr/bin/python2.7

#-*-Coding:utf-8-*-

Import socket

Import JSON

Import struct

Import OS

Class Mytcpserver (object):

sockt_family = Socket.af_inet

Socket_type = socket. Sock_stream

ALLOW_REUSE_ADDR = False

Max_trans_size = 8192

coding = ' Utf-8 '

Tcp_queue_size = 5

Upload_dir_name = ' Upload_file '

def __init__ (self,server_address,bind_and_active = True):

Self.server_address = server_address

Self.socket = Socket.socket (Self.sockt_family,self.socket_type)

If bind_and_active:

Try

Self.server_bind ()

Self.server_active ()

Except Exception:

Self.server_close ()

Raise

def server_bind (self):

If self.allow_reuse_addr:

Self.socket.setsockopt (socket. Sol_socket,socket. so_reuseaddr,1)

Self.socket.bind (self.server_address)

Self.server_address = Self.socket.getsockname () #增加一个没啥卵用的功能 to get the address that was just bound.

def server_active (self):

Self.socket.listen (Self.tcp_queue_size)

def server_close (self):

Self.socket.close ()

def get_request (self):

Return Self.socket.accept ()

def close_request (self):

Return Self.socket.close ()

def run (self):

While True:

Self.conn,self.client_ip = Self.get_request ()

Print Self.client_ip

While True:

Try

Head_struct = SELF.CONN.RECV (4) #用来接收报头

If not head_struct:

Break

Head_len = Struct.unpack (' i ', head_struct) [0] #解析出来的报头长度

Head_json = Self.conn.recv (Head_len). Decode (self.coding) #接收报头内容

Head_dict = Json.loads (Head_json) #将json报头转换为字典

Print Head_dict

cmd = head_dict[' cmd ')

If Hasattr (self,cmd):

Func = GetAttr (self,cmd)

Func (Head_dict)

Except Exception:

Break

def put (Self,args):

Upload_file_path = Os.path.normpath (Os.path.join (self.upload_dir_name,args[' file_name '))

File_size = args["File_size"]

Recv_size = 0

Print "------>" +upload_file_path

With open (Upload_file_path, ' wb+ ') as F:

While Recv_size < file_size:

Recv_data = Self.conn.recv (self.max_trans_size)

F.write (Recv_data)

Recv_size+=len (Recv_data)

Print "recvsize:%s filesize:%s"% (recv_size,file_size)

tcp_server_1 = Mytcpserver ((' 10.39.0.100 ', 8888))

Tcp_server_1.run ()

Client code:

#!/usr/bin/python2.7

#-*-Coding:utf-8-*-

Import socket

Import JSON

Import struct

Import OS

Class Mytcpclient (object):

sockt_family = Socket.af_inet

Socket_type = socket. Sock_stream

ALLOW_REUSE_ADDR = False

Max_trans_size = 8192

coding = ' Utf-8 '

Tcp_queue_size = 5

def __init__ (self,server_address,connect = True):

Self.server_address = server_address

Self.socket = Socket.socket (Self.sockt_family,self.socket_type)

If connect:

Try

Self.client_connect ()

Except

Self.client_close ()

Raise

def client_connect (self):

Self.socket.connect (self.server_address)

def client_close (self):

Self.socket.close ()

def run (self):

While True:

User_input = Raw_input (">>"). Strip ()

If not user_input:

Continue

cmd = user_input.split () [0]

If Hasattr (self,cmd):

Func = GetAttr (self,cmd)

Func (User_input.split ())

def put (Self,args):

cmd = args[0]

filename = args[1]

If not os.path.isfile (filename):

Print "File is not exists!!"

Return None

Else

FileSize = os.path.getsize (filename)

Head_dic = {' cmd ': cmd, ' file_name ': os.path.basename (filename), ' file_size ': filesize}

Print Head_dic

Head_json = Json.dumps (head_dic). Encode ("Utf-8")

Head_struct_size = Struct.pack (' i ', Len (Head_json)). Encode ("Utf-8") #准备发送的报头大小

Self.socket.send (head_struct_size) #发送报头大小

Self.socket.send (Head_json) #发送报头

Send_size = 0

with open (filename, ' RB ') as F:

For line in F:

Self.socket.send (line)

Send_size+=len (line)

Print Send_size

Else

print "File Upload success!!"

Client = mytcpclient (' 10.39.0.100 ', 8888)

Client.run ()

This article is from the "Rebirth" blog, make sure to keep this source http://suhaozhi.blog.51cto.com/7272298/1923431

10.python Network programming (solving sticky pack problems Part 2)