Socket -- accept big data, socket -- accept data

Socket -- accept big data, socket -- accept data
I. Simple ssh Functions

　　1.1 implement functions

In the previous blog, we have implemented a simple small program similar to the Linux Server ssh function. You can enter a system command to return the command running result. Today we will start with this, let's see how the socket can accept a large amount of data.

Server:

#-*-Coding: UTF-8-*-import osimport socketserver = socket. socket (socket. AF_INET, socket. SOCK_STREAM) # TCP/IP protocol, TCP. If not entered, it is the default server. bind ('localhost', 9999) server. listen () while True: # multiple clients can be accepted. conn, addr = server. accept () while True: data = conn. recv (1024) if not data: # prevent print ('client has lost... ') break print ('COMMAND execution:', data. decode () has _res = OS. popen (data. decode ()). read () if len (FIG) = 0: print ('COMMAND not found') else: # send data conn. send ('{}'. format (performance_res ). encode ('utf-8') print ('sending completed ')

View Code

Client:

#-*-Coding: UTF-8-*-import socketclient = socket. socket () client. connect ("localhost", 9999) while True: cmd = input ('>> :'). strip () # determine whether to send NULL data. if yes, resend if len (cmd) = 0: continue else: client. send (cmd. encode ('utf-8') receive_data = client. recv (1024) # The accepted data is of the bytes Type print (receive_data.decode ('utf-8', 'ignore') # If ignore is not added, an error is reported in windows.

View Code

Running result:

　　

We run both commands normally, and it looks good. It can achieve command input and result output.

　　1.2 Problems

　　Next, let's look at the problem? Since a large amount of data is received, a large amount of data is returned.

　　

It seems that there is no problem, but we will continue to execute the command

　　

Apparently an error occurred. The command dir returned part of the previous ipconfig/all instead of the information in the directory. why?

Ii. Socket buffer 2.1 What is socket buffer

After each socket is created, two buffers are allocated: the input buffer and the output buffer.

Write ()/send () does not transmit data to the network immediately, but writes the data to the buffer first, and then sends the data from the buffer to the target machine by the TCP protocol. Once data is written to the buffer zone, the function can return a successful result, regardless of whether they have reached the target machine or when they are sent to the network, which is the responsibility of the TCP protocol.

The TCP protocol is independent of the write ()/send () function. data may be sent to the network as soon as it is written to the buffer, or it may be accumulated in the buffer, data written multiple times is sent to the network at a time, depending on the network conditions at that time, whether the current thread is idle, and many other factors that cannot be controlled by programmers.

　　

2.2 send and receive data in the buffer zone

For TCP Sockets (by default), when data is sent using write ()/send:
1) First, check the buffer zone. If the available space length of the buffer zone is smaller than the data to be sent, write ()/send () will be blocked (paused ), the write ()/send () function is awakened to continue writing data until the data in the buffer zone is sent to the target machine.

2) If the TCP protocol is sending data to the network, the output buffer will be locked and cannot be written. write ()/send () will also be blocked, write ()/send () will not be awakened until the buffer is unlocked after the data is sent.

3) if the data to be written is greater than the maximum length of the buffer, the data will be written in batches.

4) data can be returned only when all data is written to the buffer zone write ()/send.

When read ()/recv () is used to read data:
1) First, check the buffer zone. If there is data in the buffer zone, read it. Otherwise, the function will be blocked until data arrives on the network.

2) if the length of the data to be read is less than the length of the data in the buffer zone, you cannot read all the data in the buffer zone at a time. The remaining data will be accumulated until read ()/recv () exists () function read again.

3) The read ()/recv () function will not return until the data is read, otherwise it will be blocked.

This is the blocking mode of TCP socket. Blocking means that the previous action is not completed and the next action is paused until the previous action is completed to maintain synchronization.

 

Iii. How to accept massive data 3.1 Solutions

Because of the existence of the buffer zone, we cannot completely transfer a large amount of data at once! In fact, it has little to do with the amount of data received and sent, that is, send (1024)/recv (1024. We didn't set these two values very large enough to solve the problem. Because each socket receives and sends a maximum amount of data, after all, the network bandwidth is limited, the maximum amount of data sent is the maximum amount of data that can be cached in the buffer zone. The maximum value of this buffer zone varies with different systems. However, the official recommendation is that it should not exceed 8 KB, that is 8192.

So we can only think about it from another angle. That is to say, I want to judge whether the data returned by a command is completely transmitted after execution. If not, the data will continue to be transmitted, until the upload is complete.

3.2 Solution

The simple method is to compare the size of the received and transmitted data. If the received data volume is equal to the sent data volume, isn't the transfer finished?

So, the Code is as follows:

Server:

#-*-Coding: UTF-8-*-import osimport socketserver = socket. socket (socket. AF_INET, socket. SOCK_STREAM) # TCP/IP protocol, TCP. If not entered, it is the default server. bind ('localhost', 9999) server. listen () while True: # multiple clients can be accepted. conn, addr = server. accept () while True: data = conn. recv (1024) if not data: # prevent print ('client has lost... ') break print ('COMMAND execution:', data. decode () has _res = OS. popen (data. decode ()). r Ead () if len (distinct _res) = 0: print ('COMMAND not found ') else: # The server first sends the data size to the client for comparison with pai_res_length = len (distinct _res) # int type # print (len ('{}'. format (performance_res ). encode ('utf-8') # If you do not encode the data first, the Chinese length statistics do not match. The data transmitted using the method described below is shorter than the actual conn. send (str (len ('{}'. format (performance_res ). encode ('utf-8 '))). encode ('utf-8') # conn. send (str (pai_res_length ). encode ('utf-8') # send data conn. send ('{}'. format (performance_res ). encode ('utf-8') print ('sending completed ')

View Code

Client:

#-*-Coding: UTF-8-*-import socketclient = socket. socket () client. connect ("localhost", 9999) while True: cmd = input ('>> :'). strip () # determine whether to send NULL data. if yes, resend if len (cmd) = 0: continue else: client. send (cmd. encode ('utf-8') data_size = client. recv (1024) # size of data sent by the receiving server print (data_size) data_length = int (data_size.decode () print ('size of returned data: ', data_length) # define the size of received data to 0 received_length = 0 # define the size of received data to 0 rece Ived_data = B ''while received_length <data_length: r_data = client. recv (1024) # The accepted data is bytes-type received_length + = len (r_data) received_data + = r_data else: print ('size of received data: ', received_length) print (received_data.decode ('utf-8', 'ignore! ')

View Code

Result:

Look, the problem is solved

　　

The middle part is too long.

　　

 

Iv. Remaining Issues

　　4.1 Data Length inconsistency

As mentioned earlier, when a string contains Chinese characters, the len () function may have different lengths, as shown in the following example:

# Section of ipconfig/all data = 'ethernet Adapter VMware Network Adapter VMnet1: '# length = len (data) print (length) calculated in UTF-8 format) # First transcode in the calculated length utf_length = len (data. encode ('utf-8') print (utf_length) # output 3749

Note: The length is 2 Characters longer than the non-encoding length,