Using Python for Socket programming

This paper mainly refer to https://docs.python.org/3/howto/sockets.html.

This article only discusses streame (such as TCP) INET (such as IPV4) sockets.

Sockets is the most popular among the many cross-process communication modes. For any given platform, there may be other faster ways to communicate across processes, but for cross-platform communication, sockets should be the only one.

Create socket Client Socket

In layman's words, when you click on a link, your browser will do the following things:

# create an INET, STREAMing socket= socket.socket(socket.AF_INET, socket.SOCK_STREAM)# 连接到服务器，如果 URL 中没有指明端口，那么端口为默认的 80s.connect(("www.python.org"80))

After the connection is established, the socket can be used s to send the request. sthe reply is then read and then destroyed. In a request-receive process (or a series of small sequential processes), client sockets is typically used only once.

Service-side sockets

For Web servers:

# create an INET, STREAMing socket= socket.socket(socket.AF_INET, socket.SOCK_STREAM)# bind the socket to a public host, and a well-known port80))# become a server socketserversocket.listen(5)

socket.gethostname()The address can be seen externally.

listenNotifies the socket library that there can be up to 5 connection requests in the listening queue, and requests that are rejected after the queue is full.

Main loop

whileTrue:    # accept connections from outside    = serversocket.accept()    # now do something with the clientsocket    # in this case, we'll pretend this is a threaded server    = client_thread(clientsocket)    ct.run()

Accept will block until there is no connection.

The main loop typically has three ways of working:

• Assign a thread to handleclientsocket
• Create a process to handleclientsocket
• Refactor to use non-blocking sockets and use select sockets and multiplexing on our servers clientsocket (multiplex)

The above code is what the server socket does. It does not send or receive any data. It's just production clientsocket . Each clientsocket is created to respond connect() to a "client" sockets (such as a browser).

After the server socket is created clientsocket , it returns to listen for more connections. The two client sockets are talking freely-using a dynamically allocated port that will be reclaimed after the conversation ends.

Using the Socket

As a designer, you have to decide the communication rules between client sockets.

sendand recv operate network buffers, they do not necessarily handle all the bytes that you pass to them, because they focus on dealing with network buffers. When the network buffers is send or recv , they return the number of bytes they have processed. It is your responsibility to call them to ensure that all data has been processed .

When recv returning b"" or send returning 0 means that the other side has closed (or is shutting down) the connection. If it is recv, then you will not receive any more data from this connection, but you may be able to successfully send the data, as discussed below. If it is send, you can no longer send any data to the socket.

Like the HTTP protocol, only one socket is used in a single conversation. The client socket sends the request, reads the reply, and then the client socket is abandoned. So the client can find that the conversation is over by accepting a reply of 0 bytes.

If you are going to reuse your socket for future transmissions, you need to know that there is no end-of-transfer (EOT) sign in the socket.

To summarize: If send or recv 0 bytes, then this connection has been closed. If a connection is not closed, you may be waiting forever recv , because the socket will not tell you that there are no more messages now.

So the information

• Must be a fixed-length
• Or the boundaries are delineated.
• Or how long is the information?
• or close the connection to end it.

It's entirely up to you to choose which method to use.

Information length refers send to recv the length of the information. For example, send bytes, then the length of the information that STR converts to bytes, rather than the length of the send information represented by Str.

The simplest method is a fixed-length message:

classMysocket:"" " Demonstration class only-coded for clarity, not efficiency    """    def __init__( Self, sock=None):ifSock is None: Self. Sock=Socket.socket (socket.af_inet, socket. SOCK_STREAM)Else: Self. Sock=Sockdef Connect( Self, host, Port): Self. Sock.Connect(host, Port)defMysend ( Self, msg): Totalsent= 0         whileTotalsent<Msglen:sent=  Self. Sock.send (msg[totalsent:])ifSent== 0:Raise RuntimeError("Socket connection Broken") totalsent=Totalsent+SentdefMyreceive ( Self): Chunks=[] Bytes_recd= 0         whileBytes_recd<Msglen:chunk=  Self. SOCK.RECV (min(Msglen-BYTES_RECD,2048))ifChunk==B'':Raise RuntimeError("Socket connection Broken") Chunks.append (Chunk) BYTES_RECD=Bytes_recd+ Len(Chunk)returnB''. Join (chunks)

The length of the selection is the maximum length of information to be sent, if the information length is insufficient, then according to the Convention to supplement the information until the length, the agreed character is up to you.

The code above is to ensure that the code sent and received is not less than the defined length.

At the time of sending, since the length of the send is not fixed, each time you want to send from the previous information sent.

When receiving, specify exactly the length of the message you want to receive. If the specified length is less than the actual length, then the information is incomplete and, conversely, it waits for the message to be sent. And because the most receive 2048 bytes, so to min(MSGLEN - bytes_recd, 2048) .

Python len() can calculate the length of the message it contains \0 , while the length of the message contained in the calculation cannot be used in the C language strlen \0 .

Use information length as prefix

Assuming that 5 characters are used as information prefixes to represent information lengths, then you may not be able to get all 5 characters in one recv , which can occur in cases of high network load. So you can call two times recv --the first decision length, the second gets the rest of the information.

Binary data

You can use the socket to send binary data. The main problem is that not all machines are using the same binary data format. For example, the Motorola chip uses two hexadecimal bytes to 00 01 represent a 16-binary integer 1 . However, Intel and DEC are byte-reversed-use 01 00 representations 1 .

on today's 32-bit machines, binary data with ASCII representation is usually smaller than the binary representation of the data. Because in many cases, the data contains 0 or 1. The string "0" is 2 bytes and the binary is 4. Therefore, it is not suitable for fixed-length information. So you need to choose the right strategy for passing information when you want to be able to pass strings and binary data using the socket.

Python Struct manipulates binary data

>>>fromimport*>>> pack('hhl'123)b'\x00\x01\x00\x02\x00\x00\x00\x03'>>> unpack('hhl', b'\x00\x01\x00\x02\x00\x00\x00\x03')(123)>>> calcsize('hhl')8

Above reference https://docs.python.org/3/library/struct.html

Disconnect Connection

Strictly speaking, close you should call before the socket shutdown . According to shutdown the parameters passed, it can be said that "I will not read from this socket or write data to this socket." Most sockets libraries, because programmers always forget shutdown to call, so the close equivalent shutdown(); close() . Therefore, in most cases, it is not necessary to display the call shutdown .

HTTP-like transmissions can be used efficiently shutdown . The client is called after the request is sent shutdown(1) . This tells the server that the client has sent out, but still receives the message. The server can know that this is EOF (the end of the document) by receiving 0 bytes.

In Python, if the socket is garbage collected, it will be executed automatically when needed close() . But relying on this is a very bad habit. If your socket is not executed before it disappears close , the socket on the other side will hang.

When should I clear Sockets?

The worst thing you can do with blocking sockets is to hang it on the other side (and not call it close ). Your socket will always hang. TCP is a reliable protocol, so it waits a long time before closing the connection. If you use a thread, the entire thread is hung up. You can't do anything about it. As long as you do not do some stupid things, such as blocking the lock when the operation, the thread will not consume a lot of resources. Do not attempt to kill this thread--the thread is more efficient than the process in part because the thread avoids automatic resource recycling. In other words, if you kill this thread, your entire process is likely to hang up.

Non-blocking Sockets

In Python, you use to socket.setblocking(0) make the socket non-blocking. The C language is more complex, but the idea is the same. You're going to do this after creating the socket.

The main difference of the mechanism is that, send recv connect accpet without doing anything, it will return. You have a lot of choices. For example, check the return code and error code, but this will make your application larger, prone to bugs and consume a lot of CPU.

Use select .

In the C language, the use select is complex. It's easy in Python, but it's close enough to the concept in C, and if you understand Python select , then you understand that C doesn't have a big problem:

=\               select.select(                  potential_readers,                  potential_writers,                  potential_errs,                  timeout)

Pass to select three parameters:

• List of all sockets you want to read
• List of all sockets you want to write
• List of all sockets you want to check for errors

You should be aware that a socket can appear in multiple lists. The call select is blocked, but you can give it a time-out.

Returns a list of 3. Contains readable, writable, and erroneous sockets.

If a socket is in a readable list, then the call recv must return something to it. The same is true for writable. You can then use the read-write method used in the blocking operation above.

• Create a server socket and set it to non-blocking
• Put the server socket into potential_readers
• Call Select with Potential_readers as parameter
• Check Ready_to_read, if it is a server socket, call accept on it, get the client socket, set the client socket to non-blocking
• Add client sockets to Potential_writers and Potential_readers
• Call Select with Potential_writers and potential_readers as arguments
• Read the information from the Potential_readers client socket and store it in the msg[client socket]
• Get the client socket from potential_writers and send it msg[client socket] if msg[client socket exists

The above reference https://pymotw.com/2/select/.

Portability warning: In Unix, it select is valid for both sockets and files. In Windows, it is select only valid for sockets. And in C, many of the advanced features of sockets are different in Windows. It is therefore recommended that you use thread in Windows.

Specific code

See https://github.com/Jay54520/python_socket/.

Reference

1. https://gist.github.com/owainlewis/3217710
2. Https://docs.python.org/3/library/struct.html
3. https://pymotw.com/2/select/

Using Python for Socket programming