Python: non-blocking or asynchronous programming

For example, for a chat room, because multiple connections need to be processed at the same time, it is clear that blocking or synchronization methods are not suitable, this is like opening only one window for a ticket and queuing for multiple users. So how can we solve this problem? There are three main methods: forking, threading, and asynchronous I/O.

The forking and threading methods are very simple and can be implemented by using the Min-in class of the socketserver service class. Forking is only applicable to Unix-like platforms. Threading requires attention to Memory Sharing.
Asynchronous I/O is a little difficult if the underlying method is implemented. To be simple, we can consider using the framework in the standard library or twisted (twisted is a very powerful framework for Asynchronous Network Programming ).

1. Use scoketserver to implement forking and threading

The following two examples are used to create a forking server and a threading server respectively.

Forking Server:

From socketserver import tcpserver, forkingmixin, streamrequesthandler

Class server (forkingmixin, tcpserver): Pass

Class handler (streamrequesthandler ):
Def handle (Self ):
ADDR = self. Request. getpeername ()
Print 'got connection from', ADDR
Self. wfile. Write ('Thank you for connection ')

Server = server ('', 1234), Handler)
Server. serve_forever ()

Threading Server:

From socketserver import tcpserver, threadingmixin, streamrequesthandler

Class server (threadingmixin, tcpserver): Pass

Class handler (streamrequesthandler ):
Def handle (Self ):
ADDR = self. Request. getpeername ()
Print 'got connection from', ADDR
Self. wfile. Write ('Thank you for connection ')

Server = server ('', 1234), Handler)
Server. serve_forever ()

Ii. Use select to Implement Asynchronous I/O

The so-called asynchronous I/O, for example, is that if a large group of people want you to listen to him, then you will give them one minute to say, if you haven't finished it, continue later. That is, the method of a time slice.

To implement asynchronous I/O, we can use the framework asyncore/asynchat or twisted, which are based on the select function or poll function (poll is only applicable to Unix-like systems. Both the select and poll functions come from the select module.

The select function requires three sequence values as parameters and an optional timeout value in seconds. In the sequence is the integer representing the file descriptor, which is the connection we are waiting. These three sequences are about input, output, and exception conditions. If the timeout value is not given, select will be in the blocking state (that is, waiting) until there is a file descriptor preparation action. If the timeout value is given, select only blocks the specified time. If the timeout value is 0, it will not be blocked. The values returned by the SELECT statement are a triple consisting of three sequences, which represent the activity subsets of the corresponding parameters. For example, the first sequence returns a sequence consisting of input file descriptors for reading.

The sequence can contain file objects (not suitable for Windows) or sockets. The following example creates a server that uses select to serve several connections (note: the socket of the server itself is also provided to select, so that it can send a signal notification when new connections are ready for acceptance ). This server simply prints the data accepted from the client. You can use Telnet (or write a simple socket-based client) to connect and test it.

Select Server

Import socket, select

S = socket. socket ()
Host = socket. gethostname ()
Port = 1234
S. BIND (host, Port ))

S. Listen (5)
Inputs = [s]
While true:
RS, WS, ES = select. Select (inputs, [], [])
For R in RS:
If R is s:
C, ADDR = S. Accept ()
Print 'got connection from', ADDR
Inputs. append (c)
Else:
Try:
Data = R. Recv (1024)
Disconnected = Not data
Failed t socket. Error:
Disconnected = true

If disconnected:
Print R. getpeername (), 'disconnected'
Inputs. Remove (r)
Else:
Print data

Iii. Twisted

Twisted is an event-driven network framework for python. It was initially developed for online games, but is now applied to various types of network software. With twisted, you can implement the event processor, which is usually similar to using the GUI toolkit (TK, GTK, QT, wxWidgets ). In this section, I will introduce some basic concepts and demonstrate how to use twisted for some relatively simple network programming. Twisted is a very powerful framework and provides a lot of support, such as web servers and clients, SSH2, SMTP, POP3, IMAP4, aim, ICQ, IRC, MSN, jabber, nntp, DNS and so on.

The socket-based servers we wrote earlier all have a display event loop: finding new connections and new data; The socketserver-based servers have an implicit loop: find a connection and create a processor for the connection. However, the read data displayed when the processor is still.

Twisted uses more event-based methods. To write a basic server, you need to implement an event processor to handle such situations as a new client connection, new data arrival, and client connection interruption. In twisted, your event processor is defined in a protocol; you also need a factory. When a new connection arrives, it can construct this Protocol object, however, if you only want to create a custom protocol class instance, you can use the factory from twisted, factory class in the module twisted. internet. protocol. When you write your protocol, use
The protocol in the twisted. Internet. protocol module serves as your parent class. When you get a connection, the event processor connectionmade is called. When you lose a connection, connectionlost is called. Use the processor datareceived to receive data from the client. However, you cannot use the event processing policy to send data to the client. To send data to the client, you can use self. Transport, which has a write method. It also has a client attribute that contains the client address (host name and port ).

The following example shows a twisted server. The factory is instantiated and its Protocol attribute is set so that it knows which Protocol to use to communicate with the client (this is called your custom protocol ). Then you use factory to start listening to the specified port. factory processes the connection through the instantiated protocol object. The listener uses the listentcp function in the reactor module. Finally, you can call the run function in the reactor module to start the server.

From twisted. Internet import Reactor
From twisted. Internet. Protocol import protocol, factory

# Define your protocol class
Class simplelogger (Protocol ):

Def connectionmade (Self ):
Print 'got connection from', self. Transport. Client
Def connectionlost (self, reason ):
Print self. Transport. Client, 'disconnected'
Def datareceived (self, data ):
Print data

# Instantiate Factory

Factory = Factory ()

# Set the factory protocol attribute so that it knows which Protocol to use to communicate with the client (this is called your custom
# Protocol)

Factory. Protocol = simplelogger

# Listening to the specified port

Reactor. listentcp (1234, factory)

# Start to run the main program
Reactor. Run ()

It is easy to write a custom protocol for your processing purpose. The module twisted. Protocols. Basic contains several useful existing protocols. The linereceiver executes datareceived and receives a complete line call event processor linereceived. If you want to use linereceived in addition to linereceived, you can use the rawdatareceived event processor defined by linereceiver. The following is an example of a server using linereceiver:

From twisted. Internet import Reactor
From twisted. Internet. Protocol import Factory
From twisted. Protocols. Basic import linereceiver

Class simplelogger (linereceiver ):

Def connectionmade (Self ):
Print 'got connection from', self. Transport. Client
Def connectionlost (self, reason ):
Print self. Transport. Client, 'disconnected'
Def linereceived (self, line ):
Print line

Factory = Factory ()
Factory. Protocol = simplelogger
Reactor. listentcp (1234, factory)
Reactor. Run ()