Python 3: select-based event driver, pythonselect

Python 3: select-based event driver, pythonselect

The basic principle of this function is that the select/epoll function will continuously poll all the sockets in charge. When a socket has data, it will notify the user process. Its Process

When a user process calls select, the entire process will be blocked. At the same time,The kernel will "Monitor" all the sockets that the select is responsible for. When the data in any socket is ready, the select will return. At this time, the user process then calls the read operation to copy data from the kernel to the user process. This graph is not much different from the blocking IO graph. In fact, it is worse. Because two system calls (select and recvfrom) need to be used here, while blocking IO only calls one system call (recvfrom ). However, the advantage of using select is that it can process multiple connections at the same time. (More. Therefore, if the number of connections to be processed is not very high, the web server using select/epoll may not have better performance than the web server using multi-threading + blocking IO, and may have a greater latency. The advantage of select/epoll is not that it can process a single connection faster, but that it can process more connections .)
In the I/O multiplexing Model, in practice, each socket is generally set to non-blocking. However, as shown in, the entire user's process is always blocked. However, process is block by the select function, rather than block by socket IO.

Therefore, the IO multiplexing Model is characterized by blocking in both stages, but waiting for data to be blocked on the select, and copying data to the recfrom.

In the multiplexing model, each socket is generally set to non-blocking. However, as shown in, the entire user's process is always blocked.However, process is block by the select function, rather than block by socket IO. Therefore, select () is similar to non-blocking IO.

Most Unix/Linux systems support the select function, which is used to detect the status changes of multiple file handles. The following is a prototype of the select Interface:
FD_ZERO (int fd, fd_set * fds)
FD_SET (int fd, fd_set * fds)
FD_ISSET (int fd, fd_set * fds)
FD_CLR (int fd, fd_set * fds)
Int select (int nfds, fd_set * readfds, fd_set * writefds, fd_set * limit TFDs,
Struct timeval * timeout)
Here, the fd_set type can be simply understood as a queue that marks the handle by bit. For example, to mark a handle with a value of 16 in an fd_set, the 16th bits of the fd_set are marked as 1. You can use a macro such as FD_SET and FD_ISSET to implement specific positions and verification. In the select () function, readfds, writefds, and limit TFDs are both input and output parameters. If the input readfds is marked with a 16th handle, select () checks whether the 16th handle is readable. After the select () Statement is returned, you can check whether readfds has a handle number 16 to determine whether the "readable" event has occurred. In addition, you can set the timeout time.
Next, we will re-simulate the model for receiving data from multiple clients in the previous example.

Figure 7 receiving data model using select ()

The model only describes the process of using the select () interface to receive data from multiple clients at the same time () the interface can detect the read, write, and error statuses of multiple handles at the same time, so it is easy to build a server system that provides independent Q & A services for multiple clients. For example.

Figure 8 event-driven server model using the select () Interface

It should be noted that a connect () operation on the client will trigger a "readable event" on the server, so select () can also detect the connect () behavior from the client..
In the above model, the most important thing is how to dynamically maintain the three select () parameters readfds, writefds, and limit TFDs. As the input parameter, readfds should mark all the "readable event" handles to be detected, which will always include the "parent" handle of the test connect (). At the same time, writefds and writable TFDs should mark all the "writable events" and "error events" handles to be detected (marked using FD_SET ).
As an output parameter, the handles of all events captured by select () are saved in readfds, writefds, and limit TFDs. The programmer needs to check all the flag bits (using FD_ISSET () to determine which handles have an event.
The above model mainly simulates the "one-question-one-answer" service process. Therefore, if select () discovers that a handle captures a "readable event", the server program should perform recv () operations in a timely manner, prepare the data to be sent based on the received data, add the corresponding handle value to writefds, and prepare the select () test for the next "writable event. Similarly, if select () finds that a handle captures a "writable event", the program should promptly perform the send () operation and prepare for the next "readable event" detection. Describes an execution cycle in the preceding model.

Figure 9 Execution cycle of a multiplexing Model

This model features that each execution cycle detects one or more events, and a specific event triggers a specific response. We can classify this model as"Event-driven model".
Compared with other models, the select () event-driven model only uses a single thread (process) for execution, consumes less resources, does not consume too much CPU, and can provide services for multiple clients. If you try to build a simple event-driven server program, this model has some reference value.
However, this model still has many problems.First, the select () interface is not the best choice to implement "event-driven. When the handle value to be tested is large, the select () interface itself consumes a lot of time to poll each handle.Many operating systems provide more efficient interfaces, such as epoll in linux, kqueue in BSD, And/dev/poll in Solaris ,.... If you want to implement more efficient server programs, Apis like epoll are more recommended. Unfortunately, the epoll interfaces provided by different operating systems are quite different. Therefore, it is difficult to use the epoll-like interfaces to implement servers with better cross-platform capabilities.
Secondly, this model integrates event detection and Event Response. Once the execution body of the Event Response is large, it is disastrous for the entire model.As shown in the following example, the large execution body 1 will directly lead to a delay in the execution body responding to event 2, and greatly reduce the timeliness of event detection.

Figure 10 impact of a large execution body on the event-driven model using select ()

Fortunately, there are many efficient event-driven libraries that can block the above difficulties. Common event-driven libraries includeLibevent LibraryAs a replacement for libevent.Libev Library. These libraries will select the most appropriate event detection interface based on the characteristics of the operating system, and add signal and other technologies to support asynchronous response, this makes these libraries the best choice for building an event-driven model. The following chapter describes how to use the libev database to replace the select or epoll interface to implement an efficient and stable server model.

In fact, the Linux kernel has introduced IO operations that support asynchronous responses since 2.6, such as aio_read and aio_write. This is asynchronous IO.

 

In python, It is encapsulated and the return value is modified. Compared with the original return value in C (an integer, it determines whether the call is successful ), the call return value in python is a sequence of readable, writable, and abnormal States returned directly. The readable, writable, and abnormal sequence in C is directly written into the parameter, that is, the input and output parameters are the same. The encapsulation design such as python is still very good. I designed a crude select call based on the event mechanism:

Server:

Import selectimport socketimport queuefrom time import sleepclass TCPServer: def _ init _ (self, server, server_address, inputs, outputs, message_queues): # Create a TCP/IP self. server = server self. server. setblocking (False) # Bind the socket to the port self. server_address = server_address print ('starting up on % s port % s' % self. server_address) self. server. bind (self. server_address) # Listen for in Coming connections self. server. listen (5) # Sockets from which we recommend CT to read self. inputs = inputs # Sockets to which we recommend CT to write # process the message self to be sent. outputs = outputs # Outgoing message queues (socket: Queue) self. message_queues = message_queues def handler_recever (self, readable): # Handle inputs # cyclically checks whether a client is connected. When a client is connected, select triggers for s in readable: # determine whether the currently triggered object is a server object. When the triggered object is a server object, it indicates that there is a new The client is connected. # indicates that a new user is connected to if s is self. server: # A "readable" socket is ready to accept a connection, client_address = s. accept () self. client_address = client_address print ('Connection from', client_address) # this is connection not server connection. setblocking (0) # Add the client object to the listener list. When the client sends a message, select triggers self. inputs. append (connection) # Give the connection a queue for data we want to send # Creates a separate message queue for the connected client to save the message self sent by the client. message_queues [connection] = queue. queue () else: # An old user sends a message and processes it. # As the server receives the client connection request when the client is connected, it adds the client to the listener list (input_list ), the client will trigger message sending # so determine whether the client object is triggered data = s. recv (1024) # The client is not disconnected if data! = B '': # A readable client socket has data print ('stored ed" % s "from % s' % (data, s. getpeername () # Put the received message into the message queue of the corresponding socket Client self. message_queues [s]. put (data) # Add output channel for response # put the socket that requires the reply operation into the output list, And let select listen if s not in self. outputs: self. outputs. append (s) else: # the client is disconnected and the listener of the client is removed from the input list # Interpret empty result as closed connection print ('closing', s. getpeername () # obtain the socket information of the Client # Stop listening for input on the connection if s in self. outputs: self. outputs. remove (s) self. inputs. remove (s) s. close () # Remove message queue # Remove the message queue del self of the corresponding socket Client object. message_queues [s] return "got it" def handler_send (self, writable): # Handle outputs # If no client request exists and no client sends a message, start to process the sent message list. Do you want to send a message? # store the client that sent the message for s in writable: try: # if there is a message in the message queue, obtain the message message_queue = self from the message queue. message_queues.get (s) send_data = ''if message_queue is not None: send_data = message_queue.get_nowait () wait t queue. empty: # the client is disconnected from self. outputs. remove (s) else: # print "sending % s to % s" % (send_data, s. getpeername) # print "send something" if message_queue is not None: s. send (send_data) else: print ("client has closed") # del message_queues [s] # writable. remove (s) # print "Client % s disconnected" % (client_address) return "got it" def handler_exception (self, exceptional ): # Handle "exceptional conditions" # Handle exceptions for s in exceptional: print ('exception condition on', s. getpeername () # Stop listening for input on the connection self. inputs. remove (s) if s in self. outputs: self. outputs. remove (s) s. close () # Remove message queue del self. message_queues [s] return "got it" def event_loop (tcpserver, inputs, outputs): while inputs: # Wait for at least one of the sockets to be ready for processing print ('Waiting for the next event') # Start the select listener, listen to the server in input_list # When the socket calls the send, recv and other functions, it will call this function again. In this case, the second parameter returned will have the value readable, writable, predictional = select. select (inputs, outputs, inputs) if readable is not None: tcp_recever = tcpserver. handler_recever (readable) if tcp_recever = 'got It': print ("server have received ed") if writable is not None: tcp_send = tcpserver. handler_send (writable) if tcp_send = 'got It': print ("server have send") if exceptional is not None: tcp_exception = tcpserver. handler_exception (exceptional) if tcp_exception = 'got It': print ("server have exception") sleep (0.8) if _ name _ = '_ main _': server_address = ('localhost', 8090) server = socket. socket (socket. AF_INET, socket. SOCK_STREAM) inputs = [server] outputs = [] message_queues = {} tcpserver = TCPServer (server, server_address, inputs, outputs, outputs) # enable event loop event_loop (tcpserver, inputs, outputs)

 

Client:

Import socketmessages = ['this is the message', 'it will be sent', 'in parts',] server_address = ('localhost', 8090) # Create aTCP/IP socketsocks = [socket. socket (socket. AF_INET, socket. SOCK_STREAM), socket. socket (socket. AF_INET, socket. SOCK_STREAM),] # Connect thesocket to the port where the server is listeningprint ('ing ing to % s port % s' % server_address) # Connect to the server for s in socks: s. connect (se Rver_address) for index, message in enumerate (messages): # Send messages on both sockets for s in socks: print ('% s: sending "% s"' % (s. getsockname (), message + str (index) send_data = message + str (index) s. sendall (bytes (send_data, encoding = 'utf-8') # Read responses on both socketsfor s in socks: data = s. recv (1024) print ('% s: received "% s"' % (s. getsockname (), data) if data! = "": Print ('closingsocket ', s. getsockname () s. close ()

As you can see from the results, the client will send an empty character and check the information. This is done when the client determines whether to disconnect, And the Server adds a judgment.
Run:

Refer:

Http://www.cnblogs.com/diegodu/p/3977739.html Unix network programming I/O model chapter 6

Http://www.cnblogs.com/lancidie/archive/2011/12/13/2286408.html [solution] Select Network Model problem -- strange sending and receiving Problem