Python network programming-epoll of IO multiplexing

Python network programming-epoll of IO multiplexing

What is epoll?

What is epoll? In linux network programming, select is used for event triggering for a long time. In the new linux kernel, there is a mechanism to replace it, that is, epoll. Of course, this is not only available in the 2.6 kernel. it is introduced in the 2.5.44 kernel (epoll (4) is a new API introduced in Linux kernel 2.5.44 ), it has almost all the advantages mentioned previously, and is widely recognized as the best multi-channel multiplexing I/O ready notification method in Linux2.6.

Compared with select, epoll does not reduce the efficiency as the number of fd listeners increases. In the select implementation in the kernel, it uses polling for processing. the larger the number of fd polling, the more time it takes.

How epoll works

Epoll also only informs the ready file descriptors. when we call epoll_wait () to obtain the ready file descriptor, the returned value is not the actual descriptor, but a value representing the number of ready descriptors, you only need to obtain the corresponding number of file descriptors in sequence in an array specified by epoll. the memory ing (mmap) technology is also used here, this completely saves the overhead of copying these file descriptors during system calls.

Another essential improvement is that epoll uses event-based readiness notification. In select/poll, the kernel scans all monitored file descriptors only after a certain method is called, and epoll registers a file descriptor through epoll_ctl () in advance, once a file descriptor is ready, the kernel uses a callback mechanism similar to callback to quickly activate the file descriptor. when the process calls epoll_wait (), it is notified.

We can see from the above that epoll improves the select and poll models, improves the network programming performance, and is widely used in large-scale concurrent requests in the C/S architecture.

Epoll in python

1. trigger method:

Edge trigger/horizontal trigger, applicable only to Unix/Linux operating systems

2. schematic diagram

3. general steps

Create an epoll object -- Create an epoll object

Tell the epoll object to monitor specific events on specific ETS -- tells the epoll object to listen to the specified event on the specified socket

Ask the epoll object which sockets may have had the specified event since the last query -- Ask the epoll object which socket has occurred and which specified events since the last query

Perform some action on those sockets -- Perform some operations on these sockets

Tell the epoll object to modify the list of sockets and/or events to monitor -- Tell epoll object, modify socket list and/or event, and monitor

Repeat steps 3 through 5 until finished -- Repeat Step 3-5 until completion

Destroy the epoll object -- Destroy the epoll object

4. usage

Import select to select module

Epoll = select. epoll () create an epoll object

Epoll. register (file handle, event type) registers the file handle and event to be monitored

Event Type:

Select. EPOLLIN readable event

Select. EPOLLOUT writable event

Select. EPOLLERR error event

Select. EPOLLHUP client disconnection event

Epoll. unregister (file handle) destroy file handle

Epoll. poll (timeout) when the file handle changes, it will report it to the user process in the form of a list, timeout

The time-out period. The default value is-1, that is, wait until the file handle changes.

Epoll reports changes to the current file handle every 1 second. If no changes are made, an empty value is returned.

Epoll. fileno () returns the epoll control file descriptor (Return the epoll control file descriptor)

Epoll. modfiy (fineno, event) fineno is the file descriptor. event is the event type. it modifies the event corresponding to the file descriptor.

Epoll. fromfd (fileno) creates an epoll object from a specified file descriptor

Epoll. close () disables the control file descriptor of the epoll object

5. instance: the client sends data to the server and returns the received data to the client.

Server code

#! /Usr/bin/env python #-*-coding: UTF-8-*-import socketimport selectimport Queue # create the socket object serversocket = socket. socket (socket. AF_INET, socket. SOCK_STREAM) # set the IP address to reuse serversocket. setsockopt (socket. SOL_SOCKET, socket. SO_REUSEADDR, 1) # IP address and port number server_address = ("127.0.0.1", 8888) # bind the IP address serversocket. bind (server_address) # listen and set the maximum number of connections to serversocket. listen (10) print "server started successfully, listener IP:", server_address # server side sets non-blocking serversocket. setblocking (False) # time-out time timeout = 10 # Create an epoll event object, and add the events to be monitored to epoll = select. epoll () # register the server to listen to fd to the epoll collection of read wait events. register (serversocket. fileno (), select. EPOLLIN) # save the Dictionary of client-connected messages. the format is {} message_queues ={}# Dictionary of the file handle to the corresponding object. the format is {handle: Object} fd_to_socket = {serversocket. fileno (): serversocket,} while True: print "waiting for active connection ...... "# poll the registered event set. the returned value is [(file handle, corresponding event), (...),...] events = epoll. poll (timeout) if not events: print "epoll times out and no active connections are available. Round Robin again ...... "continue print" has ", len (events)," new events, start to process ...... "for fd, event in events: socket = fd_to_socket [fd] # if the active socket is the current server socket, there is a new connection if socket = serversocket: connection, address = serversocket. accept () print "new connection:", address # The new connection socket is set to non-blocking connection. setblocking (False) # register a new fd connection to the epoll event collection to be read. register (connection. fileno (), select. EPOLLIN) # save the file handle and object of the new connection to the dictionary fd_to_socket [connection. fileno ()] = connection # use the object of the new connection as the key value, the value is stored in the Queue, and the information of each connection is saved. message_queues [connection] = Queue. queue () # Disable the event elif event & select. EPOLLHUP: print 'Client close' # cancel the file handle epoll of the client in epoll. unregister (fd) # Disable the file handle of the client fd_to_socket [fd]. close () # delete information related to closed clients in the dictionary del fd_to_socket [fd] # Read event elif event & select. EPOLLIN: # receive data = socket. recv (1024) if data: print "received data:", data, "client:", socket. getpeername () # put the data into the dictionary message_queues [socket] of the corresponding client. put (data) # Modify the connection to the read-to-write event set (that is, after the client receives the message, modify its fd and add it to the write event set) epoll. modify (fd, select. EPOLLOUT) # writeable event elif event & select. EPOLLOUT: try: # obtain the information of the corresponding client from the dictionary msg = message_queues [socket]. get_nowait () Queue T Queue. empty: print socket. getpeername (), "queue empty" # Modify the file handle to read the epoll event. modify (fd, select. EPOLLIN) else: print "send data:", data, "client:", socket. getpeername () # send data socket. send (msg) # deregister the server file handle epoll in epoll. unregister (serversocket. fileno () # Disable epollepoll. close () # close the socketserversocket server. close ()

Client code:

#! /Usr/bin/env python #-*-coding: UTF-8-*-import socket # create client socket object clientsocket = socket. socket (socket. AF_INET, socket. SOCK_STREAM) # server IP address and port number tuples server_address = ('2017. 0.0.1 ', 8888) # The IP address and port number specified for client connection. connect (server_address) while True: # input data = raw_input ('Please input: ') # client sends data to clientsocket. sendall (data) # the client receives data server_data = clientsocket. recv (1024) print 'data received by the client: 'server _ data # Disable the client socket clientsocket. close ()