Python simple construction non-blocking single process, select mode, epoll mode service

As often crawled article content, attached here blog article URL:, occasionally update some error data or text, suggest to my blog address:-click here

1 single-Process server-non-clogging mode

Service side:

#coding =utf-8 fromSocket Import *The import time# is used to store all the new connected sockets, this is the key g_socketlist=[]def Main (): Sersocket=socket (af_inet, Sock_stream) sersocket.setsockopt (Sol_socket, SO_REUSEADDR,1) Localaddr= ("',7788) Sersocket.bind (localaddr) #这里可以适当修改listen的值来看看不同的现象 Sersocket.listen ( +#. Set the socket to non-clogging #. This is very important, set to non-clogging, if the accept, there is no client connection, then there will be an exception, #. So we use try to handle this function sersocket.setblocking (False )         whileTrue:Try: Newclientinfo=sersocket.accept () except Exception asResult:passElse: Print (".. The new client is connected.:%s"%str (newclientinfo)) #这里依旧设置为非堵塞 newclientinfo[0].setblocking (False) #这里重点加入列表 g_socketlist.append (newclientinfo) # to store the guest that needs to be deleted User Information Needdelclientinfolist=[] #这里循环读取socket列表 to request a client connection, which is a streamlined version of the select version #但是是理解select版本的核心 forClientsocket,clientaddrinchg_socketlist:Try: RecvData= Clientsocket.recv (1024x768)                ifLen (RecvData) >0: Print ('recv[%s]:%s'%(str (CLIENTADDR), recvdata))Else: Print ('[%s] client is closed'%str (CLIENTADDR)) Clientsocket.close () G_needdelclientinfolist.append (clientsock ET,CLIENTADDR)) except Exception asResult:pass forNeeddelclientinfoinchNeedDelClientInfoList:g_socketList.remove (needdelclientinfo)if__name__ = ='__main__': Main ()

Client:

#coding =utf-8 fromSocket Import *Import Randomimport Timeserverip= Raw_input ("please lose. IP of the server:") Connnum= Raw_input ("please lose. Number of times to link the server (for example,:)") G_socketlist= [] forIinchRangeint(Connnum)): s=socket (af_inet, Sock_stream) s.connect (ServerIP,7788)) G_socketlist.append (s) print (i) whileTrue: forSinchg_socketlist:s.send (str (Random.randint (0, -)) # ... time.sleep (1)

As we can see, the key point is that for each saved socket in the For loop, the loop request has data to be sent, because it is fast enough to look like multi-process processing.

The disadvantage of this model is that when more and more connections, for the polling time spent more and more, when there are thousands of connections, actually only one client has data to send, the service side is still an endless poll to ask, to find whether there is a request. Then there is the improved version of the SELEC model service:

2 Single-process Select Edition TCP server

In the multiplexed model, there are more common select models and Epoll models. Both are system interfaces that are provided by the operating system. Of course, Python's Select module is in a more advanced package. The network communication is abstracted by UNIX system to read and write files, usually one device, which is provided by the device driver, and the driver can know whether its data is available. Device drivers that support blocking operations typically implement a set of their own waiting queues, such as the block or non-block operations required by the read/write wait queue to support the upper layer (user tier). The resource for the device's files, if available (readable or writable), notifies the process, which in turn causes the process to sleep, and then wakes the process when the data arrives available. The file descriptors for these devices are placed in an array, and then the select invokes the array and returns the file descriptor if the file descriptor is readable. When the traversal is complete, if there is still no available device file descriptor, select causes the user process to sleep until the resource is available and wakes up before traversing the array of watches. Each traversal is judged in turn.

ImportSelectImport Socketimport sysserver=Socket.socket (socket.af_inet, socket. Sock_stream) Server.bind (("',7788)) Server.listen (5) Inputs=[Server, Sys.stdin]running=True whileTrue: # callSelectfunction, blocking wait readable, writeable, exceptional=Select.Select(inputs, [], []) # data arrives, loops forSockinchreadable: # The Monitor hears a new connectionifSock = =Server:conn, addr=server.accept () #SelectMonitor the Socket Inputs.append (conn) # Monitor to hear the keyboard has input elif sock==Sys.stdin:cmd=Sys.stdin.readline () running=False Break# There's data to arriveElse: # Read data sent by client connection= Sock.recv (1024x768)            ifdata:sock.send (data)Else: # Remove Select listener socket inputs.remove (sock) Sock.close () # If user input is detected tap keyboard , then quitifNot running: Breakserver.close ()

The disadvantage of select is that there is a maximum limit on the number of file descriptors that a single process can monitor, typically 1024 on Linux, which can be improved by modifying the macro definition of a promotion to recompile the kernel, but this also results in a decrease in efficiency. In general, this number and system memory relationship is very large, the specific number can be Cat/proc/sys/fs/file-max view. A 32-bit machine defaults to 1024. 64-bit Machine The default is 2048. Scanning the socket is scanned sequentially, that is, using polling method, the efficiency is low. When the socket is more, each time select () through the traversal fd_setsize socket to complete the dispatch, regardless of which socket is active, to traverse. This can waste a lot of CPU time.

3 Service side of version Epoll

Import SocketimportSelect# Create socket S=Socket.socket (Socket.af_inet,socket. SOCK_STREAM) # settings can be repeated to make the. Binding information s.setsockopt (socket. Sol_socket,socket. SO_REUSEADDR,1) # Bind the Machine Information S.bind (("",7788) # becomes passive S.listen (Ten) # Create a. Epoll Object Epoll=Select. Epoll () # adds the created socket to the Epoll event listener Epoll.register (S.fileno (),Select. epollin|Select. Epollet) Connections={}addresses={The}# loop waits for the client to arrive or to. Send Data whileTrue: # epoll into. FD scans the ground.--timeout is not specified for blocking wait epoll_list=Epoll.poll () # on the event in. Judging forFd,eventsinchepoll_list: # If the socket created by the socket is activatedifFD = =S.fileno (): Conn,addr=s.accept () print ('There is a new client coming to%s'%Str (ADDR) # Save Conn and addr information separately Connections[conn.fileno ()]=Conn Addresses[conn.fileno ()]=Addr # Registers a readable event Epoll.register (Conn.fileno () that connects the socket to Epoll,Select. Epollin |Selectelif Events==Select. Epollin:recvdata= Connections[fd].recv (1024x768)            ifLen (RecvData) >0: Print ('recv:%s'%recvdata)Else: # Remove the connection from Epoll FD Epoll.unregister (FD) # Server side active off Close the connection fd connections[fd].close () print ("%s---offline---"%str (ADDRESSES[FD]))

The benefits of Epoll are:

1. There is no limit to the maximum concurrent connection, can open the FD (refers to the file descriptor, the popular understanding is the socket corresponding to the number of numbers) the upper limit is far greater than 1024
2. Efficiency improvement, not polling method, does not decrease with the increase of FD number. Only active FD calls the callback function, which means that the greatest advantage of epoll is that it is independent of the total number of connections, and therefore, in the actual network environment, Epoll is much more efficient than select and poll.

That is to say Epoll is an event mechanism, no longer every time you start polling, one by one to ask if there is a client to send data, but "ask" the client, who has the data? This is the active client will "raise your Hand", then epoll this function will return to the active socket link.

Python simple construction non-blocking single process, select mode, epoll mode service