Python IO multiplexing and pythonio multiplexing

Python IO multiplexing and pythonio multiplexing

IO multiplexing is a concept at the system level. Let's first figure out why I/O multiplexing is used:

Since the execution process of processes is linear (that is, sequential execution), when we call low-speed system I/O (read, write, accept, etc.), the process may be blocked, in this case, the process is blocked on this call and cannot perform other operations. blocking is normal. next, consider the following problem: a server process communicates with a client process, and the server side reads (sockfd1, bud, bufsize). At this time, the client process does not send data, so read (blocking calls) it will be blocked until the client calls write (sockfd, but, size) to send data. this is no problem when one customer communicates with the server. When multiple customers communicate with the server, if the server is blocked by one customer sockfd1, when the data of the other customer reaches sockfd2, the server cannot process and is still blocked in read (sockfd1 ,...) at this time, the problem arises and the service of another customer cannot be processed in time. What should I do? I/O multiplexing to solve this problem! IO multiplexing is essentially to execute multiple IO operations in the same thread or process by means of a switch. Note that each IO operation is actually performed independently. It is changed from one-to-one to many-to-many. For example:
Select, poll, and epoll are all specific implementations of I/O multiplexing. The reason why these three ghosts exist is that they appear sequentially.

After the concept of I/O multiplexing is proposed, select is the first implementation (about 1983 implemented in BSD ).

After the select statement is implemented, many problems are quickly exposed.

• Select modifies the input parameter array, which is unfriendly to a function that needs to be called many times.
• If any sock (I/O stream) has data, select only returns data, but does not tell you that the sock has data, so you can only find data one by one, A dozen sock may be okay. If tens of thousands of sock are searched every time, this meaningless overhead will be quite awesome.
• Select can only monitor 1024 links, which has nothing to do with the grass roots. linux is defined in the header file. For more information, seeFD_SETSIZE.
• The select statement is not thread-safe. If you add a sock to the select statement, then another thread suddenly finds that Nima does not need this sock and needs to be withdrawn. Sorry, this select statement is not supported. If you turn off the sock when you are mad, the standard act of the select statement is .. Uh .. Unpredictable. this is written in the document.

"If a file descriptor being monitored by select () is closed in another thread, the result is unspecified"
Domineering

So after 14 years (1997), a group of people implemented poll, which fixed many select problems, such

• Poll removes the limit of 1024 links, so how many links are required? The Master is happy.
• In terms of design, poll does not modify the input array, but it depends on your platform. Therefore, it is a good idea to be careful when you are on the road.

In fact, 14 years of delay is not a problem of efficiency, but the hardware in that era is too weak. One server handles more than links, which is simply the same as God, the select statement has been met for a long time.

However, poll is still not thread-safe, which means that no matter how powerful the server is, you can only process a group of I/O streams in one thread. Of course you can cooperate with those multi-process, but then you have various problems with the multi-process.

Five years later, in 2002, Davide Libenzi implemented epoll.

Epoll is the latest implementation of I/O multiplexing. epoll fixes most poll and select problems, such:

• Epoll is thread-safe now.
• Epoll not only tells you the data in the sock group, but also tells you which sock has the data. You don't have to find it yourself.

Epoll only supports Linux and is integrated into Linux kernel by default.

Windows Python: provided: selectMac Python: provided: selectLinux Python: provided: select, poll, epoll

For select operations:

Handle list 11, handle list 22, handle list 33 = select. select (handle sequence 1, handle sequence 2, handle sequence 3, timeout) parameters: four parameters are acceptable (the first three must be) return values: the three select methods are used to monitor the file handle. If the handle changes, the handle is obtained. 1. When the handle in the sequence of parameter 1 is readable (accetp and read ), obtain the changed handle and add it to the return value 1 sequence. 2. When the parameter 2 sequence contains a handle, add all the handles in this sequence to the return value 2 sequence. 3. When the handle in the parameter 3 sequence is incorrect, add the handle with the error to the returned value 3 sequence. 4. When the timeout time is not set, select will be blocked until the listening handle changes. When the timeout time is 1, if the handle of the listener does not change, the select statement will be blocked for 1 second, and three empty lists will be returned. If the handle of the listener changes, it will be executed directly.

　　

#! /Usr/bin/env python #-*-coding: UTF-8-*-import socketimport selectsk1 = socket. socket (socket. AF_INET, socket. SOCK_STREAM) sk1.setsockopt (socket. SOL_SOCKET, socket. SO_REUSEADDR, 1) sk1.bind ('2017. 0.0.1 ', 8002) sk1.listen (5) sk1.setblocking (0) inputs = [sk1,] while True: readable_list, writeable_list, error_list = select. select (inputs, [], inputs, 1) for r in readable_list: # if sk1 = r: print 'access' request, address = r when the client first connects to the server. accept () request. setblocking (0) inputs. append (request) # else: sent Ed = r. recv (1024) # if received ed: print 'Received ed data: ', received # else: inputs when the client closes the program normally. remove (r) sk1.close ()Use select to implement pseudo-Concurrent Server #! /Usr/bin/env python #-*-coding: UTF-8-*-import socketip_port = ('2017. 0.0.1 ', 8002) sk = socket. socket () sk. connect (ip_port) while True: indium = raw_input ('Please input: ') sk. sendall (indium) sk. close () uses select to implement pseudo-simultaneous processing of multiple Socket Client Requests: ClientUse select to implement pseudo-concurrent Client