Directory
- 0. Reference Address
- 1. Preface
- 2. Core class
- 3. Selectselector Core Function Code Analysis
- 3.1 Registration
- 3.2 Logout
- 3.3 Queries
- 4. Aliases
- 5. Summary
- 6. Code Error problem
- 1. Number of file descriptors
- 2. Whether the listener list can be empty
- 7. Diagram
0. Reference Address
Basic Introduction https://www.cnblogs.com/yinheyi/p/8127871.html
Experimental demo https://www.cnblogs.com/xybaby/p/6406191.html#_label_2
Detailed explanation http://aju.space/2017/07/31/Drive-into-python-asyncio-programming-part-1.html
Official Document Selecotors https://docs.python.org/3/library/selectors.html
Official Document Select Https://docs.python.org/3/library/select.html
1. Preface
In a concurrent solution, the synchronization model (serial/multi-process/multithreaded) is not suitable for large-scale high concurrency because of the blocking of IO calls. In non-blocking IO calls, we can use a thread to accomplish high concurrency, but because non-blocking IO returns immediately, How to determine when IO is ready and what to do when it is ready becomes critical, so we need to attach additional processing.
Regardless of which additional processing is used, the core is to know that the IO is ready and perform the corresponding operation, and one of the additional processing methods is the callback + event loop.
OShas provided us with a select/poll/epoll/kqueue variety of underlying operating system interfaces to handle IO readiness notifications (i.e. the OS interface can be easily programmed to write event loops). And the program needs to be done: How to perform scheduled operations when IO is ready .
selecotrsmodule, the total code 611 line, where there 5 is a class that is the same level, but differs depending on OS the type. The module also contains a large number of comments, so the number of core code is 100 around the line. selectors The module gives us the callback model in asynchronous programming (and later writes the co-model in asynchronous programming), so I think it is necessary to study this module.
2. Core class
selectorsThe core classes in the module are as follows:
BaseSelector: is an abstract base class that defines a function interface for a core subclass. BaseSelector The core interfaces of the class definition are as follows:
@abstractmethodregister(self, fileobj, events, data=None) # 提供文件对象的注册@abstractmethodunregister(self, fileobj) # 注销已注册的文件对象@abstractmethodselect(self, timeout=None) # 向OS查询准备就绪的文件对象
The first two functions encapsulate the file object and provide a data variable to hold the additional data, which provides the context for the callback. The third function is the select encapsulation of the OS underlying interface to select/poll/epoll provide a unified external interface.
_BaseSelectorImpl: Is a register base class that implements the and, unregister note that this base class does not implement a select function because the function is different from the select OS上 underlying interface used by the different, so it should be defined in the corresponding subclass
SelectSelector: windows interface when used
EpollSelector: linux interface when used (other 3 classes are similar, just apply to different OS )
DefaultSelector: This is the category name, the selectors module chooses the optimal interface according to the type of the operating system.
The following is an analysis of the selectselector core code of the class, and the code logic of the other corresponding classes is basically consistent.
3. Selectselector Core Function Code Analysis
Famous Yuan zuselectorkey
SelectorKey = namedtuple(‘SelectorKey‘, [‘fileobj‘, ‘fd‘, ‘events‘, ‘data‘])
This object is a well-known meta-ancestor, which can be thought of as the encapsulation of file objects fileobj , corresponding descriptor values fd , corresponding events events , and the accompanying data of data these several attributes. This object is the core operand , associated with the file object that needs to be monitored, associated with OS the event that needs attention, and holds the accompanying data ( in fact, the callback function that is put here )
3.1 Registration
def __init__(self): super().__init__() self._readers = set() # 使用集合处理唯一性 self._writers = set()
First, the constructor defines _readers and variables are _writers used to hold the file descriptor values of the file objects that need to be listened to, and the collection attributes are used to handle uniqueness.
def register(self, fileobj, events, data=None): key = super().register(fileobj, events, data) if events & EVENT_READ: self._readers.add(key.fd) if events & EVENT_WRITE: self._writers.add(key.fd) return key
Generally we use the register function as the first action, which represents the file object you need to listen to, and how you want to handle it when it happens to the event you are concerned about.
This function has 3 a parameter, which is a file object, a listener event (readable as 1 , writable as 2 ), with data attached.
fileobjA file object is a class file object that is strongly related to the platform, and windows socket can be linux any linux supported file object on the top.
eventsis a int type of value, that is, EVENT_ERAD andEVENT_WRITE
dataIs the accompanying data, we can put the callback function here
This function returns key a well- selectorkey known meta-ancestor.
registerThe function registers the file objects and events that the user listens to in the named meta-ancestor, and joins the listening collection _readers and _writers
3.2 Logout
def unregister(self, fileobj): key = super().unregister(fileobj) self._readers.discard(key.fd) self._writers.discard(key.fd) return key
When we do not need to listen to a file object, use to unregister unregister it. This will cause it to _readers _writers be ejected from and.
3.3 Queries
def select(self, timeout=None): timeout = None if timeout is None else max(timeout, 0) ready = [] try: r, w, _ = self._select(self._readers, self._writers, [], timeout) except InterruptedError: return ready r = set(r) w = set(w) for fd in r | w: events = 0 if fd in r: events |= EVENT_READ if fd in w: events |= EVENT_WRITE key = self._key_from_fd(fd) if key: ready.append((key, events & key.events)) return ready
This code describes the OS query logic that is initiated by the user select . The parameters of the function timeout are by default None , which means that by default, if no one of the ready events occurs, the select call is blocked forever.
selectThe function calls the underlying interface, which differs from the definition of the select/poll/epoll SelectSelector class and EpollSelector class, OS calls the corresponding interface according to the type, windows and compares the underlying interface of the linux actual invocation as follows:
用户统一调用高层select函数,此函数实际调用的接口为:# windows下使用select(SelectSelector类)r, w, _ = self._select(self._readers, self._writers, [], timeout)# linux下使用epoll(EpollSelector类)fd_event_list = self._epoll.poll(timeout, max_ev)
The function uses ready variables to save the ready-to-Ganso(key, events)
In windows , once the underlying select interface returns, a list is given, the 3 first two represent a list of readable and writable file objects, and use collection processing as uniqueness. The ready-to-Ganso object is added to the ready list to return. If you define a timeout non None - And a timeout occurs, an empty list is returned.
4. Aliases
# Choose the best implementation, roughly:# epoll|kqueue|devpoll > poll > select.# select() also can‘t accept a FD > FD_SETSIZE (usually around 1024)if ‘KqueueSelector‘ in globals(): DefaultSelector = KqueueSelectorelif ‘EpollSelector‘ in globals(): DefaultSelector = EpollSelectorelif ‘DevpollSelector‘ in globals(): DefaultSelector = DevpollSelectorelif ‘PollSelector‘ in globals(): DefaultSelector = PollSelectorelse: DefaultSelector = SelectSelector
selectorsThe module defines an alias DefaultSelector used to OS automatically point to the optimal interface class based on the type.
5. Summary
1 The interface provided by the operating system select/poll/epoll can be used to write event loops , while modules selectors encapsulate select modules, which select are low-level modules that encapsulate select/poll/epoll/kqueue such interfaces.
The 2 selectors module defines a named meta-ancestor selectorkey , which encapsulates the file object/Descriptor Value/Event/accompanying data, selectorkey providing us with a callback environment
3 using selectors A module can be implemented using a callback model to accomplish high concurrency scenarios.
4 (very important) asynchronous callback model, most of the events and energies are the design of the callback function. The callback model allows each IO operation to be separated as a function, which splits the code, resulting in a decrease in readability and a rise in maintenance difficulty.
The communication between the 5 callback functions is difficult and needs to be passed through layer functions.
6 Back adjustment model is difficult to understand
6. Code error question 1. Number of file descriptors
Traceback (most recent call last): File "F:/projects/hello/hello.py", line 119, in <module> loop() File "F:/projects/hello/hello.py", line 102, in loop events = selector.select() File "F:\projects\hello\selectors.py", line 323, in select r, w, _ = self._select(self._readers, self._writers, [], timeout) File "F:\projects\hello\selectors.py", line 314, in _select r, w, x = select.select(r, w, w, timeout)ValueError: too many file descriptors in select()
On the windows bottom, the underlying is using an select interface that can support the number of file descriptors theory is that 1024 the actual test descriptor must be less than 512 (My computer is win10 64bit )
linuxon the use epoll , you can support 1024 the number of file descriptors greater than, but the test found in the left and right 4000 when the error will be.
stack overflowExplanation 1:https://stackoverflow.com/questions/31321127/too-many-file-descriptors-in-select-python-in-windows
stack overflowExplanation 2:
Https://stackoverflow.com/questions/47675410/python-asyncio-aiohttp-valueerror-too-many-file-descriptors-in-select-on-win
2. Whether the listener list can be empty
Traceback (most recent call last): File "F:/projects/hello/world.py", line 407, in <module> loop() File "F:/projects/hello/world.py", line 378, in loop events = selector.select() File "F:\projects\hello\selectors.py", line 323, in select r, w, _ = self._select(self._readers, self._writers, [], timeout) File "F:\projects\hello\selectors.py", line 314, in _select r, w, x = select.select(r, w, w, timeout)OSError: [WinError 10022] 提供了一个无效的参数。
On windows , the list of monitored file objects cannot be empty:
7. Diagram
Python asynchronous Programming-callback model (selectors module)