threadlocal variables in Python

Werkzeug as a WSGI tool library, due to some considerations, it does not directly use the Python built-in threadlocal class, but instead implements a series of local classes. Includes simple local, and Localstack,localmanager and Localproxy implemented on this basis. Let's take a look at how these classes are used, the purpose of the design, and the specific implementation techniques.

Design of the Local class

Werkzeug's designers believe that Python's own threadlocal does not meet demand, mainly because of the following two reasons:

Werkzeug mainly use "ThreadLocal" to meet the concurrency requirements, Python's own ThreadLocal can only implement thread-based concurrency. There are many other concurrency methods in python, such as common Greenlet, so a local object that supports the process needs to be implemented.

WSGI does not guarantee that a new thread will be generated each time to process the request, which means that the thread is reusable (a thread pool can be maintained to process the request). This way, if Werkzeug uses Python's own threadlocal, a thread that is "dirty" (with data about previously processed requests) will be used to process the new request.

To solve these two problems, the local class is implemented in Werkzeug. The local object can be used to isolate the data between threads and processes, and also to support the cleanup of data from a thread or a process (so that after processing a request, the corresponding data can be cleaned up and then waited for the next request to arrive).

Specifically how to achieve, the idea is actually very simple, we in-depth understanding of Python in the threadlocal variable (above) the end of the article, is to create a global dictionary, and then the thread (or the association) identifier as a key, the corresponding thread (or association) local data as Value Here Werkzeug is implemented according to the above ideas, but the use of some Python black magic, finally provide users with a clear, simple interface.

Specific implementation

The implementation of the Local class is analyzed in werkzeug.local with the 8a84b62 version of the code. Through the first two threadlocal, we already know the characteristics of the local object and how to use it. So here is no longer an example of the use of local objects, we look directly at the code.

Class Local (object):     __slots__ = (' __storage__ ', ' __ident_func__ ')          def __init__ (self):         object.__setattr __ (self, ' __storage__ ', {})         object.__setattr__ (self, ' __ident_func__ ', get_ident)     ...

Because there may be a large number of local objects, in order to save space occupied by the local object, this uses __slots__ to write dead the properties that local can have:

__STORAGE__: The value is a dictionary, used to save the actual data, initialized to null;

__IDENT_FUNC__: value is a function used to find the current thread or the identifier of the co-path.

Because the local object's actual data is stored in __storage__, the operation on the local property is actually a __storage__ operation. For the Get property, here is the Magic method __getattr__ intercept the __storage__ and __ident_func__ properties, and directs it to the data of the current thread or the coprocessor stored by the __storage__. For the set or del of the attribute value, the __setattr__ and __setattr__ are used respectively (the introduction of these magic methods is described in attribute control). The key code is as follows:

def __getattr__ (self, name):     try:         return self.__storage__[self.__ident_func__ ()][name]     except Keyerror :         raise Attributeerror (name)  def __setattr__ (self, Name, value):     ident = self.__ident_func__ ()     Storage = self.__storage__     try:         storage[ident][name] = value     except Keyerror:         storage[ident] = {Name : value}  def __delattr__ (self, name):     try:         del self.__storage__[self.__ident_func__ ()][name]     Except Keyerror:         raise Attributeerror (name)

Let's say we have an ID of. N threads or processes, each with a local object that holds some of its own local data, the contents of the local object are as follows:

In addition, the local class provides a __release_local__ method that frees the current thread or the data saved by the process.

Local Expansion Interface

Werkzeug implements Localstack and Localmanager on a Local basis to provide more friendly interface support.

Localstack

Localstack by encapsulating the local to achieve a thread (or association) independent stack structure, note that there is a specific use of the method, a simple use example is as follows

ls = Localstack () ls.push () print ls.top    # print ls._local.__storage__ # {140735190843392: {' stack ': [12]}}

Localstack's implementation is interesting, it takes a local object as its own property _local, and then defines the interface push, pop, and top methods for the appropriate stack operation. Here we use the list of _local.__storage__._local.__ident_func__ () to simulate the stack structure. In the interface push, pop and top, by manipulating the list to simulate the operation of the stack, it is important to note that when the list is acquired inside the interface function, it is not as complex as the blackbody above and can be directly used with the _local getattr () method. Take the push function as an example to achieve the following:

def push (self, obj): "" "     pushes a new item to the stack" "     RV = getattr (self._local, ' stack ', None)     if RV is No NE:         self._local.stack = RV = []     rv.append (obj)     return RV

The implementation of pop and top is similar to the general stack, with the corresponding operation on the list of stack = GetAttr (self._local, ' stack ', None). In addition, Localstack allows us to customize __ident_func__, which generates descriptors with built-in function properties, encapsulates __ident_func__ get and set operations, and provides a property value __ident_func__ As an interface, the specific code is as follows:

def _get__ident_func__ (self):     return self._local.__ident_func__  def _set__ident_func__ (self, value):     object.__setattr__ (self._local, ' __ident_func__ ', value) __ident_func__ = Property (_get__ident_func__, _set__ident_ func__) del _get__ident_func__, _set__ident_func__

Localmanager

Local and Localstack are both thread-independent and individual objects, and many times we need a thread or a separate container to organize multiple local or Localstack objects (as if we were using a list to organize multiple int or string types )。

Werkzeug implements the Localmanager, which stores the managed local or Localstack objects through a list-type property locals, and also provides a cleanup method to release all local objects. Werkzeug Localmanager Most important interface is the adorner method Make_middleware, the code is as follows:

def make_middleware (self, app): "" "     Wrap a WSGI application so cleaning up happens after     request end."     "     def application (environ, start_response):         return Closingiterator (App (environ, start_response), Self.cleanup)     return application

This decorator registers the callback function cleanup, and when a thread (or a coprocessor) finishes processing the request, it calls cleanup to clean up the local or Localstack object it manages (Closingiterator implementation in WERKZEUG.WSGI )。 Here is a simple example of using Localmanager:

From werkzeug.local Import Local, localmanager  local = local () local_2 = local () Local_manager = Localmanager ([Local, Local2])  def application (environ, start_response):     local.request = Request = Request (environ) ...  # application will automatically clean up the contents of Local_manager after processing is finished

With Localmanager's make_middleware we can empty all local or Localstack objects after a thread (coprocessor) finishes processing a request, so that the thread can process another request. At this point, the second problem mentioned at the beginning of the article can be solved. Werkzeug.local also implements a localproxy used as the local object of the agent, is also very worthwhile to learn.

