Decrypts the descriptor in Python

Decrypts the descriptor in Python

This article mainly introduces how to decrypt the descriptor (descriptor) in Python. This article describes in detail the role of the descriptor (descriptor), the Access descriptor, the assignment of the descriptor, and the deletion of the descriptor, for more information, see

Python contains many built-in language features that make the code concise and easy to understand. These features include list/set/dictionary derivation, property, and decorator ). For most of the features, these "intermediate" language features are well documented and easy to learn.

But here is an exception, that is, the descriptor. At least for me, descriptors are the longest feature that bothers me at the core of the Python language. There are several reasons:

1. the official documentation on descriptors is quite difficult, and there are no good examples to tell you why you need to write Descriptors (I have to defend Raymond Hettinger, the Python articles and videos on other topics he wrote are of great help to me)

2. the syntax for writing descriptors seems a little weird.

3. Custom descriptors may be the least used feature in Python, so it is difficult for you to find excellent examples in open-source projects.

However, once you understand it, the descriptor does have its application value. This article tells you what descriptors can be used for and why they should attract your attention.

In a word, descriptors are reusable attributes.

Here I want to tell you: basically, descriptors are reusable attributes. That is to say, the descriptor allows you to write such code:

The Code is as follows:

F = Foo ()

B = f. bar

F. bar = c

Del f. bar

When the interpreter executes the above Code, it finds that you are trying to access the attribute (B = f. bar), assign values to attributes (f. bar = c) or delete the attribute of an instance variable (del f. bar), it will call the custom method.

Let's explain why it is a great benefit to disguise function calls for Attribute access.

Property -- disguise function calls for Attribute access

Imagine that you are writing code for managing movie information. The Movie class you wrote last may look like this:

The Code is as follows:

Class Movie (object ):

Def _ init _ (self, title, rating, runtime, budget, gross ):

Self. title = title

Self. rating = rating

Self. runtime = runtime

Self. budget = budget

Self. gross = gross

Def profit (self ):

Return self. gross-self. budget

You started to use this class elsewhere in the project, but then you realized: What if you accidentally score a movie? You think this is a wrong behavior. We hope the Movie class can block this error. First, you can change the Movie class to the following:

The Code is as follows:

Class Movie (object ):

Def _ init _ (self, title, rating, runtime, budget, gross ):

Self. title = title

Self. rating = rating

Self. runtime = runtime

Self. gross = gross

If budget <0:

Raise ValueError ("Negative value not allowed: % s" % budget)

Self. budget = budget

Def profit (self ):

Return self. gross-self. budget

But this does not work. Because other parts of the Code are directly through Movie. budget: The newly modified class only captures the wrong data in the init method, but it is powerless for existing class instances. If someone tries to run m. budget =-100, no one can stop it. As a Python programmer and a movie fan, what should you do?

Fortunately, the Python property solves this problem. If you have never seen the use of property, the following is an example:

The Code is as follows:

Class Movie (object ):

Def _ init _ (self, title, rating, runtime, budget, gross ):

Self. _ budget = None

Self. title = title

Self. rating = rating

Self. runtime = runtime

Self. gross = gross

Self. budget = budget

@ Property

Def budget (self ):

Return self. _ budget

@ Budget. setter

Def budget (self, value ):

If value <0:

Raise ValueError ("Negative value not allowed: % s" % value)

Self. _ budget = value

Def profit (self ):

Return self. gross-self. budget

M = Movie ('casablanca', 97,102,964 000, 1300000)

Print m. budget # CILS m. budget (), returns result

Try:

M. budget =-100 # CILS budget. setter (-100), and raises ValueError

Failed t ValueError:

Print "Woops. Not allowed"

964000

Woops. Not allowed

The @ property modifier is used to specify a getter method, and the @ budget. setter modifier is used to specify a setter method. When we do this, Python automatically calls the corresponding getter/setter method whenever someone tries to access the budget attribute. For example, when you encounter Code such as m. budget = value, budget. setter is automatically called.

Take some time to appreciate how elegant Python is in doing so: without property, we will have to hide all instance properties and provide a large number of explicit methods similar to get_budget and set_budget. If you write classes like this, the getter/setter methods will be constantly called, which looks like a bloated Java code. Worse, if we do not adopt this encoding style, we can directly access the instance attributes. Then we won't be able to add a condition check for non-negative numbers in a clear way later -- we have to re-create the set_budget method and then search for the source code in the entire project. replace budget = value with m. set_budget (value ). It hurts a lot !!

Therefore, property allows us to link the custom code with the access/setting of the variable, while maintaining a simple interface for accessing the attribute for your class. Pretty good!

Insufficient property

For property, the biggest drawback is that they cannot be reused. For example, if you want to add non-negative checks for the rating, runtime, and gross fields. The new modified classes are as follows:

The Code is as follows:

Class Movie (object ):

Def _ init _ (self, title, rating, runtime, budget, gross ):

Self. _ rating = None

Self. _ runtime = None

Self. _ budget = None

Self. _ gross = None

Self. title = title

Self. rating = rating

Self. runtime = runtime

Self. gross = gross

Self. budget = budget

# Nice

@ Property

Def budget (self ):

Return self. _ budget

@ Budget. setter

Def budget (self, value ):

If value <0:

Raise ValueError ("Negative value not allowed: % s" % value)

Self. _ budget = value

# OK

@ Property

Def rating (self ):

Return self. _ rating

@ Rating. setter

Def rating (self, value ):

If value <0:

Raise ValueError ("Negative value not allowed: % s" % value)

Self. _ rating = value

# Uhh...

@ Property

Def runtime (self ):

Return self. _ runtime

@ Runtime. setter

Def runtime (self, value ):

If value <0:

Raise ValueError ("Negative value not allowed: % s" % value)

Self. _ runtime = value

# Is this forever?

@ Property

Def gross (self ):

Return self. _ gross

@ Gross. setter

Def gross (self, value ):

If value <0:

Raise ValueError ("Negative value not allowed: % s" % value)

Self. _ gross = value

Def profit (self ):

Return self. gross-self. budget

We can see that the Code has increased a lot, but there are also a lot of repeated logic. Although property can make the class looks neat and beautiful from the outside, it cannot be as neat and beautiful as inside.

Descriptor debut (final killer)

This is the problem solved by the descriptor. Descriptor is an upgraded version of property, allowing you to write a separate class for repeated property logic. The following example shows how the descriptor works (now you don't have to worry about the implementation of the NonNegative class ):

The Code is as follows:

From weakref import WeakKeyDictionary

Class NonNegative (object ):

"A descriptor that forbids negative values """

Def _ init _ (self, default ):

Self. default = default

Self. data = WeakKeyDictionary ()

Def _ get _ (self, instance, owner ):

# We get here when someone CILS x. d, and d is a NonNegative instance

# Instance = x

# Owner = type (x)

Return self. data. get (instance, self. default)

Def _ set _ (self, instance, value ):

# We get here when someone callx. d = val, and d is a NonNegative instance

# Instance = x

# Value = val

If value <0:

Raise ValueError ("Negative value not allowed: % s" % value)

Self. data [instance] = value

Class Movie (object ):

# Always put descriptors at the class-level

Rating = NonNegative (0)

Runtime = NonNegative (0)

Budget = NonNegative (0)

Gross = NonNegative (0)

Def _ init _ (self, title, rating, runtime, budget, gross ):

Self. title = title

Self. rating = rating

Self. runtime = runtime

Self. budget = budget

Self. gross = gross

Def profit (self ):

Return self. gross-self. budget

M = Movie ('casablanca', 97,102,964 000, 1300000)

Print m. budget # callmovie. budget. _ get _ (m, Movie)

M. rating = 100 # callmovie. budget. _ set _ (m, 100)

Try:

M. rating =-1 # callmovie. budget. _ set _ (m,-100)

Failed t ValueError:

Print "Woops, negative value"

964000

Woops, negative value

Some New syntaxes are introduced here. Let's look at them one by one:

NonNegative is a descriptor object because it defines the _ get __,__ set _ or _ delete _ method.

The Movie class now looks very clear. We have created four descriptors at the class level and treated them as common instance attributes. Obviously, the descriptor performs a non-negative check for us here.

Access Descriptor

When the interpreter encounters print m. when buget is used, it regards budget as a descriptor with the _ get _ method and calls Movie. budget. instead of directly passing m. budget to print. This is similar to accessing a property. Python automatically calls a method and returns the result.

_ Get _ receives two parameters: one is the instance object on the left of the node number (here, m in m. budget), and the other is the type of the Instance (Movie ). In some Python documents, Movie is called the descriptor owner ). If we need to access Movie. budget, Python will call Movie. budget. _ get _ (None, Movie ). We can see that the first parameter is either the owner's instance or None. These input parameters may look strange, but here they tell you which object the descriptor belongs. This makes sense when we see the implementation of NonNegative classes.

Assign a value to the descriptor

When the interpreter sees m. when rating = 100, Python identifies rating as a descriptor with a set method, and CALLS Movie. rating. _ set _ (m, 100 ). Like _ get _, the first parameter __set _ is the class instance on the left of the vertex (m. rating = 100 ). The second parameter is the assigned value (100 ).

Delete Descriptor

For the sake of completeness, here we will mention the deletion. If you call del m. budget, Python will call Movie. budget. _ delete _ (m ).

How does the NonNegative class work?

With the above confusions, we will finally reveal how the NonNegative class works. Each NonNegative instance maintains a dictionary, which stores the ing between the owner instance and the corresponding data. When we call m. budget, the __get _ method searches for data associated with m and returns this result (if this value does not exist, a default value is returned ). The _ set _ method is the same, but an extra non-negative check is included here. We use WeakKeyDictionary to replace normal dictionaries to prevent memory leaks-we don't want to keep a useless shard instance in the descriptor dictionary.

Using the descriptor will be a bit awkward. Because they act on the class level, each class instance shares the same descriptor. This means that the descriptor has to manually manage different States for different instance objects, at the same time, you must explicitly pass the class instance as the first parameter to the _ get _, _ set _, and _ delete _ methods.

I want this example to explain what descriptors can be used for-they provide a way to isolate the property logic into a separate class for processing. If you find that you are repeating the same logic between different properties, this article may become a clue for you to think about why using descriptors to refactor the code is worth a try.

Tips and traps

Put the descriptor at the class level)

To make descriptors work properly, they must be defined at the class level. If you do not do this, Python cannot automatically call the _ get _ and _ set _ methods for you.

The Code is as follows:

Class Broken (object ):

Y = NonNegative (5)

Def _ init _ (self ):

Self. x = NonNegative (0) # NOT a good descriptor

B = Broken ()

Print "X is % s, Y is % s" % (B. x, B. y)

X is <__ main _. NonNegative object at 0x000032c250>, Y is 5

As you can see, the descriptor y at the category level can automatically call _ get __. However, the descriptor x at the access instance level only returns the descriptor itself, which is actually magic.

Make sure that the instance data only belongs to the instance.

You may write NonNegative descriptors like this:

The Code is as follows:

Class BrokenNonNegative (object ):

Def _ init _ (self, default ):

Self. value = default

Def _ get _ (self, instance, owner ):

Return self. value

Def _ set _ (self, instance, value ):

If value <0:

Raise ValueError ("Negative value not allowed: % s" % value)

Self. value = value

Class Foo (object ):

Bar = BrokenNonNegative (5)

F = Foo ()

Try:

F. bar =-1

Failed t ValueError:

Print "Caught the invalid assignment"

Caught the invalid assignment

This seems to work properly. But the problem here is that all Foo instances share the same bar, which produces some painful results:

The Code is as follows:

Class Foo (object ):

Bar = BrokenNonNegative (5)

F = Foo ()

G = Foo ()

Print "f. bar is % s \ ng. bar is % s" % (f. bar, g. bar)

Print "Setting f. bar to 10"

F. bar = 10

Print "f. bar is % s \ ng. bar is % s" % (f. bar, g. bar) # ouch

F. bar is 5

G. bar is 5

Setting f. bar to 10

F. bar is 10

G. bar is 10

This is why we need to use the data dictionary in NonNegative. The first parameters of _ get _ and _ set _ tell us which instance we need to care about. NonNegative uses this parameter as the dictionary key and saves a separate copy of data for each Foo instance.

The Code is as follows:

Class Foo (object ):

Bar = NonNegative (5)

F = Foo ()

G = Foo ()

Print "f. bar is % s \ ng. bar is % s" % (f. bar, g. bar)

Print "Setting f. bar to 10"

F. bar = 10

Print "f. bar is % s \ ng. bar is % s" % (f. bar, g. bar) # better

F. bar is 5

G. bar is 5

Setting f. bar to 10

F. bar is 10

G. bar is 5

This is the most awkward part of the descriptor (Frankly speaking, I don't understand why Python won't let you define the descriptor at the instance level, in addition, the actual processing is always distributed to get and set. There must be a reason for this failure)

Note that the descriptor owner cannot be hashed.

The NonNegative class uses a dictionary to separately store data dedicated to instances. This is generally okay unless you use an unhashable object:

The Code is as follows:

Class MoProblems (list): # you can't use lists as dictionary keys

X = NonNegative (5)

M = MoProblems ()

Print m. x # womp

---------------------------------------------------------------------------

TypeError Traceback (most recent call last)

　　 In ()

3

4 m = MoProblems ()

----> 5 print m. x # womp

　　 In _ get _ (self, instance, owner)

9 # instance = x

10 # owner = type (x)

---> 11 return self. data. get (instance, self. default)

12

13 def _ set _ (self, instance, value ):

TypeError: unhashable type: 'moproblems'

Because MoProblems instances (list subclasses) cannot be hashed, they cannot be keys used as data dictionaries for MoProblems. x. There are some ways to avoid this problem, but they are not perfect. The best way is to add tags to your descriptor.

The Code is as follows:

Class Descriptor (object ):

Def _ init _ (self, label ):

Self. label = label

Def _ get _ (self, instance, owner ):

Print '_ get _', instance, owner

Return instance. _ dict _. get (self. label)

Def _ set _ (self, instance, value ):

Print '_ set __'

Instance. _ dict _ [self. label] = value

Class Foo (list ):

X = Descriptor ('x ')

Y = Descriptor ('y ')

F = Foo ()

F. x = 5

Print f. x

_ Set __

_ Get _ []

5

This method depends on the parsing sequence (MRO) of the Python method ). We add a tag name to each Descriptor in Foo. The name is the same as the variable name assigned to the Descriptor, for example, x = Descriptor ('x '). Then, the descriptor stores instance-specific data in f. _ dict _ ['X. This dictionary entry is usually the return value given by Python when we request f. x. However, because Foo. x is a descriptor, Python cannot use f. _ dict _ ['X'] normally, but the descriptor can store data securely here. Remember not to add tags to this descriptor elsewhere.

The Code is as follows:

Class Foo (object ):

X = Descriptor ('y ')

F = Foo ()

F. x = 5

Print f. x

F. y = 4 # oh no!

Print f. x

_ Set __

_ Get _ <__main _. Foo object at 0x000032c810>

5

_ Get _ <__main _. Foo object at 0x000032c810>

4

I don't like this method, because such code is fragile and has many nuances. However, this method is indeed common and can be used in owner classes that do not support hashing. David Beazley used this method in his book.

Use a tag-specific descriptor in the meta-class

Because the Tag Name of the descriptor is the same as the variable name assigned to it, someone uses the Meta class to automatically process this bookkeeping task.

The Code is as follows:

Class Descriptor (object ):

Def _ init _ (self ):

# Notice we aren't setting the label here

Self. label = None

Def _ get _ (self, instance, owner ):

Print '_ get _. Label = % s' % self. label

Return instance. _ dict _. get (self. label, None)

Def _ set _ (self, instance, value ):

Print '_ set __'

Instance. _ dict _ [self. label] = value

Class DescriptorOwner (type ):

Def _ new _ (cls, name, bases, attrs ):

# Find all descriptors, auto-set their labels

For n, v in attrs. items ():

If isinstance (v, Descriptor ):

V. label = n

Return super (DescriptorOwner, cls). _ new _ (cls, name, bases, attrs)

Class Foo (object ):

_ Metaclass _ = DescriptorOwner

X = Descriptor ()

F = Foo ()

F. x = 10

Print f. x

_ Set __

_ Get _. Label = x

10

I will not explain the details about the meta-class-references David Beazley has explained clearly in his article. It should be noted that the Meta class automatically adds tags to the descriptor and matches the variable name assigned to the descriptor.

Although this solves the inconsistency between the descriptor label and the variable name, it introduces a complex metadata class. Although I doubt it, you can determine whether it is worthwhile.

Access descriptor Method

Descriptors are just classes. Maybe you want to add some methods for them. For example, descriptor is a good method for property callback. For example, we want to notify us immediately when the status of a part of a class changes. Most of the following code is used to do this:

The Code is as follows:

Class CallbackProperty (object ):

"A property that will alert observers when upon updates """

Def _ init _ (self, default = None ):

Self. data = WeakKeyDictionary ()

Self. default = default

Self. callbacks = WeakKeyDictionary ()

Def _ get _ (self, instance, owner ):

Return self. data. get (instance, self. default)

Def _ set _ (self, instance, value ):

For callback in self. callbacks. get (instance, []):

# Alert callback function of new value

Callback (value)

Self. data [instance] = value

Def add_callback (self, instance, callback ):

"Add a new function to call everytime the descriptor updates """

# But how do we get here ?!?!

If instance not in self. callbacks:

Self. callbacks [instance] = []

Self. callbacks [instance]. append (callback)

Class BankAccount (object ):

Balance = CallbackProperty (0)

Def low_balance_warning (value ):

If value: <100:

Print "You are poor"

Ba = BankAccount ()

# Will not work -- try it

# Ba. balance. add_callback (ba, low_balance_warning)

This is an attractive mode-we can customize callback functions to respond to state changes in a class, and there is no need to modify the code of this class. This is really a matter of relief. Now, all we need to do is call ba. balance. add_callback (ba, low_balance_warning) so that low_balance_warning will be called every time the balance changes.

But how do we do it? When we try to access them, the descriptor always calls _ get __. It's like the add_callback method is not accessible! In fact, the key lies in the use of a special situation, that is, when accessing from the class level, the first parameter of the __get _ method is None.

The Code is as follows:

Class CallbackProperty (object ):

"A property that will alert observers when upon updates """

Def _ init _ (self, default = None ):

Self. data = WeakKeyDictionary ()

Self. default = default

Self. callbacks = WeakKeyDictionary ()

Def _ get _ (self, instance, owner ):

If instance is None:

Return self

Return self. data. get (instance, self. default)

Def _ set _ (self, instance, value ):

For callback in self. callbacks. get (instance, []):

# Alert callback function of new value

Callback (value)

Self. data [instance] = value

Def add_callback (self, instance, callback ):

"" Add a new function to call everytime the descriptor within instance updates """

If instance not in self. callbacks:

Self. callbacks [instance] = []

Self. callbacks [instance]. append (callback)

Class BankAccount (object ):

Balance = CallbackProperty (0)

Def low_balance_warning (value ):

If value: <100:

Print "You are now poor"

Ba = BankAccount ()

BankAccount. balance. add_callback (ba, low_balance_warning)

Ba. balance = 5000

Print "Balance is % s" % ba. balance

Ba. balance = 99

Print "Balance is % s" % ba. balance

Balance is 5000

You are now poor

Balance is 99

Conclusion

I hope you have an understanding of what descriptors are and their applicable scenarios. Cool!