Deep understanding of metaclass in Python)

This article mainly introduces metaclass in Python. it is a wonderful foreign language translation, I explained that classes are also objects, dynamically creating classes, and what is meta-classes. if you need them, refer to this article: this is a hot post on Stack overflow. The questioner claimed that he had mastered various concepts in Python OOP programming, but he always felt that metaclass was hard to understand. He knows that this is definitely related to introspection, but he still does not understand it. I hope you can give some practical examples and code snippets to help understand it, and under what circumstances do you need to perform metaprogramming. So e-satis gave a magical reply, which won 985 points of approval. some people commented that this reply should be added to the official Python document. E-satis earned 64271 points in Stack Overflow. The following is a wonderful response (note: very long)

Class is also an object

Before you understand the meta class, you must first master the class in Python. The concept of classes in Python draws on Smalltalk, which is somewhat odd. In most programming languages, classes are a set of code segments used to describe how to generate an object. This is still true in Python:

The code is as follows:

>>> Class ObjectCreator (object ):
... Pass
...
>>> My_object = ObjectCreator ()
>>> Print my_object
<__ Main _. ObjectCreator object at 0x8974f2c>

However, there are far more classes in Python. Class is also an object. Yes, that's right. it's an object. As long as you use the keyword class, the Python interpreter creates an object during execution. The following code snippet:

The code is as follows:

>>> Class ObjectCreator (object ):
... Pass
...

An object will be created in the memory and its name is ObjectCreator. This object (class) itself has the ability to create an object (class instance), which is why it is a class. However, its essence is still an object, so you can perform the following operations on it:

1. you can assign a value to a variable.
2. you can copy it.
3. you can add attributes for it.
4. you can pass it as a function parameter.

The following is an example:

The code is as follows:

>>> Print ObjectCreator # you can print a class because it is actually an object

>>> Def echo (o ):
... Print o
...
>>> Echo (ObjectCreator) # you can pass the class as a parameter to the function.

>>> Print hasattr (ObjectCreator, 'New _ attribute ')
Fasle
>>> ObjectCreator. new_attribute = 'foo' # you can add attributes to the class.
>>> Print hasattr (ObjectCreator, 'New _ attribute ')
True
>>> Print ObjectCreator. new_attribute
Foo
>>> ObjectCreatorMirror = ObjectCreator # you can assign a class to a variable.
>>> Print ObjectCreatorMirror ()
<__ Main _. ObjectCreator object at 0x8997b4c>

Dynamically create a class

Because classes are also objects, you can dynamically create them at runtime, just like any other object. First, you can create a class in the function and use the class keyword.

The code is as follows:

>>> Def choose_class (name ):
... If name = 'foo ':
... Class Foo (object ):
... Pass
... Return Foo # The returned class is not an instance of the class.
... Else:
... Class Bar (object ):
... Pass
... Return Bar
...
>>> MyClass = choose_class ('Foo ')
>>> Print MyClass # the function returns a class, not an instance of the class.

>>> Print MyClass () # you can use this class to create a class instance, that is, an object
<__ Main _. Foo object at 0x89c6d4c>

But this is not dynamic enough, because you still need to write the entire class code by yourself. Classes are also objects, so they must be generated by something. When you use the class keyword, the Python interpreter automatically creates this object. But like most things in Python, Python still provides you with a method for manual processing. Do you still remember the built-in function type? This old but powerful function allows you to know the type of an object, just like this:

The code is as follows:

>>> Print type (1)

>>> Print type ("1 ")

>>> Print type (ObjectCreator)

>>> Print type (ObjectCreator ())

Here, type has a completely different capability, and it can also dynamically create classes. Type can take the description of a class as a parameter and return a class. (I know that it is silly to use two completely different functions based on different input parameters, but this is to maintain backward compatibility in Python)

Type can work like this:

The code is as follows:

Type (class name, the tuples of the parent class (can be blank for inheritance), including the attribute Dictionary (name and value ))

For example, the following code:

The code is as follows:

>>> Class MyShinyClass (object ):
... Pass

You can create it manually as follows:

The code is as follows:

>>> MyShinyClass = type ('myshinyclass', (), {}) # return a class object
>>> Print MyShinyClass

>>> Print MyShinyClass () # create an instance of this class
<__Main _. MyShinyClass object at 0x8997cec>

We will find that we use "MyShinyClass" as the class name, and we can also use it as a variable to reference the class. Classes and variables are different. there is no reason to make things complicated here.

Type accepts a dictionary to define attributes for the class. therefore

The code is as follows:

>>> Class Foo (object ):
... Bar = True

It can be translated:

The code is as follows:

>>> Foo = type ('foo', (), {'bar': True })

In addition, Foo can be used as a common class:

The code is as follows:

>>> Print Foo

>>> Print Foo. bar
True
>>> F = Foo ()
>>> Print f
<__ Main _. Foo object at 0x8a9b84c>
>>> Print f. bar
True

Of course, you can inherit from this class, so the following code:

The code is as follows:

>>> Class FooChild (Foo ):
... Pass

You can write it as follows:

The code is as follows:

>>> FooChild = type ('foochild ', (Foo ,),{})
>>> Print FooChild

>>> Print FooChild. bar # The bar attribute is inherited by Foo.
True

In the end, you will want to add methods to your class. You only need to define a function with an appropriate signature and assign values as attributes.

The code is as follows:

>>> Def echo_bar (self ):
... Print self. bar
...
>>> FooChild = type ('foochild ', (Foo,), {'echo _ bar': echo_bar })
>>> Hasattr (Foo, 'echo _ bar ')
False
>>> Hasattr (FooChild, 'echo _ bar ')
True
>>> My_foo = FooChild ()
>>> My_foo.echo_bar ()
True

As you can see, in Python, classes are also objects, and you can dynamically create classes. This is what Python does behind the scenes when you use the keyword class, which is implemented through the meta class.

What is metadata?

A metadatabase is a "thing" used to create a class ". You create a class to create instance objects of the class, right? But we have learned that classes in Python are also objects. Well, the meta class is used to create these classes (objects), and the meta class is the class of the class. you can understand it as follows:

The code is as follows:

MyClass = MetaClass ()
MyObject = MyClass ()

You have seen that type allows you to do as follows:

The code is as follows:

MyClass = type ('myclass ',(),{})

This is because the function type is actually a metadata class. Type is the metadata used by Python to create all classes. Now you want to know why all types are in lower case instead of type? Well, I guess this is to maintain consistency with str. str is the class used to create string objects, and int is the class used to create integer objects. Type is the class for creating class objects. You can see this by checking the class attributes. All things in Python. Note that I mean everything -- all objects. This includes integers, strings, functions, and classes. They are all objects, and they are all created from a class.

The code is as follows:

>>> Age = 35
>>> Age. _ class __

>>> Name = 'Bob'
>>> Name. _ class __

>>> Def foo (): pass
>>> Foo. _ class __

>>> Class Bar (object): pass
>>> B = Bar ()
>>> B. _ class __

Now, what is the class attribute of any class?

The code is as follows:

>>> A. _ class _. _ class __

>>> Age. _ class _. _ class __

>>> Foo. _ class _. _ class __

>>> B. _ class _. _ class __

Therefore, meta-classes are what creates objects such as classes. If you like, you can call the metadatabase class "class Factory" (do not mix it with the factory class: D) type is the built-in Metadatabase class of Python. of course, you can also create your own metadata.

Metaclass attributes

You can add the metaclass attribute to a class.

The code is as follows:

Class Foo (object ):
_ Metaclass _ = something...
[…]

If you do this, Python will use the meta class to create the class Foo. Be careful. Here are some tips. First, write down class Foo (object), but the class object Foo has not been created in the memory. Python looks for the metaclass attribute in the class definition. if the metaclass attribute is found, Python will use it to create the class Foo. if not found, the built-in type will be used to create the class. Read the following passage several times. When you write the following code:

The code is as follows:

Class Foo (Bar ):
Pass

Python performs the following operations:

Is metaclass in Foo? If yes, Python will create a class object named Foo through metaclass in the memory (I am talking about class objects, please keep up with my ideas ). If Python does not find metaclass, it will continue to look for the metaclass attribute in Bar (parent class) and try the same operation as before. If Python cannot find metaclass in any parent class, it searches for metaclass at the module level and tries the same operation. If metaclass still cannot be found, Python will use the built-in type to create this class object.

Now the question is, what code can you place in metaclass? The answer is: you can create a class. So what can be used to create a class? Type, or any stuff that uses type or subclass type.

Custom metadata
The main purpose of a metadatabase is to automatically change the class when a class is created. Generally, you will do this for the API, and you want to create a class that conforms to the current context. Suppose a silly example. you decide that the attributes of all classes in your module should be in upper case. There are several ways to do this, but one of them is to set metaclass at the module level. In this way, all classes in this module will be created through this Metadatabase. we only need to tell the metadatabase to change all attributes to uppercase.

Fortunately, metaclass can be called at will, and it does not need to be a formal class (I know that something with 'class' in some names does not need to be a class, it is helpful to draw and draw images ). So here we will start with a simple function as an example.

The code is as follows:

# The metadata class automatically uses the parameter you normally pass to 'type' as its own parameter.
Def upper_attr (future_class_name, future_class_parents, future_class_attr ):
''' A class object is returned, and all attributes are converted to uppercase '''
# Select all attributes not starting '_'
Attrs = (name, value) for name, value in future_class_attr.items () if not name. startswith ('__'))

# Convert them to uppercase
Uppercase_attr = dict (name. upper (), value) for name, value in attrs)

# Create a class object using 'type'
Return type (future_class_name, future_class_parents, uppercase_attr)

_ Metaclass _ = upper_attr # this will affect all classes in this module

Class Foo (object ):
# We can also define _ metaclass __here so that it will only act on this class.
Bar = 'bip'

Print hasattr (Foo, 'bar ')
# Output: False
Print hasattr (Foo, 'bar ')
# Output: True

F = Foo ()
Print f. BAR
# Output: 'bip'

Now let's do it again. this time, we use a real class as the meta class.

The code is as follows:

# Remember, 'type' is actually a class, just like 'str' and 'int'
# Therefore, you can inherit from type
Class UpperAttrMetaClass (type ):
# _ New _ is a special method called before _ init _
# _ New _ is the method used to create an object and return it.
# _ Init _ is used to initialize the passed parameters to the object.
# You rarely use _ new __unless you want to control the creation of objects
# Here, the created object is a class, and we want to be able to customize it, so here we rewrite _ new __
# If you want to, you can also do something in _ init _.
# Some advanced operations involve the special _ call _ method, but we do not need
Def _ new _ (upperattr_metaclass, future_class_name, future_class_parents, future_class_attr ):
Attrs = (name, value) for name, value in future_class_attr.items () if not name. startswith ('__'))
Uppercase_attr = dict (name. upper (), value) for name, value in attrs)
Return type (future_class_name, future_class_parents, uppercase_attr)

However, this method is not OOP. We called type directly, and we didn't rewrite the new method of the parent class. Now let's handle it like this:

The code is as follows:

Class UpperAttrMetaclass (type ):
Def _ new _ (upperattr_metaclass, future_class_name, future_class_parents, future_class_attr ):
Attrs = (name, value) for name, value in future_class_attr.items () if not name. startswith ('__'))
Uppercase_attr = dict (name. upper (), value) for name, value in attrs)

# Reuse the type. _ new _ method
# This is basic OOP programming, with no magic
Return type. _ new _ (upperattr_metaclass, future_class_name, future_class_parents, uppercase_attr)

You may have noticed that there is an additional parameter upperattr_metaclass, which is nothing special. The first parameter of a class method always represents the current instance, just like the self parameter in a common class method. Of course, for sake of clarity, I have a long name here. But like self, all parameters have their traditional names. Therefore, in the real product code, a Metaclass should be like this:

The code is as follows:

Class UpperAttrMetaclass (type ):
Def _ new _ (cls, name, bases, dct ):
Attrs = (name, value) for name, value in dct. items () if not name. startswith ('__')
Uppercase_attr = dict (name. upper (), value) for name, value in attrs)
Return type. _ new _ (cls, name, bases, uppercase_attr)

If the super method is used, we can also make it clearer, which will alleviate inheritance (yes, you can own the meta class, inherit from the meta class, and inherit from the type)

The code is as follows:

Class UpperAttrMetaclass (type ):
Def _ new _ (cls, name, bases, dct ):
Attrs = (name, value) for name, value in dct. items () if not name. startswith ('__'))
Uppercase_attr = dict (name. upper (), value) for name, value in attrs)
Return super (UpperAttrMetaclass, cls). _ new _ (cls, name, bases, uppercase_attr)

In this case, there is really nothing to say about the meta class. The code used for the meta-class is complicated. the reason behind this is not that the meta-class itself, but that you usually use the meta-class to do obscure things and rely on introspection, control inheritance. Indeed, it is particularly useful to use the meta class to create "dark magic", so it will make some complicated things. However, for the metadata itself, they are actually very simple:

1. create an interception class
2. modify the class
3. return the modified class

Why should we use metaclass classes instead of functions?

Since metaclass can accept any Callable object, why should we use classes? Obviously, it is more complicated to use classes? There are several reasons:

1. the intention will be clearer. When you read UpperAttrMetaclass (type), you know what will happen next.
2. you can use OOP programming. The metadata class can inherit from the metadata class and rewrite the method of the parent class. Metadata can even be used.
3. you can better organize the code. When you use a metadata class, it will not be a simple scenario as I mentioned above. it is usually aimed at complicated problems. It is helpful to sum up multiple methods into a class and make the code easier to read.
4. you can use special methods such as new, init, and call. They help you process different tasks. Even if you can delete everything in new, some people still feel more comfortable using init.
5. Wow, the name of this item is metaclass. it must be not a good class. please be careful!

Why do we need to use a Metadatabase?

Now back to our big topic, why are you using this error-prone and obscure feature? Well, in general, you can't use it at all:

The meta class is the magic of depth. 99% of users do not have to worry about it. If you want to determine whether or not you need to use the metadatabase, you do not need it. Those who actually use the metadatabase know exactly what they need to do and do not need to explain why they need the metadatabase. -- Tim Peters, Python leader

The primary purpose of a metadatabase is to create an API. A typical example is Django ORM. It allows you to define it like this:

The code is as follows:

Class Person (models. Model ):
Name = models. CharField (max_length = 30)
Age = models. IntegerField ()

But if you do this:

The code is as follows:

Guy = Person (name = 'Bob', age = '35 ')
Print guy. age

This does not return an IntegerField object, but returns an int, or even retrieves data directly from the database. This is possible because models. Model defines metaclass and uses some magic to transform the simple Person class you just defined into a complex hook for the database. The Django framework simplifies these complex things by exposing a simple meta-class API and re-creates code through this API to complete the real work behind the scenes.

Conclusion

First, you know that a class is actually an object that can create a class instance. Well, in fact, classes are also instances, and of course they are metadata instances.

The code is as follows:

>>> Class Foo (object): pass
>>> Id (Foo)
142630324

Everything in Python is an object. they are either class instances or meta-class instances, except type. Type is actually its own meta-class. in a pure Python environment, this is not what you can do. it is achieved through some small means at the implementation level. Second, metadata is complex. For a very simple class, you may not want to modify the class by using the meta class. You can use two other technologies to modify the class:

The code is as follows:

Monkey patching
Class decorators

When you need to dynamically modify the class, you 'd better use the above two technologies in 99% of the time. Of course, you don't need to modify classes dynamically in 99% of the time.