Deep understanding of metaclass in Python)

Source: http://blog.jobbole.com/21351/

Http://stackoverflow.com/questions/100003/what-is-a-metaclass-in-python:

:This is a hot post on Stack overflow. The questioner claimed that he had mastered various concepts in Python OOP programming, but he always felt that metaclass was hard to understand. He knows that this is definitely related to introspection, but he still does not understand it. I hope you can give some practical examples and code snippets to help understand it, and under what circumstances do you need to perform metaprogramming. So e-satis gave a magical reply, which won 985 points of approval. Some people commented that this reply should be added to the official Python document. E-satis earned 64271 points in Stack Overflow. The following is a wonderful response (Note: Very Long)

Class is also an object

Before you understand the Meta class, you must first master the class in Python. The concept of classes in Python draws on Smalltalk, which is somewhat odd. In most programming languages, classes are a set of code segments used to describe how to generate an object. This is still true in Python:

>>> class ObjectCreator(object):…       pass…>>> my_object = ObjectCreator()>>> print my_object<__main__.ObjectCreator object at 0x8974f2c>

However, there are far more classes in Python. Class is also an object. Yes, that's right. It's an object. As long as you use the keyword class, the Python interpreter creates an object during execution. The following code snippet:

>>> class ObjectCreator(object):…       pass…

An object will be created in the memory and its name is ObjectCreator. This object (class) itself has the ability to create an object (class instance), which is why it is a class. However, its essence is still an object, so you can perform the following operations on it:

1) You can assign a value to a variable.

2) You can copy it.

3) You can add attributes for it.

4) you can pass it as a function parameter.

The following is an example:

>>> Print objectcreator # You can print a class because it is actually an object <class '_ main _. objectcreator' >>>> def echo (o ):... Print o... >>> Echo (objectcreator) # You can pass the class as a parameter to the function <class '_ main __. objectcreator '>>> print hasattr (objectcreator, 'new _ attribute') fasle >>> objectcreator. new_attribute = 'foo' # You can add attributes to the class >>> print hasattr (objectcreator, 'new _ attribute') True >>> print objectcreator. new_attributefoo >>> objectcreatormirror = objectcreator # You can assign a class value to a variable >>> print objectcreatormirror () <__ main __. objectcreator object at 0x8997b4c>

Dynamically create a class

Because classes are also objects, You can dynamically create them at runtime, just like any other object. First, you can create a class in the function and use the class keyword.

>>> Def choose_class (name ):... If name = 'foo ':... Class Foo (object ):... Pass... Return Foo # The returned class is not an instance of the class... Else :... Class Bar (object ):... Pass... Return Bar…… >>> MyClass = choose_class ('foo') >>> print MyClass # The function returns a class, not an instance of the class <class '_ main __'. foo >>>> print MyClass () # You can use this class to create a class instance, that is, the object <__ main __. foo object at 0x89c6d4c>

But this is not dynamic enough, because you still need to write the entire class code by yourself. Classes are also objects, so they must be generated by something. When you use the class keyword, the Python interpreter automatically creates this object. But like most things in Python, Python still provides you with a method for manual processing. Do you still remember the built-in function type? This old but powerful function allows you to know the type of an object, just like this:

>>> print type(1)<type 'int'>>>> print type("1")<type 'str'>>>> print type(ObjectCreator)<type 'type'>>>> print type(ObjectCreator())<class '__main__.ObjectCreator'>

Here, type has a completely different capability, and it can also dynamically create classes. Type can take the description of a class as a parameter and return a class. (I know that it is silly to use two completely different functions based on different input parameters, but this is to maintain backward compatibility in Python)

Type can work like this:

Type (class name, The tuples of the parent class (can be blank for inheritance), including the attribute Dictionary (name and value ))

For example, the following code:

>>> class MyShinyClass(object):…       pass

You can create it manually as follows:

>>> MyShinyClass = type ('myshinyclass', (), {}) # returns a class Object >>> print MyShinyClass <class '_ main __. myShinyClass '>>>> print MyShinyClass () # create an instance of this class <__ main __. myShinyClass object at 0x8997cec>

You will find that we use "MyShinyClass" as the class name, and we can also use it as a variable to reference the class. Classes and variables are different. There is no reason to make things complicated here.

Type accepts a dictionary to define attributes for the class. Therefore

>>> class Foo(object):…       bar = True

It can be translated:

>>> Foo = type('Foo', (), {'bar':True})

In addition, Foo can be used as a common class:

>>> print Foo<class '__main__.Foo'>>>> print Foo.barTrue>>> f = Foo()>>> print f<__main__.Foo object at 0x8a9b84c>>>> print f.barTrue

Of course, you can inherit from this class, so the following code:

>>> class FooChild(Foo):…       pass

You can write it as follows:

>>> FooChild = type ('foochild ', (Foo,), {}) >>> print FooChild <class' _ main __. fooChild '>>>> print FooChild. bar # The bar property is inherited by Foo and True

In the end, you will want to add methods to your class. You only need to define a function with an appropriate signature and assign values as attributes.

>>> def echo_bar(self):…       print self.bar…>>> FooChild = type('FooChild', (Foo,), {'echo_bar': echo_bar})>>> hasattr(Foo, 'echo_bar')False>>> hasattr(FooChild, 'echo_bar')True>>> my_foo = FooChild()>>> my_foo.echo_bar()True

As you can see, in Python, classes are also objects, and you can dynamically create classes. This is what Python does behind the scenes when you use the keyword class, which is implemented through the Meta class.

 

What is the Meta class? (the topic is reached)

A metadatabase is a "thing" used to create a class ". You create a class to create instance objects of the class, right? But we have learned that classes in Python are also objects. Well, the Meta class is used to create these classes (objects), and the Meta class is the class of the class. You can understand it as follows:

MyClass = MetaClass()MyObject = MyClass()

You have seen that type allows you to do as follows:

MyClass = type('MyClass', (), {})

This is because the function type is actually a metadata class. Type is the metadata used by Python to create all classes. Now you want to know why all types are in lower case instead of type? Well, I guess this is to maintain consistency with str. str is the class used to create string objects, and int Is the class used to create integer objects. Type is the class for creating class objects. You can see this by checking the _ class _ attribute. All things in Python. Note that I mean everything -- all objects. This includes integers, strings, functions, and classes. They are all objects, and they are all created from a class.

>>> age = 35>>> age.__class__<type 'int'>>>> name = 'bob'>>> name.__class__<type 'str'>>>> def foo(): pass>>>foo.__class__<type 'function'>>>> class Bar(object): pass>>> b = Bar()>>> b.__class__<class '__main__.Bar'>

What is the _ class _ attribute of any _ class?

>>> a.__class__.__class__<type 'type'>>>> age.__class__.__class__<type 'type'>>>> foo.__class__.__class__<type 'type'>>>> b.__class__.__class__<type 'type'>

Therefore, meta-classes are what creates objects such as classes. If you like, you can call the metadatabase class "class factory" (do not mix it with the factory class: D) type is the built-in metadatabase class of Python. Of course, you can also create your own metadata.

_ Metaclass _ attributes

You can add the _ metaclass _ attribute to a class.

class Foo(object):    __metaclass__ = something…[…]

If you do this, Python will use the Meta class to create the class Foo. Be careful. Here are some tips. First, write down class Foo (object), but the class object Foo has not been created in the memory. Python will search for the _ metaclass _ attribute in the class definition. if it finds the attribute, Python will use it to create the class Foo. If not, the built-in type is used to create this class. Read the following passage several times. When you write the following code:

class Foo(Bar):    pass

Python performs the following operations:

Is the _ metaclass _ attribute in Foo? If yes, python will create a class object named Foo through _ metaclass _ in the memory (I am talking about class objects, please keep up with my ideas ). If Python does not find _ metaclass __, it will continue to search for the _ metaclass _ attribute in bar (parent class) and try the same operations as the previous one. If Python cannot find _ metaclass __in any parent class, it searches for _ metaclass __at the module level and tries the same operation. If _ metaclass __is still not found, python will use the built-in type to create this class object.

Now the question is, what code can you place in _ metaclass? The answer is: you can create a class. So what can be used to create a class? Type, or any stuff that uses type or subclass type.

 

Custom metadata

The main purpose of a metadatabase is to automatically change the class when a class is created. Generally, you will do this for the API, and you want to create a class that conforms to the current context. Suppose a silly example. You decide that the attributes of all classes in your module should be in upper case. There are several ways to do this, but one of them is to set _ metaclass _ at the module level __. In this way, all classes in this module will be created through this metadatabase. We only need to tell the metadatabase to change all attributes to uppercase.

Fortunately, __metaclass _ can be called at will, and it does not need to be a formal class (I know, some things with 'class' in some names do not need to be a class. It is helpful to draw a picture ). So here we will start with a simple function as an example.

# The meta-class will automatically pass the parameter that you normally pass to 'type' as its own parameter into def upper_attr (future_class_name, future_class_parents, future_class_attr): ''' to return a class object, convert all attributes to uppercase ''' # select all attributes not starting with '_' attrs = (name, value) for name, value in future_class_attr.items () if not name. startswith ('_') # convert them into upper-case formats uppercase_attr = dict (name. upper (), value) for name, value in attrs) # Use 'type' to create a return type (future_class_name, future_class_parents, uppercase_attr) for class objects) _ metaclass _ = upper_attr # This will apply to all classes in this module Foo (object): # We can also define _ metaclass __, this will only apply to bar = 'bip' print hasattr (Foo, 'bar') # output: Falseprint hasattr (Foo, 'bar') # output: true f = Foo () print f. BAR # output: 'bip'

Now let's do it again. This time, we use a real class as the Meta class.

# Remember, 'type' is actually a class, just like 'str' and 'int' # So you can inherit class upperattrmetaclass (type) from type ): # _ new _ is a special method called before _ init _ # _ new _ is the method used to create an object and return it # and _ init _ _ is only used to initialize the input parameters to the object # You seldom use _ new __, unless you want to control object creation # Here, the created object is a class and we want to be able to customize it, so here we rewrite _ new _ # If you want it, you can also do some things in _ init _ # There are some advanced usage that will involve the special method of rewriting _ call, however, Def _ new _ (upperattr_metaclass, future_class_name, future_class_parents, future_class_attr): attrs = (name, value) for name, value in values () if not name. startswith ('_') uppercase_attr = dict (name. upper (), value) for name, value in attrs) return type (future_class_name, future_class_parents, uppercase_attr)

However, this method is not oop. We called type directly, and we didn't rewrite the _ new _ method of the parent class. Now let's handle it like this:

Class UpperAttrMetaclass (type): def _ new _ (Future, future_class_name, future_class_parents, future_class_attr): attrs = (name, value) for name, value in minute () if not name. startswith ('_') uppercase_attr = dict (name. upper (), value) for name, value in attrs) # reuse type. _ new _ method # This is the basic OOP programming, and there is no magic return type. _ new _ (upperattr_metaclass, future_class_name, future_class_parents, uppercase_attr)

You may have noticed that there is an additional parameter upperattr_metaclass, which is nothing special. The first parameter of a class method always represents the current instance, just like the self parameter in a common class method (this can be understood, but the truth is, __new _ the method is a static method, but you do not need to specify @ staticmethod or _ new __= staticmethod (_ new __), because the python interpreter automatically treats _ new _ as a static method and regards the first parameter as a Class Object ). Of course, for sake of clarity, I have a long name here. But like self, all parameters have their traditional names. Therefore, in the real product code, a metaclass should be like this:

class UpperAttrMetaclass(type):    def __new__(cls, name, bases, dct):        attrs = ((name, value) for name, value in dct.items() if not name.startswith('__')        uppercase_attr  = dict((name.upper(), value) for name, value in attrs)        return type.__new__(cls, name, bases, uppercase_attr)

If the super method is used, we can also make it clearer, which will alleviate inheritance (yes, you can own the Meta class, inherit from the Meta class, and inherit from the type)

class UpperAttrMetaclass(type):    def __new__(cls, name, bases, dct):        attrs = ((name, value) for name, value in dct.items() if not name.startswith('__'))        uppercase_attr = dict((name.upper(), value) for name, value in attrs)        return super(UpperAttrMetaclass, cls).__new__(cls, name, bases, uppercase_attr)

In this case, there is really nothing to say about the Meta class. The code used for the meta-class is complicated. The reason behind this is not that the meta-class itself, but that you usually use the meta-class to do obscure things and rely on introspection, control inheritance. Indeed, it is particularly useful to use the Meta class to create "Dark Magic", so it will make some complicated things. However, for the metadata itself, they are actually very simple:

1) Create an interception class

2) modify the class

3) return the modified class

 

Why should we use metaclass classes instead of functions?

Since _ metaclass _ can accept any callable object, why should we use classes? Obviously, it is more complicated to use classes? There are several reasons:

1) The intention is clearer. When you read upperattrmetaclass (type), you know what will happen next.

2) You can use OOP programming. The metadata class can inherit from the metadata class and rewrite the method of the parent class. Metadata can even be used.

3) You can better organize the code. When you use a metadata class, it will not be a simple scenario as I mentioned above. It is usually aimed at complicated problems. It is helpful to sum up multiple methods into a class and make the code easier to read.

4) You can use special methods such as _ new __, _ init _ and _ call. They help you process different tasks. Even if you can delete everything in _ new _, some people still feel more comfortable using _ init.

5) Wow, the name of this item is metaclass. It must be not a good class. Please be careful!

 

Why do we need to use a metadatabase?

Now back to our big topic, why are you using this error-prone and obscure feature? Well, in general, you can't use it at all:

"The Meta class is the magic of depth. 99% of users do not have to worry about it. If you want to determine whether or not you need to use the metadatabase, you do not need it. Those who actually use the metadatabase know exactly what they need to do and do not need to explain why they need the metadatabase ." -- Tim Peters, Python leader

The primary purpose of a metadatabase is to create an API. A typical example is Django ORM. It allows you to define it like this:

class Person(models.Model):    name = models.CharField(max_length=30)    age = models.IntegerField()

But if you do this:

guy  = Person(name='bob', age='35')print guy.age

This does not return an IntegerField object, but returns an int, or even retrieves data directly from the database. This is possible because models. Model defines _ metaclass __and uses some magic to transform the simple Person class you just defined into a complex hook for the database. The Django framework simplifies these complex things by exposing a simple meta-class API and re-creates code through this API to complete the real work behind the scenes.

 

Conclusion

First, you know that a class is actually an object that can create a class instance. Well, in fact, classes are also instances, and of course they are metadata instances.

>>>class Foo(object): pass>>> id(Foo)142630324

Everything in Python is an object. They are either class instances or meta-class instances, except type. Type is actually its own meta-class. In a pure Python environment, this is not what you can do. It is achieved through some small means at the implementation level. Second, metadata is complex. For a very simple class, you may not want to modify the class by using the Meta class. You can use two other technologies to modify the class:

1) Monkey patching

2) class decorators

When you need to dynamically modify the class, you 'd better use the above two technologies in 99% of the time. Of course, you don't need to modify classes dynamically in 99% of the time.