God-level programmers take you through two sentences to fully grasp the most difficult knowledge of Python-the meta-class!

Do not be intimidated by the so-called "meta-class is a feature not used by Python programmers 99%". Because every Chinese is a natural meta-class user

Learn to understand the Meta class, you only need to know two words:

Daosh One, life two, two three, Sansheng everything

Who am I? Where did I come from? Where am I going to go?

In the Python world, there is an eternal word, that is "type", please remember in the mind, type is the Tao. The vast expanse of the Python biosphere is created by type. Before you share, small make recommend a very good exchange treasure, inside are a group of love and learning Python's small partners, big thousands of, a variety of people have, especially like to see the atmosphere of this kind of people together to solve the problem, the group of information also uploaded a lot, all kinds of Daniel to solve the problem of small white , this Python group: 330637182 Welcome to join us in a conversation, progress together and master this Python language as early as possible.

Daosh One, life two, two three, Sansheng everything.

The Tao is the type

One is metaclass (meta class, or class builder)

Second is class (class, or instance builder)

Three is instance (example)

Everything is an example of the various properties and methods that we normally use when Python is called.

Tao and one are the propositions we are discussing today, and two or three, and all things, are the classes, instances, attributes, and methods that we often use, with the example of Hello World:

The creator can directly create a single person, but it is a slave labor. The creator will first create the "human" species, and then batch create specific individuals. And the three eternal propositions, have been passed down.

"Tao" can directly give birth to "two", but it will be Mr. "one", and then the production of "two" in bulk.

Type can directly generate classes (class), but can also be Mr. Narimoto Class (Metaclass), and then use the meta-class to bulk-customize classes (Class).

Meta class--Daosh one, life two

In general, the meta-class is named suffix Metalass. Imagine that we need an automatic greeting of the meta-class, it's the class method, sometimes need to say_hello, sometimes need to say_hi, sometimes need to say_sayolala, sometimes need say_nihao.

If every built-in say_xxx needs to be declared in the class, it would be a terrible slave labor! Instead, use a meta-class to solve the problem.

Here is a meta-class code that creates a special "hello":

To! Together according to Daosh one or one born two or two three or three living things guidelines, into the life cycle of the meta-class it!

Note: The class created by the Meta class, the first parameter is the parent class, the second parameter is Metaclass

Ordinary people are not born to speak, but some are born to say hello, "hello", "Sayolala", this is the power of talent. It will save us countless problems with object-oriented programming.

Now, keeping the meta class intact, we can also continue to create Sayolala, the Nihao class, as follows:

That's great! Have you already experienced the pleasure of the creator in learning this place?

Everything in Python is in the hands of the world.

Young Creator, please join me in creating a new world.

We choose two domains, one is the core idea of Django, "Object Relational Mapping", i.e. object-relational mapping, or ORM.

This is a big pain in Django, but after learning the meta-class, everything becomes clear. Your understanding of Django will be a higher level!

Another area is the reptile domain (hacker domain), an automatic search network of available agents, and then the IP to break other people's anti-crawler restrictions.

These two skills are very useful and very fun!

Challenge One: Create an ORM from a meta class

Get ready to work, create a field class

Its role is to

The initialization of the parent class is automatically called when the Stringfield,integerfield instance is initialized.

Daosh A

It did the following several things

Create a new dictionary mapping

The properties of each class are traversed by the. Items () and their key-value pairs. If the value is a field class, the key value is printed and the pair of key values is bound to the mapping dictionary.

Delete the property that just passed in the value of the field class.

Create a dedicated __mappings__ property that holds the dictionary mapping.

Create a dedicated __table__ property that holds the name of the passed-in class.

Life Two

Integerfield (' id ') will automatically resolve to:

Model. SetAttr (self, ' id ', Integerfield (' id '))

Because Intergerfield (' id ') is an instance of the subclass of field and automatically triggers the __new__ of the Meta class, the Intergerfield (' ID ') is stored in __mappings__ and the key-value pair is deleted.

Three or three living things

When you initialize an instance and call the Save () method

U = User (id=12345, name= ' Batman ', email= '[email protected]', password= ' Iamback ') U.save ()

This completes the two-born three process:

Call model first. setattr, loading a key value into a private object

Then call the "talent" of the meta-class, Modelmetaclass. New, the private object in model, as long as it is an instance of field, is automatically saved to U. mappings.

The next step is to complete the process of Sansheng everything:

The operation is stored via the U.save () simulation database. Here we just do a little bit of traversing the __mappings__ operation, virtual SQL and printing, in the real case by entering SQL statements and database to run.

Here, we use the request package, the source of Baidu crawled out.

Try to catch Baidu

Stick This section behind the get_page.py and try to delete it.

Let's get to the point: Using meta-classes to bulk crawl agents

Bulk processing of crawl agents

Daosh: In the __new__ of the meta-class, four things are done:

Push the name of the class method that starts with "Crawl_" into Proxygetter. Crawlname

The class method that starts with "Crawl_" pushes itself into proxygetter. Crawlfunc

Calculate the number of class methods that begin with "Crawl_"

Remove all class methods that begin with "Crawl_"

What do you think? Is it very similar to the __mappings__ process that created the ORM before?

Life Two: The class defines a method for fetching page elements using Pyquery

All the agents displayed on the page were crawled from three free proxy sites.

If you are unfamiliar with yield usage, you can view:

Two students three: Create an Instance object crawler

Slightly

Sansheng everything: traversing every __crawlfunc__

On the proxygetter.__crawlname__, get the URL name that can be crawled.

Trigger class Method Proxygetter.get_raw_proxies (site)

Traverse Proxygetter. Crawlfunc, if the method name and URL name are the same, execute this method

The agent that obtains each URL is aggregated into the array output.

So... How to use the bulk agent, impact on other people's website, take someone else's password, crazy hair ads water stickers, regularly harass customers? Uh! What are you thinking? These self-realizations! If you do not understand, please listen to tell!

The young creator, the tool that created the world is already in your hands, please make it the power to the extreme!

Remember the formula for waving the tool:

Daosh One, life two, two three, Sansheng everything

Who I am, where I come from, where I'm going

Before you understand a meta-class, you need to master the classes in Python, and the concepts of classes in Python are similar to those in Smalltalk.

In most languages, a class is a snippet of code that describes how to create an object, which is also true in Python

What about Python-what is a meta-class? Daosh One, life two, two three, sansheng all things Yuan

Learn to understand the Meta class, you only need to know two words:

Daosh One, life two, two three, Sansheng everything

Who am I? Where did I come from? Where am I going to go?

In the Python world, there is an eternal word, that is "type", please remember in the mind, type is the Tao. The vast expanse of the Python biosphere is created by type.

Daosh One, life two, two three, Sansheng everything.

The Tao is the type

One is metaclass (meta class, or class builder)

Second is class (class, or instance builder)

Three is instance (example)

Everything is an example of the various properties and methods that we normally use when Python is called.

Tao and one are the propositions we are discussing today, and two or three, and all things, are the classes, instances, attributes, and methods that we often use, with the example of Hello World:

Python

1

2

3

4

5

6

7

8

9

10

11

Create a Hello class with attributes Say_hello the origin of----two

Classhello ():

Defsay_hello (self,name= ' World '):

Print (' Hello,%s. '%name)

Create an instance from the Hello class Hello----II three

Hello=hello ()

Using the Hello call method Say_hello----Sansheng Everything

Hello.say_hello ()

Output effect:

Hello,world.

This is a standard "Sansheng three, all things" process. These two steps are used from the class to the method that we can invoke.

Then we can not help but ask, where does the class come from? Go back to the first line of code.

Class Hello is actually a function of the "semantic abbreviation", just to make the code more understandable, it is another method of writing:

DEFFN (self,name= ' World '): # If we have a function called FN

Hello=type (' Hello ', (object,), Dict (SAY_HELLO=FN)) # Create a Hello class through type----mysterious "Tao", you can give everything, this time we directly from the "Tao" gave birth to "two"

This is exactly the same as the previous class Hello, you can try to create an instance and call

Create an instance from the Hello class hello----binary three, exactly the same as using the Hello call method Say_hello----Sansheng Everything, exactly the same

Hello,world.----Call result is exactly the same.

We looked back at the most exciting place, the road directly produced two:

Hello = Type (' Hello ', (object,), Dict (SAY_HELLO=FN))

This is the "Tao", the origin of the Python world, and you can marvel at it.

Pay attention to its three parameters! Coincidence the three eternal propositions of mankind: who I am, where I come from, and where I am going.

First parameter: Who am I? Here, I need a name that distinguishes everything else, and the above example names me "Hello"

Second parameter: Where do I come from?

Here, I need to know where to come from, that is my "parent class", and my parent class in the above example is a very elementary class in the "Object"--python.

The third parameter: Where do I go?

Here, we will need to call the methods and properties contained in a dictionary, and then passed in as parameters. In the above example, we have a Say_hello method wrapped in a dictionary.

It is worth noting that the three eternal propositions are all classes, all instances, and even all instance properties and methods. Rightfully, their "creator", Tao and one, that is, the type and the Meta class, also have these three parameters. But normally, the three eternal propositions of a class are not passed in as parameters, but are passed in as follows

Classhello (object) {

After class declares "Who am I" in parentheses, declares "Where am I coming from" and declares "Where am I going?"

Defsay_hello () {

}

The creator can directly create a single person, but it is a slave labor. The creator will first create the "human" species, and then batch create specific individuals. And the three eternal propositions, have been passed down.

"Tao" can directly give birth to "two", but it will be Mr. "one", and then the production of "two" in bulk.

Type can directly generate classes (class), but can also be Mr. Narimoto Class (Metaclass), and then use the meta-class to bulk-customize classes (Class).

Meta class--Daosh one, life two

In general, the meta-class is named suffix Metalass. Imagine that we need an automatic greeting of the meta-class, it's the class method, sometimes need to say_hello, sometimes need to say_hi, sometimes need to say_sayolala, sometimes need say_nihao.

If every built-in say_xxx needs to be declared in the class, it would be a terrible slave labor! Instead, use a meta-class to solve the problem.

Here is a meta-class code that creates a special "hello":

Classsaymetaclass (type):

Def__new__ (cls,name,bases,attrs):

attrs[' Say_ ' +name]=lambdaself,value,saying=name:print (saying+ ', ' +value+ '! ')

ReturnType. New (Cls,name,bases,attrs)

Remember two points:

1, the meta-class is derived from the "type", so the parent class needs to pass in the type. "Daosh one, so one must contain the Tao"

2, the operation of the meta-class is completed in __new__, its first parameter is the class that will be created, after which the parameters are three eternal propositions: who I am, where I come from, where I will go. The object it returns is also the three eternal propositions, and the next three parameters will accompany us.

In __new__, I have only one operation, which is

It creates a class method with the name of the class. For example, our class created by the meta-class called "Hello", the creation of a class called "Say_hello" method, and then the class name "Hello" as the default parameter saying, passed to the method inside. It then passes the parameters of the Hello method invocation as value and prints them out.

So, how does a meta-class go from creation to invocation?

To! Together according to Daosh one or one born two or two three or three living things guidelines, into the life cycle of the meta-class it!

12

13

14

15

16

17

18

19

Daosh: Incoming type incoming three eternal propositions: Class name, parent class, attribute create "talent" inheritance Three eternal propositions: Class name, parent class, attribute life two: Creating a class

Classhello (Object,metaclass=saymetaclass):

Pass

Two born three: Create real Lesan Everything: invoke instance method

Hello.say_hello (' world! ')

Output to

hello,world!

Note: The class created by the Meta class, the first parameter is the parent class, the second parameter is Metaclass

Ordinary people are not born to speak, but some are born to say hello, "hello", "Sayolala", this is the power of talent. It will save us countless problems with object-oriented programming.

Now, keeping the meta class intact, we can also continue to create Sayolala, the Nihao class, as follows:

Classsayolala (Object,metaclass=saymetaclass):

S=sayolala ()

S.say_sayolala (' japan! ')

Output

sayolala,japan!

can also speak Chinese

Classnihao (Object,metaclass=saymetaclass):

N=nihao ()

N.say_nihao (' Zhonghua! ')

Nihao, China!

One more small example:

Daosh A

Classlistmetaclass (type):

Talent: Bind values by the Add method

attrs[' Add ']=lambdaself,value:self.append (value)

Life Two

Classmylist (List,metaclass=listmetaclass):

Two students three

L=mylist ()

Sansheng All Things

L.add (1)

Now let's print the L

Print (L)

[1]

And the normal list does not have the Add () method

L2=list ()

L2.add (1)

Attributeerror: ' list ' Objecthas no attribute ' add '

That's great! Have you already experienced the pleasure of the creator in learning this place?

Everything in Python is in the hands of the world.

Young Creator, please join me in creating a new world.

We choose two domains, one is the core idea of Django, "Object Relational Mapping", i.e. object-relational mapping, or ORM.

This is a big pain in Django, but after learning the meta-class, everything becomes clear. Your understanding of Django will be a higher level!

Another area is the reptile domain (hacker domain), an automatic search network of available agents, and then the IP to break other people's anti-crawler restrictions.

These two skills are very useful and very fun!

Challenge One: Create an ORM from a meta class

Get ready to work, create a field class

Classfield (object):

Def__init__ (Self,name,column_type):

Self.name=name

Self.column_type=column_type

Def__str__ (self):

Return ' <%s:%s> '% (self. class. name, self.name)

Its role is to

When the field class is instantiated, it will get two parameters, name and Column_type, which will be bound to the private property of field, and "Field:xxx" will be returned if field is to be converted to a string, and XXX is the name passed in.

Preparation: Creating Stringfield and Intergerfield

Classstringfield (Field):

Def__init__ (Self,name):

Super (Stringfield,self). Init (Name, ' varchar (100) ')

Classintegerfield (Field):

Super (Integerfield,self). Init (Name, ' bigint ')

The initialization of the parent class is automatically called when the Stringfield,integerfield instance is initialized.

Daosh A

Classmodelmetaclass (type):

ifname== ' Model ':

Print (' Found model:%s '%name)

Mappings=dict ()

Fork,vinattrs.items ():

Ifisinstance (V,field):

Print (' Found mapping:%s ==>%s '% (k,v))

Mappings[k]=v

Forkinmappings.keys ():

Attrs.pop (k)

attrs['mappings']=mappings# save attribute and column mappings

attrs['table ']=name# assumes the table name and class name match

It did the following several things

Create a new dictionary mapping

The properties of each class are traversed by the. Items () and their key-value pairs. If the value is a field class, the key value is printed and the pair of key values is bound to the mapping dictionary.

Delete the property that just passed in the value of the field class.

Create a dedicated __mappings__ property that holds the dictionary mapping.

Create a dedicated __table__ property that holds the name of the passed-in class.

Life Two

20

21st

22

23

24

Classmodel (Dict,metaclass=modelmetaclass):

Def__init__ (Self,**kwarg):

Super (Model,self). Init (**kwarg)

Def__getattr__ (Self,key):

Try

Returnself[key]

Exceptkeyerror:

Raiseattributeerror ("' Model ' object has no attribute '%s '"%key)

Def__setattr__ (Self,key,value):

Self[key]=value

Simulating a table operation

Defsave (self):

Fields=[]

Args=[]

Fork,vinself. mappings. Items ():

Fields.Append (V.name)

Args.append (GetAttr (Self,k,none))

Sql= ' insert into%s (%s) values (%s) '% (self. table, ', '. Join (Fields), ', '. Join ([Str (i) Foriinargs]))

Print (' SQL:%s '%sql)

Print (' args:%s '%str (args))

If you create a subclass from model User:

Classuser (Model):

To define a property-to-column mapping for a class:

Id=integerfield (' id ')

Name=stringfield (' username ')

Email=stringfield (' email ')

Password=stringfield (' password ')

Then

Id= Integerfield (' id ') will automatically resolve to:

Model. SetAttr (self, ' id ', Integerfield (' id '))

Because Intergerfield (' id ') is an instance of the subclass of field and automatically triggers the __new__ of the Meta class, the Intergerfield (' ID ') is stored in __mappings__ and the key-value pair is deleted.

Three or three living things

When you initialize an instance and call the Save () method

U=user (id=12345,name= ' Batman ', email= '[email protected]', password= ' Iamback ')

U.save ()

This completes the two-born three process:

Call model first. setattr, loading a key value into a private object

Then call the "talent" of the meta-class, Modelmetaclass. New, the private object in model, as long as it is an instance of field, is automatically saved to U. mappings.

The next step is to complete the process of Sansheng everything:

The operation is stored via the U.save () simulation database. Here we just do a little bit of traversing the __mappings__ operation, virtual SQL and printing, in the real case by entering SQL statements and database to run.

The output result is

Found Model:user

Found mapping:name==>

Found mapping:password==>

Found mapping:id==>

Found mapping:email==>

Sql:insert into User (username,password,id,email) VALUES (batman,iamback,12345,[email protected])

args:[' Batman ', ' iamback ', 12345, '[email protected]'

Young Creator, you have experienced with me the great course of "Tao" Evolution "Everything", which is the core principle of Django Model section.

Next, please join me in a more fun reptile combat (well, you are now a junior hacker): Network Agent Crawl it!

Challenge two: crawling of network agents

Get ready to work, first crawl a page to play

Make sure that both the requests and pyquery packages are installed.

Files: get_page.py

Importrequests

base_headers={

' User-agent ': ' mozilla/5.0 (Windows NT 10.0; Win64; x64) applewebkit/537.36 (khtml, like Gecko) chrome/54.0.2840.71 safari/537.36 ',

' accept-encoding ': ' gzip, deflate, SDCH ',

' Accept-language ': ' zh-cn,zh;q=0.8 '

Defget_page (URL):

Headers=dict (Base_headers)

Print (' Getting ', url)

R=requests.get (Url,headers=headers)

Print (' Getting result ', Url,r.status_code)

IFR.STATUS_CODE==200:

Returnr.text

Exceptconnectionerror:

Print (' Crawling Failed ', url)

Returnnone

Here, we use the request package, the source of Baidu crawled out.

Try to catch Baidu

Stick This section behind the get_page.py and try to delete it.

if (name= = 'main'):

Rs=get_page (' https://www.baidu.com ')

Print (' result:\r\n ', RS)

Try to catch the agent

Frompyquery Importpyquery ASPQ

Start_url= '//www.proxy360.cn/region/china '

Print (' crawling ', start_url)

Html=get_page (Start_url)

Ifhtml:

DOC=PQ (HTML)

Lines=doc (' div[name= ' list_proxy_ip "] '). Items ()

Forline Inlines:

Ip=line.find ('. Tbbottomline:nth-child (1) '). Text ()

Port=line.find ('. Tbbottomline:nth-child (2) '). Text ()

Print (ip+ ': ' +port)

Let's get to the point: Using meta-classes to bulk crawl agents

Bulk processing of crawl agents

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

Fromgetpage Importget_page

Daosh: Creating a metaclass for the extraction agent

Classproxymetaclass (type):

"""

Meta-class, adding in the Freeproxygetter class

__crawlfunc__ and Crawlfunccount

Two parameters, representing the number of crawler functions, and Reptilian functions, respectively.

Count=0

attrs['crawlfunc']=[]

attrs['crawlname']=[]

If ' Crawl_ ' ink:

attrs['crawlname'].append (k)

attrs['crawlfunc'].append (v)

Count+=1

forkinattrs['crawlname']:

attrs['crawlfunccount']=count

Life Two: Create a proxy get class

Classproxygetter (Object,metaclass=proxymetaclass):

Defget_raw_proxies (Self,site):

Proxies=[]

Print (' site ', site)

Forfunc inself. Crawlfunc:

Iffunc. name==site:

This_page_proxies=func (self)

Forproxy inthis_page_proxies:

Print (' Getting ', proxy, ' from ', site)

Proxies.append (proxy)

Returnproxies

Defcrawl_daili66 (self,page_count=4):

Start_url= '//www.66ip.cn/{}.html '

Urls=[start_url.format (page) forpage InRange (1,page_count+1)]

Forurl Inurls:

Print (' crawling ', url)

Html=get_page (URL)

Trs=doc ('. Containerbox table tr:gt (0) '). Items ()

FORTR Intrs:

Ip=tr.find (' Td:nth-child (1) '). Text ()

Port=tr.find (' Td:nth-child (2) '). Text ()

Yield ': '. Join ([Ip,port])

Defcrawl_proxy360 (self):

Defcrawl_goubanjia (self):

Start_url= '//www.goubanjia.com/free/gngn/index.shtml '

Tds=doc (' Td.ip '). Items ()

FORTD Intds:

Td.find (' P '). Remove ()

Yieldtd.text (). Replace (",")

if__name__== 'main':

Two students three: instantiation of Proxygetter

Crawler=proxygetter ()

Print (crawler. Crawlname)

Forsite_label inrange (crawler. Crawlfunccount):

Site=crawler. Crawlname [Site_label]

Myproxies=crawler.get_raw_proxies (site)

Daosh: In the __new__ of the meta-class, four things are done:

Push the name of the class method that starts with "Crawl_" into Proxygetter. Crawlname

The class method that starts with "Crawl_" pushes itself into proxygetter. Crawlfunc

Calculate the number of class methods that begin with "Crawl_"

Remove all class methods that begin with "Crawl_"

What do you think? Is it very similar to the __mappings__ process that created the ORM before?

Life Two: The class defines a method for fetching page elements using Pyquery

All the agents displayed on the page were crawled from three free proxy sites.

If you are unfamiliar with yield usage, you can view:

Liaoche python Tutorial: Builder

Two students three: Create an Instance object crawler

Slightly

Sansheng everything: traversing every __crawlfunc__

On the proxygetter.__crawlname__, get the URL name that can be crawled.

Trigger class Method Proxygetter.get_raw_proxies (site)

Traverse Proxygetter. Crawlfunc, if the method name and URL name are the same, execute this method

The agent that obtains each URL is aggregated into the array output.

So... How to use the bulk agent, impact on other people's website, take someone else's password, crazy hair ads water stickers, regularly harass customers? Uh! What are you thinking? These self-realizations! If you do not understand, please listen to tell!

The young creator, the tool that created the world is already in your hands, please make it the power to the extreme!

God-level programmers take you through two sentences to fully grasp the most difficult knowledge of Python-the meta-class!