11.python Concurrency Primer (part12 Initial Knowledge Association)

First, the introduction of the co-process.

The coprocessor, also known as a micro-thread, is single-process, single-threaded, but in some cases, the execution of the threads in Python is more efficient than multi-threading.

This is because the transitions between the processes and the threads are completely different! The switch of the process is controlled by the program itself (the developer of the program uses yield to control, the switch between the co-process and the co-process is controllable, and when the switch is to be switched).

When multi-threading is used, the more threads you open, the more obvious the benefits of the process will be.

Another advantage of the process is that there is no need to lock the mechanism, because there is only one process, and threads, there is no multi-threaded or multi-process access to the common resources of the conflict, so there is no need to lock in the association, if more than one coprocessor to operate the same public resources, then only need to make a decision on the state of the

The process is actually can use CPU multi-core or multiple CPUs, want to achieve parallel effect, it is necessary to use, multi-process + association, this can not only ensure parallelism, but also to ensure the high efficiency of the association.

(The advantages of the co-process: 1, there is no concept of lock.) 2, the co-process between the switching overhead than multi-threading, more than a small number of processes! ）

Just mentioned earlier, the process is based on the yield keyword in the function to achieve!

If you do not understand the yield keyword, or the concept of the generator, click the portal below, which is described in detail.

http://suhaozhi.blog.51cto.com/7272298/1909032

Second, a quick review of yield and generator quick review.

The above links are described in detail, here again a quick review.

Here is an example:

Example 1:

#首先定义了一个函数func1.

Def func1 ():

Print "ok!"

Yield

#然后去调用func1这个函数.

Func1 ()

#此时print "ok! "Will it be executed?"

Can try to run, actually print "ok!" This sentence is not to be executed!

Since Python detects the yield keyword in a function, the function is no longer a normal function, and the function becomes a generator object.

We can see this phenomenon using the type function.

Print type (func1 ())

Results of the output:

<type ' Generator ' >

1. Review of the yield implementation iterator object.

1.1 If you want the code in the generator to execute, you must only use the next function, or use the __iter__ method to get an iterator object, and then execute the next method of the iterator or the Send method to trigger the execution of the generator.

(When you run a generator for the first time!) You must only use the next function, or use the __iter__ method to get an iterator object, and then execute the next method of the iterator to trigger the run! Otherwise you will get an error! ）

Def func1 ():

Print "ok!"

Yield

Next (Func1 ()) #执行了next函数后, print "ok!" This sentence is really executed!

#func1 (). __iter__ (). Next () #第二种调动方法, first gets an iterator object and then executes the next method under the Iterator object.

Output Result:

Ok!

As can be seen from the example above, the first feature of a generator created with the yield keyword is to trigger the generator execution, which can only be triggered by the next function or the Send method.

1.2 Yield and return, you can return a returned value!

Cases:

#!/usr/local/bin/python2.7

#-*-Coding:utf-8-*-

Def func1 ():

Print "ok!"

yield [I/a] #返回一个列表

L1 = Next (Func1 ())

Print L1

Output Result:

Ok!

[1, 2, 3]

1.3 Yield's hang feature.

Def func1 ():

Print "1"

Yield

Print "2"

Yield

Print "3"

Yield

Print "4"

G1 = Func1 ()

Next (G1) #第一次执行next, the generator hangs first to the position of the first yield keyword and saves the position state of the current run.

#输出结果:

1

Next (G1) #第二次执行next, the generator will continue to execute the following code from the location just suspended, and if it executes again after the yield keyword, the generator continues to hang after the state is saved.

#输出结果:

2

Next (G1) #第三次执行next, the generator will continue to run from the location just suspended, continue execution of the following code, and if it encounters yield, continue to save state pending.

#输出结果:

3

Next (G1) #第四次执行next, the generator will continue to execute the following code from where it was just suspended, but the difference is that there is no yield at the back!! After the generator executes the print "4" code, it throws a Stopiteration exception.

The 1.4 Generator's Send method, which gives the attribute the value of the variable name preceding the yield.

#!/usr/local/bin/python2.7

#-*-Coding:utf-8-*-

Def func1 ():

Print "1"

Name = Yield 11111

Print Name

Yield

G1 = Func1 ()

V1 = G1.next () #如果想直接给yield左边的name变量传值, you must first run next and then send! Otherwise it will error!!

G1.send ("Test!!!") #通过yield给左边的name变量赋值

Output Result:

1

11111

Test!!!

Third, the use of yield to achieve the most basic association process.

Personal understanding, the bottom of the process is based on yield to achieve, before said multithreading, there is a producer of consumer models, we can achieve through yield, and then through the process to achieve this model.

Before writing a producer consumer model based on the co-process, let's add a few concepts:

The traditional producer consumer model is realized through one thread to generate the message, another thread to fetch the message, if you want to operate on the public data, must lock mechanism, but accidentally can cause "deadlock" phenomenon.

Through the process to achieve the producer consumer model, the producer generates the message, directly through the yield jump to the consumer's function began to execute, when the consumer function is finished, continue to jump back to the producer's function continues to produce, efficiency than multithreading is much higher.

1. Here is a simple process that is implemented using the yield keyword.

usr/local/bin/python2.7

#-*-Coding:utf-8-*-

#这个程序的思路是, the Producer function generates data, jumps to the consumer's generator function by yield, waits for the consumer to finish the execution

#继续切换回生产者函数进行生产.

Import time

DEF consumer (name): #这里的消费者函数是个生成器!

ret = '

While True:

Bun_num = yield ret

If not bun_num:

Return

Print "%s eating bun%s"% (Name,bun_num)

Time.sleep (1)

ret = ' consumer----> Go chi so u sa ma de shi ta ~ ~ '

def produce (con):

Next (con) #第一次要先使用next函数去启动consumer生成器! (equivalent to notification)

num = 0

While Num < 5:

num = num + 1

Print "producer-producing%s"% (num)

Consumer_return = con.send (num) #一旦产生了数据! Switch to consumer (consumer), and send the resulting data to consumer.

Print "Producer---> Consumer return message:%s"% (Consumer_return)

#当生产者拿到了消费者的返回结果, the output of a message.

Con.close () #当生产者停止生产, close consumer, this close method is used to terminate the iteration.

if __name__ = = ' __main__ ':

Con1 = Consumer ("Suhaozhi") #创建消费者对象

Produce (Con1)

Output Result:

Producer--Producing 1

Suhaozhi eating Bun 1

Producer---> Consumer return message:consumer----> go chi so u sa ma de shi ta ~!

Producer--Producing 2

Suhaozhi Eating Bun 2

Producer---> Consumer return message:consumer----> go chi so u sa ma de shi ta ~!

Producer--Producing 3

Suhaozhi eating Bun 3

Producer---> Consumer return message:consumer----> go chi so u sa ma de shi ta ~!

Producer--Producing 4

Suhaozhi Eating Bun 4

Producer---> Consumer return message:consumer----> go chi so u sa ma de shi ta ~!

Producer--Producing 5

Suhaozhi eating Bun 5

Producer---> Consumer return message:consumer----> go chi so u sa ma de shi ta ~!

#从上面输出结果的速度来看, the execution of producers and consumers is a concurrency effect.

Code Analysis:

First of all, the consumer generator function, consumer (consumer) through yield to get the producer (producer sent data),

Then through this yield to return the processing results to producer (producer), read the above code, we can find that yield also has the same function as return, executed to yield this keyword, the function of the running state is suspended, It waits until the function caller executes the next function next time or executes the Send method or for loop, and the function continues to run from where it was last suspended.

After executing this code you will find that the producer and the consumer are collaborating and are not using the lock mechanism at all. (This is because the process and the coprocessor do not preempt resources at all like multithreading.) ）

Another way to realize the co-process (Greenlet).

Greenlet is a process module that comes with Python, which is more flexible and simple than the yield implementation, and it does not need to declare a function as a generator (generator).

Here is a brief introduction and demonstration of the basic use of greenlet.

Common methods:

Greenlet.greenlet () generates a Greenlet object that can pass in two parameters before generating the Greenlet object, which is run and Parent,run used to pass in a callable object, which is used by the parent to pass in a father Greenlet. The default is the current Greenlet.

Greenlet.switch () This method is used to switch between the co-process and the association, and switch is able to pass parameters to the Greenlet object.

Here is a demonstration of the basic use of Greenlet:

Import Greenlet

Def func1 ():

Print "Func1 start!"

Print "the"

Gr2.switch () #切换到func2函数

Print "7,8,9"

Gr2.switch ()

Def FUNC2 ():

Print "Func2 start!"

Print "4,5,6"

Gr1.switch () #切换到func1函数

Print "10,11,12"

Gr1 = Greenlet.greenlet (RUN=FUNC1)

GR2 = Greenlet.greenlet (RUN=FUNC2)

Gr1.switch () #启动greenlet对象gr1.

Possible methods to use:

Gr1 = Greenlet.greenlet (RUN=FUNC1)

Gr1.switch (*args, **kwargs) #切换到指定函数 (func1 function in the example) and can pass parameters to the function.

Gr1.dead Once the Greenlet object is dead, return a true.

Gr1.bool determines whether this Greenlet object is active and returns a true if the active state.

Five, more simple implementation of the process, Gevent module.

Although we can implement a basic co-process within Python through yield, it is not completely functional and Gevent provides more complete functionality support.

Gevent and greenlit are all modules that implement the co-process, but they still have a different point.

Greenlit: When encountering an I/O operation (accessing the network, writing files, and so on), I need to switch the Greenlit object, wait until the I/O operation is finished, and then back and forth again. (When using the Greenlit module, you need to manually find out where all the I/O operations are performed in the function and manually specify the switchover in the program.) ）

Gevent:gevent and greenlit The biggest difference is that it is easier than greenlit operation, it can automatically detect I/O operation, automatic to switch the co-process.

Here are examples of use: (an example of a grilled Web page)

From gevent Import Monkey

Monkey.patch_all () #可以理解为阻塞的类型

Import Gevent

Import Urllib

Import time

def get_web (URL):

Print "get:%s"% (URL)

resp = urllib.urlopen (URL)

data = Resp.geturl ()

Print "%d bytes recv from%s"% (len (data), URL)

Start_time = Time.time ()

Gevent.joinall ([Gevent.spawn (Get_web, ' http://www.baidu.com/'), Gevent.spawn (Get_web, ' https://www.github.com/'), Gevent.spawn (Get_web, ' http://zhihu.com/')])

Print "over! %s "% (Time.time ()-start_time)

Monkey.patch_all (): Used gevent will know, will be at the beginning of the place Gevent.monkey.patch_all (); the thread/in the standard library Socket, etc. so that we can use the socket in the back, as usual, no need to modify any code, but it becomes non-blocking.

Attention!! Monkey patching enables gevent to modify most of the blocking system calls in the standard library, including modules such as socket,ssl,threading and select, and becomes a collaborative operation!!!

The following is an example of an official document provided:

Import Gevent

From gevent import socket

URLs = [' www.google.com.hk ', ' www.example.com ', ' www.python.org ']

Jobs = [Gevent.spawn (socket.gethostbyname, url) for URL in URLs]

Gevent.joinall (Jobs, timeout=2)

[Job.value for job in jobs]

The Gevent.spawn () method creates some tasks and then, by Gevent.joinall, joins the task into the micro-thread execution queue to wait for it to complete, setting the timeout to 2 seconds. The results after execution are checked by gevent. Greenlet.value values to collect. The Gevent.socket.gethostbyname () function has the same interface as the standard Socket.gethotbyname (), but it does not block the entire interpreter, so that other greenlets can be executed following an unimpeded request.

There's so much more to the process than that .... This is just the beginning ...

This article is from the "Rebirth" blog, make sure to keep this source http://suhaozhi.blog.51cto.com/7272298/1926528

11.python Concurrency Primer (part12 Initial Knowledge Association)