11.python Concurrency Primer (part11 process sync lock, Process pool, and callback concept)

First, about the process lock.

In fact, about the process of the lock is not good to talk about, the role and the thread of the mutex (also known as the global lock is called synchronous lock) effect is almost the same.

are used to lock up public resources for data protection.

When a process wants to operate a public resource, it can "lock" the public resource process, and if other processes want to access or manipulate the public resource, other processes can only block, wait for the process to release the lock, and the next process can manipulate the public resource.

Here is a demonstration of the use of process locks:

#!/usr/local/bin/python2.7

#-*-Coding:utf-8-*-

Import multiprocessing

def func1 (lock_1,name):

Lock_1.acquire ()

The print "Hello%s"% (name) #当执行到这句话的时候, is no longer parallel, and the concept of lock in threads is the same.

Lock_1.release ()

if __name__ = = ' __main__ ':

Lock = multiprocessing. Lock ()

For NUM in range (10):

Multiprocessing. Process (target=func1,args= (Lock,num)). Start ()

Output Result:

Hello 0

Hello 1

Hello 2

Hello 3

Hello 4

Hello 5

Hello 6

Hello 7

Hello 8

Hello 9

！！ Add, in fact in the process, can also use Rlock recursive lock, Semaphore, event and other locks, usage basic and multithreading is the same.

Second, about the process pool.

What is a process pool?

A process pool is a sequence of processes maintained in a program that gets a process from the process pool when a process is needed, and when there are no more processes in the process pool, the program blocks until a new process is added to the process pool.

So what are the benefits of the process pool? Take a look at the following example.

If there is a function, the content of this function needs to be executed 100 times to complete the task we want to accomplish.

We assume that it takes 10 seconds to run this function once, and if it is executed serially using a single-process single-threaded mode, it will take 1000 seconds to execute this function 100 times, very inefficient!

So many threads? We can open 100 threads at the same time, each thread executes the function once, and it can be executed very quickly, but only if the operation performed by this function must be I/O intensive, so that it can play out the high efficiency of the multi-threading, what if the function performs computationally intensive? (If the computationally intensive task is given to multithreading, it may reduce efficiency because of the nature of the global interpreter lock) then we need to think about how to open the process.

Well, then we'll deal with this problem in a multi-threaded way.

Since this function is going to execute 100 times, I will simply open 100 processes and let the 100 functions execute this function in parallel.

This really can greatly improve the efficiency, but, whether it is to open 100 processes, or to open 100 threads, so the cost is too big!!! Multiple threads in the same process are good, the data resources are shared between multiple processes, each switch between threads and threads, the value needs to save the current execution state, but the thread, the threads and threads of data are independent, multiple processes want to achieve data sharing, It is necessary to copy a data resource for each process, open 100 threads, data resources will be copied 100 copies! The cost is too great!

In fact, these methods are not very good, but can compromise, create a thread pool.

Next, let's analyze the benefits of the process pool.

Let's say that we created a process pool, and when we created the process pool, we specified that the process pool could hold up to 10 processes, which meant that the maximum concurrency at one time was 10 processes to execute the function.

Next, these 10 processes have done their own things, of course, these 10 processes can not be completely executed together (unless your CPU has 10 cores), there must be a task before the completion of the process.

So in other words, the 10 concurrent running processes, now if a process has been completed, then there are 9 processes in the process pool, my process pool now empty out a location, vacated this location, you can continue to open a new process to work (as long as the process pool is empty, You can start a new process to continue working. ）

So, through the above example, it is clear that the benefit of the process pool is that it can maintain the maximum concurrency of a process's work, such as we manually set the process pool to accommodate up to 10 processes, then when the program runs, the number of processes will not be higher than 10, or less than 10, This maximum concurrency can be maintained.

Although there is no way to perform this function 100 times in 10 seconds, using a process pool is certainly much faster than serial execution!

So, the process pool is a compromise solution.

The following is an example of a process pool usage:

#!/usr/local/bin/python2.7

#-*-Coding:utf-8-*-

Import multiprocessing

Import OS

Import time

def func1 (value):

Time.sleep (5)

Print "---------->%s"% (value)

Return value+100

def display_pro_info (value):

Print Os.getpid ()

Print Os.getppid ()

Print "log:%s"% (value)

Pro_pool = multiprocessing. Pool #创建线程池对象, and specifies and the maximum concurrency, if not specified, is automatically adjusted according to the current computer's CPU core count.

Display_pro_info (1)

Print "--------------->"

For I in range: #创建100个线程

# Pro_pool.apply_async (func=func1,args= (i)) #这一步可以暂时理解为往进程池里添加进程, this step is described in detail later in this article.

Pro_pool.apply_async (func=func1,args= (i,), Callback=display_pro_info) #这里使用了callback回调函数, what is a callback function, which is described later in this article.

Pro_pool.close () #在这需要注意一点, is to use the process pool, must first close and then join, otherwise it will be an error. First close after join this order is fixed!!

Pro_pool.join () #没有join的话, processes in the process pool will not run.

Print "The end!"

The following outputs the results of this program's operation:

44720

40833

log:0

--------------->

---------->0

---------->1

---------->3

---------->2

---------->5

---------->9

---------->4

---------->7

---------->6

---------->8

44720

40833

log:100

44720

40833

Log:101

44720

40833

log:103

44720

40833

log:102

44720

40833

log:109

44720

40833

log:104

44720

40833

log:107

44720

40833

log:106

44720

40833

log:105

44720

40833

log:108

......

The end!

The following begins the analysis of the code and explains some common methods in the process pool:

1. First, there are two modes of adding processes to the process pool.

Apply (synchronous execution mode) and Apply_async (asynchronous execution mode), respectively.

1.1 Apply (synchronous execution mode)

This mode is typically not used by anyone, if the process pool uses this mode, when the process pool's first process finishes executing, the second process executes, and after the second process finishes executing, the third process is executed .... (that is, this mode blocks the main process!) ), the parallel effect cannot be achieved. (more than that, this mode does not support the callback callback function.) ）

1.2 Apply_async (Asynchronous execution mode)

Asynchronous execution mode is the one that can implement the parallel effect, support the callback callback function, when a process is not completed, no results are returned, the asynchronous execution mode does not block the main process!

Add a little! Although Apply_async is non-blocking, the Get method that returns the result is blocked, such as using Result.get () to block the main process!

1.3 Pro_pool.apply_async (func=func1,args= (i,), Callback=display_pro_info)

There is no difference between syntax and creating multithreaded multi-process, Func is used to specify a function to be run by the process, and args is used to pass arguments to the function, callback is used to specify the callback function.

2. About Join,close,terminate.

2.1 Join main process block waits for the child process to exit after the run ends, the join method is used after close or terminate. (in other words, without the Join method, processes in the process pool will not execute at all ~)

2.2 Close closes the process pool and no longer accepts new tasks

2.3 Terminate directly ends the process and no longer handles tasks that are not processed

#其实join和close比较常用.

3. A simple description of the callback function.

Before you know the callback function, you need to pay attention!! The callback function is called by the main process, not the child process!! You can see the results from the example just now!

After the completion of a function, a function or action executed successfully, then go to execute a function, and the previous execution of the function of the return value, will be passed as a parameter to the callback function.

What are the benefits of having the callback function invoke under the main process?

For example, now we need to open 10 threads to operate the data in the database, now there is a requirement, that is, the operation of the database must record a log, we can write the log this function as a callback (callback) function.

Before using the callback function, if you want to open 10 processes to operate the database, the 10 processes are executed concurrently, each process to operate the database, at the same time to write a log file, if not locked, it is easy to cause the log file corruption, So we can give the log function to the main process to execute, when the process pool of 10 processes run, the main thread directly executes a write log operation, this is what I understand the usefulness of the callback function.

Take the callback in the process pool for example, Pro_pool.apply_async (func=func1,args= (i,), Callback=display_pro_info) When the FUNC1 function is executed internally (as long as the return value is not false), the Display_pro_info function is executed as a callback function (which is, of course, the main process of executing this function!). ), the return value of FUNC1 is passed as a parameter to the Display_pro_info function.

Finally, add some points to note when using the callback callback function.

1. The function before executing the callback function must have a return value! This return value has two purposes, the first purpose is to determine whether the function succeeds, execution succeeds to execute the callback function, and another use is that the return value is passed as a parameter to the callback function.

2. The callback callback function itself must receive a parameter that is used to receive the return value of the last function after execution.

   
   

This article is from the "Rebirth" blog, make sure to keep this source http://suhaozhi.blog.51cto.com/7272298/1926096

11.python Concurrency Primer (part11 process sync lock, Process pool, and callback concept)