Python Day35 Process Pool, callback function

Tag: SYN End is implemented according to join start proc DIV code function

One, Process pool:

　　

What is a process pool: creating a certain number of processes

Synchronous and asynchronous: Two ways to submit a task.

Pool: The number of process pools and control processes created, the default number is based on the number of cores of the CPU

Apply: Pass in two parameters, the first one is the specified task. Submits a task to the process pool, implementing serial and synchronous calls. After the task is finished, you will get the result immediately.

The number of open processes is several, and there are several PID.

What is a synchronous call: Commit a task and wait until the end of the task to perform the next task.

ImportMultiprocessingImport TimeImportRandomImportOSdefWalk (N):Print('%s is walking'%os.getpid ()) Time.sleep (Random.random () )returnNif __name__=='__main__': P=multiprocessing. Pool (4)     forIinchRange (10): Q=p.apply (walk,args=(i,))Print(q)

Apply_async: Submit a task to the process pool, after the task has been submitted, just submit the task, cannot directly perform the task. Implements a concurrent and bad asynchronous call

Execution method: Close first: Shut down the task so that the task ends

Then join: Wait for a process pool not to submit the task, and the task ends and counts the number of tasks

Last get: Get return value

What is an asynchronous call: After a task is committed, it does not wait in place, but continues to submit the next task. Wait for all tasks to be completed after getting the task with get

ImportMultiprocessingImport TimeImportRandomImportOSdefWalk (N):Print('%s is walking'%os.getpid ()) Time.sleep (Random.random () )returnNif __name__=='__main__': P=multiprocessing. Pool (4) Li=[]     forIinchRange (10): Q=p.apply_async (walk,args=(i,)) Li.append (q) p.close () P.join () forIinchLi:Print(I.get ())

Second, the callback function:

What is a callback function: a function called by a function memory, if the address of the function is passed as a parameter to another function, when the address of the function is used to invoke the function that it only wants, the function that this point refers to is the callback function.

Callback: The callback function is added later

The process of the callback function is actually the main process.

Using a callback function to implement a web crawler, need to use the requests module

Get: Get URL

Status_code: Return Status code

Text: View the contents of the download URL

Details visit:http://www.cnblogs.com/hainan-zhang/p/6222552.html

 fromMultiprocessingImportpool,processImportRequestsImportOSImportTime,randomdefget (URL):Print('%s GET%s'%(Os.getpid (), URL)) Response=requests.get (URL) time.sleep (Random.randint (1,3))    ifResponse.status_code = = 200:        Print('%s Done%s'%(Os.getpid (), URL))return{'URL': URL,'text': Response.text}defParse (DIC):Print('%s PARSE%s'% (Os.getpid (), dic['URL'])) Time.sleep (1) Res='%s:%s\n'% (dic['URL'],len (dic['text'])) with open ('Db.txt','a') as F:f.write (res)if __name__=='__main__': URLs=[        'https://www.baidu.com',        'https://www.python.org',        'https://www.openstack.org',        'https://help.github.com/',        'http://www.sina.com.cn/'] P=pool (2) Start_time=time.time () Objs=[]     forUrlinchUrls:obj=p.apply_async (get,args= (URL,), callback=parse)#The main process is responsible for the work of the callback functionobjs.append (obj) p.close () P.join ()Print('Master', (Time.time ()-start_time))

Python Day35 Process Pool, callback function