Python Crawler crawler Performance Chapter

First, think of a For loop, single-threaded crawl each URL, but if there is a problem with the URL, the following URL will have to wait, low performance.

Second, we consider the problem of thread pool, below we define the thread pool within the maximum of 10 tasks, that is, up to the same time can only have 10 crawl tasks, so that is their own do not affect each other, plus the main thread is n+1 threads, the shortcomings are also obvious, The longest time depends on the time that the longest task is used. Pool.shutdown (True) The main thread waits for the child thread to finish executing the city before exiting.

ImportRequests fromConcurrent.futuresImportThreadpoolexecutordeffetch_request (URL): Requests.get (URL) pool= Threadpoolexecutor (10) Url_list= [    'https://www.baidu.com',    'https//:www.douban.com'] forUrlinchurl_list:pool.submit (Fetch_request,url) Pool.shutdown (True)

Simple thread pool

Three, multi-process, process pool basic notation.

ImportRequests fromConcurrent.futuresImportProcesspoolexecutordeffetch_request (URL): Requests.get (URL) pool= Processpoolexecutor (10) Url_list= [    'https://www.baidu.com',    'https//:www.douban.com'] forwr.inchurl_list:pool.submit (Fetch_request,url) Pool.shutdown (True)

simple multi-process

Summarize:
1, first use for the loop is definitely the most time serial notation, and then we discuss the efficiency of multi-process and multi-threading.
2, multi-process first to open a lot of memory space, consumption space. The IO aspect is basically the same, we know that threads exist in the process, so we can conclude that multithreading is the most efficient.

Python Crawler crawler Performance Chapter