I want to capture the content of a website in multiple threads now. if the content of this website has 105 pages, but due to machine restrictions, I can only enable 10 threads to capture the content, so how can I achieve the first thread to capture 1-10 pages, the second thread is 11-20 pages, and so on, until the last tenth thread is responsible for capturing page 91-105 ,... I want to capture the content of a website in multiple threads now. if the content of this website has 105 pages, but due to machine restrictions, I can only enable 10 threads to capture the content, so how can I achieve the first thread to capture 1-10 pages, the second thread is 11-20 pages, and so on, until the last tenth thread is responsible for capturing page 91-105. how should I write the python code?
Reply content:
I want to capture the content of a website in multiple threads now. if the content of this website has 105 pages, but due to machine restrictions, I can only enable 10 threads to capture the content, so how can I achieve the first thread to capture 1-10 pages, the second thread is 11-20 pages, and so on, until the last tenth thread is responsible for capturing page 91-105. how should I write the python code?
Python3
Import urllibimport queueimport threadingdef download (queue, lck): "worker, which exits when there is no task in the queue. "While not queue. empty (): pg = queue. get () # write the code to capture the web page # write the captured content to the file lck. acquire () print ('% d page completed' % pg) lck. release () queue. task_done () def main (): "main thread," "print ('Start downloading ...... ') Lck = threading. lock () q = queue. queue () for pg in range (1,106): # The website contains 105 page q. put (pg) for I in range (10): # ten threads t = threading. thread (target = download, args = (q, lck) t. start () q. join () # wait for the task to complete print ('stop') if _ name _ = '_ main _': main ()