Python multi-thread task distribution

Source: Internet
Author: User
I want to capture the content of a website in multiple threads now. if the content of this website has 105 pages, but due to machine restrictions, I can only enable 10 threads to capture the content, so how can I achieve the first thread to capture 1-10 pages, the second thread is 11-20 pages, and so on, until the last tenth thread is responsible for capturing page 91-105 ,... I want to capture the content of a website in multiple threads now. if the content of this website has 105 pages, but due to machine restrictions, I can only enable 10 threads to capture the content, so how can I achieve the first thread to capture 1-10 pages, the second thread is 11-20 pages, and so on, until the last tenth thread is responsible for capturing page 91-105. how should I write the python code?

Reply content:

I want to capture the content of a website in multiple threads now. if the content of this website has 105 pages, but due to machine restrictions, I can only enable 10 threads to capture the content, so how can I achieve the first thread to capture 1-10 pages, the second thread is 11-20 pages, and so on, until the last tenth thread is responsible for capturing page 91-105. how should I write the python code?

Python3

Import urllibimport queueimport threadingdef download (queue, lck): "worker, which exits when there is no task in the queue. "While not queue. empty (): pg = queue. get () # write the code to capture the web page # write the captured content to the file lck. acquire () print ('% d page completed' % pg) lck. release () queue. task_done () def main (): "main thread," "print ('Start downloading ...... ') Lck = threading. lock () q = queue. queue () for pg in range (1,106): # The website contains 105 page q. put (pg) for I in range (10): # ten threads t = threading. thread (target = download, args = (q, lck) t. start () q. join () # wait for the task to complete print ('stop') if _ name _ = '_ main _': main ()

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.