Ruby: multi-thread Queue (Queue) download blog posts to local

Source: Internet
Author: User

Ruby: complete code for downloading blog articles locally using multiple threads

require =page_nums = 8|num|+== |f|==  doc.xpath().each do |link|= link[ href.include?            $queue.push(link[= File.new(+save_to+,$total_time_begin =threadNums = 10<<== !url.include?= Nokogiri::HTML(html).css(+ Time.now.strftime() +  + title +  + + title.gsub(/\//,),html)  url.include?|t|$total_time_end = + + ($total_time_end - $total_time_begin).to_s + 
Multithreading
$queue =page_nums = 8|num|+

First declare a Queue queue, and then add the article list page to the Queue so that you can extract the article link from these list pages. In addition, the queue is declared as a global variable ($ ), so that you can access
My previous blog post list of a civil engineer has a total of 8 pages, so we need to assign page_nums a value of 8.

threadNums = 10<<== !url.include?= Nokogiri::HTML(html).css(+ Time.now.strftime() +  + title +  + + title.gsub(/\//,),html)  url.include?|t| t.join}

Use Thread. new to create a Thread
After a thread is created, it will enter until $ queue. empty? Loop until the task queue is empty (that is, there is no URL to be collected)
Every time a thread is opened up, a url is obtained from the task queue (queue) and the webpage source code is obtained through the get_html function.
The URLs in the task queue are divided into paging URLs and article URLs.
If it is a paging url (the url contains "? Page), directly extract the article link.
If it is an article url, it is saved to the local device (save_to (), and the file name is the article title)
After creating a Thread in vitro, You need to execute the Thread # join method for the Thread to wait for the main Thread,
The main thread is not terminated until all threads are executed.

Code execution time statistics
$total_time_begin =$total_time_end = + ($total_time_end - $total_time_begin).to_s + 

The # now method of the TIme module can get the current TIme, and then use to_ I to convert the current TIme to the number of seconds that have elapsed since January 1, 1970 00:00:00 UTC.

Obtain webpage source code
= |f|=

In ruby, the Net: HTTP module and OpenURI module are used to obtain web pages. The OpenURI module is the simplest and can operate a specified webpage as a common file in diameter.

Execution result: multiple threads are used to collect more than 130 articles, which takes 15 seconds (about seconds for a single thread)

Recommended reading:

Thread and Queue

Ruby thread learning: Initiate http requests in parallel

Ruby thread: thread synchronization issues

Thread # function of join

Thread Synchronization: MonitorMixin

Three methods for Thread Synchronization

[Ruby] threads and processes

Ruby OpenURI Module

Ruby: Net: Http and open-uri

Author: Civil Engineer (http://www.cnblogs.com/hongfei)

Address: http://www.cnblogs.com/hongfei/p/3696392.html

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.