Ruby: complete code for downloading blog articles locally using multiple threads
require =page_nums = 8|num|+== |f|== doc.xpath().each do |link|= link[ href.include? $queue.push(link[= File.new(+save_to+,$total_time_begin =threadNums = 10<<== !url.include?= Nokogiri::HTML(html).css(+ Time.now.strftime() + + title + + + title.gsub(/\//,),html) url.include?|t|$total_time_end = + + ($total_time_end - $total_time_begin).to_s +
Multithreading
$queue =page_nums = 8|num|+
First declare a Queue queue, and then add the article list page to the Queue so that you can extract the article link from these list pages. In addition, the queue is declared as a global variable ($ ), so that you can access
My previous blog post list of a civil engineer has a total of 8 pages, so we need to assign page_nums a value of 8.
threadNums = 10<<== !url.include?= Nokogiri::HTML(html).css(+ Time.now.strftime() + + title + + + title.gsub(/\//,),html) url.include?|t| t.join}
Use Thread. new to create a Thread
After a thread is created, it will enter until $ queue. empty? Loop until the task queue is empty (that is, there is no URL to be collected)
Every time a thread is opened up, a url is obtained from the task queue (queue) and the webpage source code is obtained through the get_html function.
The URLs in the task queue are divided into paging URLs and article URLs.
If it is a paging url (the url contains "? Page), directly extract the article link.
If it is an article url, it is saved to the local device (save_to (), and the file name is the article title)
After creating a Thread in vitro, You need to execute the Thread # join method for the Thread to wait for the main Thread,
The main thread is not terminated until all threads are executed.
Code execution time statistics
$total_time_begin =$total_time_end = + ($total_time_end - $total_time_begin).to_s +
The # now method of the TIme module can get the current TIme, and then use to_ I to convert the current TIme to the number of seconds that have elapsed since January 1, 1970 00:00:00 UTC.
Obtain webpage source code
= |f|=
In ruby, the Net: HTTP module and OpenURI module are used to obtain web pages. The OpenURI module is the simplest and can operate a specified webpage as a common file in diameter.
Execution result: multiple threads are used to collect more than 130 articles, which takes 15 seconds (about seconds for a single thread)
Recommended reading:
Thread and Queue
Ruby thread learning: Initiate http requests in parallel
Ruby thread: thread synchronization issues
Thread # function of join
Thread Synchronization: MonitorMixin
Three methods for Thread Synchronization
[Ruby] threads and processes
Ruby OpenURI Module
Ruby: Net: Http and open-uri
Author: Civil Engineer (http://www.cnblogs.com/hongfei)
Address: http://www.cnblogs.com/hongfei/p/3696392.html