Multi-threaded grab watercress programming books (Linux\windows can be run)

Two queues are created first, and the queue queue stores the URLs to crawl, and Out_queue stores the source code of the pages to fetch the content.

To create a thread class, the Threadurl is to crawl the content of the Web page from a given URL and store the content on the Out_queue queue, and dataminethread the role of fetching the desired content from the Web page source code.

If this code is running under window, you will need to remove the. Decode (' UTF8 ') in front of the # Delete is OK, the encoding needs to be changed.

If you want to write it yourself, do not forget to call the Task_done () function when you use the queue, or the program will run all the time.

When using regular expressions, grab the format of the content must pay attention to, and the source code in line with the beginning of the time I lost a few spaces to enter, all kinds of caught ah.

The run time is more computer-configurable and may be longer, and you can control the time by adjusting the number of crawled pages.

#-*-coding:utf-8-*-import queueimport threadingimport urllib2import timeimport refrom BeautifulSoup Import BeautifulSoup host = ' http://www.douban.com/tag/%E7%BC%96%E7%A8%8B/book?start= ' queue = Queue.queue () Out_queue = Queue.queue () class Threadurl (threading. Thread): Def __init__ (self,queue,out_queue): Threading.            Thread.__init__ (self) self.queue = Queue Self.out_queue = Out_queue def run (self): while True: Host = self.queue.get () URL = urllib2.urlopen (host) chunk = Url.read () self.out_q Ueue.put (Chunk) Self.queue.task_done () class Dataminethread (threading. Thread): Def __init__ (self,out_queue): Threading. Thread.__init__ (self) self.out_queue = Out_queue def run (self): while true:chunk = Self.out_q Ueue.get () title = Re.findall (' class= ' title ' target= ' _blank ' > (. *?) </a> ", chunk) apprise = Re.findall (" <span class= "Rating_nums";(.*?)      </span> ", chunk) desc = Re.findall (" <div class= "desc" > (. *?) </div> ", chunk) #print desc[14] for i in range: print (' title: ' +title[i]+ ' \ n    Price: ' +apprise[i]+ ' \ n Description: ' +desc[i]+ ' \ n ') #.decode (' UTF8 ') self.out_queue.task_done () start = Time.time () def main (): For I in range (5): T = Threadurl (queue,out_queue) T.setdaemon (True) T.start () for Booknum in ran GE: Queue.put (host+str (booknum*15)) for I in range (5): dt = Dataminethread (out_queue) DT.SETDA Emon (True) Dt.start () Queue.join () out_queue.join () #print ' ++++++++++++++++++++++++++++++++jd_spider+++++++++ ++++++++++++++++++++++++++++ ' main () print "Total time:%s"% (Time.time ()-start) #print ' +++++++++++++++++++++++++++++                         +++jd_spider+++++++++++++++++++++++++++++++++++++ '

Multi-threaded grab watercress programming books (Linux\windows can be run)