Python Making embarrassing encyclopedia crawler

Morning up idle to do nothing, inexplicably pop up embarrassing things Encyclopedia of the jokes, then a think since you come to, then I write a crawler to your site crawl, one as practice practiced hand, and second also count for some fun.

In fact, these two days are also in touch with the contents of the database, you can keep the crawled data in the database for later use. All right, don't say a word, let's take a look at the data results of the program crawl

It is worth mentioning that in the program I want to crawl the content of the encyclopedia 30 page, but there is a connection error, when I put the page down to 20 pages, the program can run normally, do not know what the reason, eager to know the great God can tell me, grateful.

The program is very simple, directly on the source code slightly

# coding=utf8import Reimport requestsfrom lxml import etreefrom multiprocessing.dummy import Pool as Threadpoolimport SYSR Eload (SYS) sys.setdefaultencoding (' Utf-8 ') def getnewpage (URL, total): nowpage = Int (Re.search (' (\d+) ', URL, re. S). Group (1)) URLs = [] for I in range (nowpage, total + 1): link = re.sub (' (\d+) ', '%s '% i, URL, re. S) urls.append (link) return urlsdef spider (URL): html = requests.get (URL) selector = etree. HTML (html.text) Author = Selector.xpath ('//*[@id = "Content-left"]/p/p[1]/a[2]/@title ') content = Selector.xpath ('//*[ @id = "Content-left"]/p/p[2]/text () ') vote = Selector.xpath ('//*[@id = "Content-left"]/p/p[3]/span/i/text () ') length = Len (author) for I in range (0, length): F.writelines (' Author: ' + author[i] + ' \ n ') f.writelines (' content: ' + str (content[i]). Re  Place (' \ n ', ') + ' \ n ') F.writelines (' Support: ' + vote[i] + ' \ n ') if __name__ = = ' __main__ ': F = open (' Info.txt ', ' a ') URL = ' http://www.qiushibaike.com/text/page/1/' urls = getnewpage (URL, a) pool = ThreadPool (4) Pool.map (Spider,urls) F.close () 

If you don't know the part, you can refer to my top three articles in turn.