Python allows crawlers to download beautiful pictures,

Python allows crawlers to download beautiful pictures,

The post crawled this time is Baidu's beauty. It gives some encouragement to the masses of male compatriots.

Before crawling, You need to log on to the Baidu post Bar account in the browser. You can also use post in the code to submit or add cookies.

Crawling address: http://tieba.baidu.com? Kw = % E7 % BE % 8E % E5 % A5 % B3 & ie = UTF-8 & pn = 0

#-*-Coding: UTF-8-*-import urllib2import reimport requestsfrom lxml import etree these are the libraries to import. The Code does not use regular expressions and uses xpath, we recommend that you use the basic library to write regular expressions, in this way, you can learn more about links = [] # url traversal address k = 1 print U'. Enter the last page number: 'endpage = int (raw_input ()) # The final page number (R' \ d + (? = \ S * Page) This is a common code for capturing the total number of pages with regular expressions. Of course, The last step is to group # Here Is to manually enter the number of pages to avoid too much content for j in range (0, endPage): url = 'HTTP: // tieba.baidu.com/f? Kw = % E7 % BE % 8E % E5 % A5 % B3 & ie = UTF-8 & pn = '+ str (j) # url of the page size html = urllib2.urlopen (url ). read () # read the homepage content selector = etree. HTML (html) # convert to xml, used to identify links = selector in the next step. xpath ('// div/a [@ class = "j_th_tit"]/@ href ') # capture the URLs of all posts on the current page # You can use the source code viewing tool that comes with your browser to view the elements at the specified destination, which is faster for I in links: url1 = "http://tieba.baidu.com" + I # because the crawling address is relative address, so add Baidu domain html2 = urllib2.urlopen (url1 ). read () # read the content of the current page selector = etree. HTML (html2) # Conversion Use xml to identify link = selector. xpath ('// img [@ class = "BDE_Image"]/@ src') # capture an image. You can also change it to a regular expression, or other content you want # Here is the traversal download for each in link: # print each print U'. % d' % k fp = open ('image/'character str(kw.w.w.w.'.bmp ', 'wb') # download it to the image folder in the current directory. The image format is bmp image1 = urllib2.urlopen (each ). read () # read the image content fp. write (image1) # write the image fp. close () k + = 1 # k is the file name. 1 print U' is added to each downloaded file. Download is complete!'

If you want to crawl the content of other sites, you can refer