Python crawl to sister web the entire Picture resource tutorial (most detailed version)

The low-level tutorials for crawling girls ' nets are connected as follows: [Climbing sister net] (68958267)

PS: Support only a single set of diagram download, does not support the overall download

Talking about my design idea of this reptile code:

① When we browse this site, we will find that each page URL is based on the site's domain name +page+ pages, so that we can visit the site of each page.

② when we look at the picture list, put the mouse to the picture, right click to check, we found that the contents of the picture by the UL wrapped Li composition, the arrow refers to the place for each set of map address, so that we can enter the set diagram, a such page contains 24 such a set of diagrams, we use BeautifulSoup, Processing.

③ we enter the set diagram, the mouse to 40, right-click, found that the picture of the set of pictures of the maximum number of images is the tenth span of the value, and each set of the URL of the map is the same as the ① principle of the url+ of the set of pictures (such as 3 for the second picture), The URL of the last download consists of a class of main-titlediv. Extract the SRC attribute of the IMG tag to get the download link

From BS4 import BeautifulSoup

Import requests

Import OS

Base_url= ' http://www.mzitu.com/page/'

Header = {' Referer ': ' http://www.mzitu.com '}

#反 ' anti-hotlinking '

For x in range (11,20):

#盗取第十一页到19页图片

Html_a=requests.get (Base_url+str (x), Headers=header)

Soup_a=beautifulsoup (html_a.text,features= ' lxml ')

#解析第一个网页

Pages=soup_a.find (' ul ', {' id ': ' pins '}). Find_all (' a ')

#选出a标签, as the arrow in the second step refers to the place

B=1

For Y in pages:

if (b%2!=0):

#因为一个li标签里面有两个a标签, so to remove duplicates

Html=requests.get (y[' href '],headers=header)

Soup_b=beautifulsoup (html.text,features= ' lxml ')

#进入套图, Parse set diagram

Pic_max=soup_b.find_all (' span ') [10].text

#选出该套图的最大图片数

Tittle=soup_b.find (' H2 ', {' class ': ' Main-title '}). Text

Os.makedirs ('./img/' +str (tittle))

#制造一个目录

For I in range (1,int (Pic_max) +1):

#循环, download the set of pictures,

href=y[' href ']+ '/' +str (i)

Html2=requests.get (Href,headers=header)

Soup2=beautifulsoup (html2.text,features= ' lxml ')

Pic_url=soup2.find (' img ', alt=tittle)

Html_name=requests.get (pic_url[' src '],headers=header,stream=true)

file_name=pic_url[' src '].split (r '/') [-1]

With open ('./img/' +str (tittle) + '/' +file_name, ' WB ') as F:

#按32字节下载

For x in Html_name.iter_content (chunk_size=32):

F.write (x)

B=b+1

Print (' OK ')

From bs4 import beautifulsoupimport requestsimport osbase_url= ' http://www.mzitu.com/ page/' header = {  ' Referer ': ' http://www.mzitu.com '}for x in range (13,20):     html_a=requests.get (Base_url+str (x), Headers=header)     soup_a= BeautifulSoup (html_a.text,features= ' lxml ')     pages=soup_a.find (' ul ', {' id ': ' pins '}). Find_ All (' a ')     b=1    for y in pages:         if (b%2!=0):             html=requests.get (y[' href '],headers=header)              soup_b=beautifulsoup (html.text,features= ' lxml ')              pic_max=soup_b.find_all (' span ') [10].text             tittle=sOup_b.find (' H2 ', {' class ': ' Main-title '}) .text             os.makedirs ('./img/' +str (tittle))              for i in range (1,int (Pic_max) +1):                 href=y[' href ']+ '/' +str (i)                  html2=requests.get (Href,headers=header)                  soup2=beautifulsoup ( html2.text,features= ' lxml ')                  pic_url=soup2.find (' img ', alt=tittle)                  html_name=requests.get (pic_url[' src '],headers=header,stream=true)                 file_name=pic_url[' src '].split (r '/') [-1]                                  with open ('./img/' +str (tittle) + '/ ' +file_name, ' WB ')  as f:                     for x in html_name.iter_content (chunk_size=32):                          f.write (x)                                   b=b+1print (' OK ')

The above code is the original code,

Crawl Results

Python crawl to sister web the entire Picture resource tutorial (most detailed version)