Python3 Save the page with requests and BeautifulSoup save the picture, and the content and picture of the article can be displayed locally.

Using the requests module to do a simple crawler applet, a blog post and pictures saved to the local, the article format as '. html '. When the article is saved to the local, the connection of the picture may be the absolute or relative path of the target site, so if you want to display the image locally, you will need to replace the local path of the saved image with the local HTML file.

Save the page with the requests module, save the picture with the time BeautifulSoup, these two are third-party modules, need to install, use need to manually import.

* * Installation Method:

Pip Install Requsts

In Python3 may use pip install BeautifulSoup will error, you can directly pip install BS4, so that can be installed successfully.

Because in fact BeautifulSoup in the BS4 installation package, use when using: from BS4 import BeautifulSoup

The specific code is as follows:

1  fromBs4ImportBeautifulSoup2 ImportRequests,os3TargetDir = Os.path.join (Os.path.dirname (Os.path.abspath (__file__)),'IMGS1')#picture saved path, eg, forward folder for ' D:\Coding ', i.e. picture saved in ' D:\Coding\imgs1\ '4 if  notOs.path.isdir (TargetDir):#No creation path exists5 Os.mkdir (TargetDir)6URL ='http://www.cnblogs.com/nancyzhu/p/8146408.html'7Domain ='http://www.cnblogs.com'8 #save page to local9 defsave_html ():TenR_page =requests.get (URL) Onef = open ('page.html','WB') AF.write (r_page.content)#Save to Page.html - f.close () -     returnR_page the #Modify the file to change the picture path to a local path - defUpdate_file (old,new): -With open ('page.html', encoding='Utf-8') as F, open ('page_bak.html','W', -encoding='Utf-8') as FW:#Open two files, the original file is used for reading, and the other file writes the modified content to +          forLineinchF:#iterate through each line, taking out a string, so you can replace it with the Replace method -New_line = Line.replace (old, new)#Progressive Substitution +Fw.write (New_line)#Write new file AOs.remove ('page.html')#Delete original file atOs.rename ('page_bak.html','page.html')#Modify new file name, old-- - #save picture to local - defsave_file_to_local (): -obj = BeautifulSoup (save_html (). Content,'lxml')#The following is specified using lxml parsing, lxml resolution faster, fault-tolerant high.  -IMGs = Obj.find_all ('img') -     #Add a link to a page sheet to list inURLs = [] -      forImginchIMGs: to         if 'data-src' inchStr (IMG): +Urls.append (img['data-src']) -         Else: theUrls.append (img['src']) *     #Traverse all the image links, save the picture to the local designated folder, the image name with 0,1,2 ... $i =0Panax Notoginseng      forUrlinchURLs#See what the pictures in the article have in the format, one by one processing -         ifUrl.startswith ('//'): theNew_url ='http:'+URL +R =requests.get (New_url) A         elifUrl.startswith ('/') andUrl.endswith ('gif'): theNew_url = domain +URL +R =requests.get (New_url) -         elifUrl.endswith ('. PNG')orUrl.endswith ('jpg')orUrl.endswith ('gif'): $R =requests.get (URL) $t = os.path.join (TargetDir, str (i) +'. jpg')#Specify directory -FW = Open (T,'WB')#specifying an absolute path -Fw.write (r.content)#save picture to local specified directory thei + = 1 -Update_file (url,t)#Modify the old link (possibly a relative link) to a local link so that the entire HTML is opened locally to access the imageWuyi fw.close () the  - if __name__=='__main__': Wu save_html () -Save_file_to_local ()

Python3 Save the page with requests and BeautifulSoup save the picture, and the content and picture of the article can be displayed locally.