Python crawler Combat (ii): Using requests-html

Python crawler Combat (a): Using requests and beautifulsoup, we use the requests do network requests, get the Web data and then use BeautifulSoup parsing, just before long, requests author Kennethreitz Out of a new library requests-html,pythonic HTML parsing for humans?, which can be used to parse HTML documents. Requests-html is based on the existing framework Pyquery, requests, lxml and other libraries carried out two times package, more convenient for developers to call.

Installation

Mac:

pip3 install requests-html

Windows:

pip install requests-html

Instance

Code is much more, let us look at the sister paper, crawling site I choose is http://www.win4000.com/zt/xinggan.html, open the site, observed that this is a list, the picture is thumbnail, to save the picture to the local, of course, need a large HD map, so you have to enter the list of details , further parsing, the complete code is as follows:

 fromRequests_htmlImportHtmlsessionImportRequestsImportTimesession=Htmlsession ()# parse Picture ListdefGet_girl_list ():# Returns a Response objectResponse=Session.get (' http://www.win4000.com/zt/xinggan.html ')# units of secondsContent=Response.html.find (' Div. Left_bar ', first=True) li_list=Content.find (' Li ') forLiinchLi_list:url=Li.find (' A ', first=True). attrs[' href '] Get_girl_detail (URL)# parse Picture detailsdefGet_girl_detail (URL):# Returns a Response objectResponse=Session.get (URL)# units of secondsContent=Response.html.find (' Div.scroll-img-cont ', first=True) li_list=Content.find (' Li ') forLiinchLi_list:img_url=Li.find (' img ', first=True). attrs[' data-original '] Img_url=img_url[0: Img_url.find ('_')]+ '. jpg '        Print(Img_url+ '. jpg ') Save_image (Img_url)# Keep The big picturedefSave_image (img_url): Img_response=Requests.get (Img_url) t= int(round(Time.time ()*  +))# Millisecond Time-stampingF= Open('/users/wuxiaolong/desktop/girl/%d. jpg ' %T' AB ')# Store pictures, multimedia files require parameter B (binary file)F.write (img_response.content)# Multimedia Storage contentF.close ()if __name__ == ' __main__ ': Get_girl_list ()

The code is so much, is not feeling very simple ah.

Description

1, requests-html and BeautifulSoup different, can be directly through the label to find, generally as follows:
Label
tags. someclass
Label #someid
tags [Target=_blank]
The parameter first is True, which means that only the Element found in the second is returned, more use: http://html.python-requests.org/;

2, here Save Local path /Users/wuxiaolong/Desktop/Girl/ I write dead, need to change the reader to their own, if the file name is directly, save the path will be the project directory.

Legacy issues

Example of the crawl site is paged, did not do, can be timed cycle to crawl sister paper Oh, interested readers to play their own.

Reference

Requests-html

Today I used a requests-html library (Python crawler)

Public number

My public number: Wu Xiaolong students, Welcome to Exchange ~

Python crawler Combat (ii): Using requests-html