Getting started with Python crawlers | 5 Crawl Piggy short rent rental information __python

Piggy short rent is a rental site, which has a lot of quality accommodation rental information, the following we take the rental information in Chengdu as an example, to try to crawl the data.

Piggy Short rent (Chengdu) page: http://cd.xiaozhu.com/

1. Crawl Rental title

By convention, first climb down the title and try the water, find the title, and copy the XPath.

Multiple copies of the title XPath for several houses are compared:

*[@id = "Page_list"]/ul/li[1]/div[2]/div/a/span
//*[@id = "Page_list"]/ul/li[2]/div[2]/div/a/span
//*[@ Id= "Page_list"]/ul/li[3]/div[2]/div/a/span 

Instantly found that the XPath for the title only changes after the <li> ordinal, so the second writes out the XPath for crawling the entire page title:

*[@id = "Page_list"]/ul/li/div[2]/div/a/span 

Or a fixed set of routines, let's try to climb down the title of the whole page:

Piglets in the IP restrictions are more stringent, the code must add sleep () function to control the frequency of crawling

Okay, let's compare the XPath information:

Follow the label on the web and find the entire Housing Information tab, as compared to the XPath:

*[@id = "Page_list"]/ul/li   #整体
//*[@id = "Page_list"]/ul/li/div[2]/div/a/span   #标题 

You should know how to change the code, write a loop:

File=s.xpath ('//*[@id = ' page_list ']/ul/li ') for
div in file:    
  title=div.xpath ("./div[2]/div/a/span/text ()") [0] 

All right, let's try it out:

2. Crawling information for multiple elements

Compare XPath to other elements:

*[@id = "Page_list"]/ul/li   #整体
//*[@id = "Page_list"]/ul/li/div[2]/div/a/span   #标题
//*[@id = " Page_list "]/ul/li/div[2]/span[1]/i   #价格
//*[@id =" Page_list "]/ul/li/div[2]/div/em   #描述
//*[@ Id= "Page_list"]/ul/li/a/img   #图片 

You can then write code:

File=s.xpath ("//*[@id =" page_list "]/ul/li") for
div in file:    
  title=div.xpath ("./div[2]/div/a/span/text ()") [0]          
  Price=div.xpath ("./div[2]/span[1]/i/text ()") [0]    
  Scrible=div.xpath ("./div[2]/div/em/text ()") [0].strip ()    
  Pic=div.xpath ("./a/img/@lazy_src") [0] 

To try to run:

3. Flip page, crawl more pages

Look at the changes to the URL when you turn the page:

http://cd.xiaozhu.com/search-duanzufang-p1-0/    #第一页
http://cd.xiaozhu.com/search-duanzufang-p2-0/    # Second page
http://cd.xiaozhu.com/search-duanzufang-p3-0/    #第三页
http://cd.xiaozhu.com/ search-duanzufang-p4-0/    #第四页 ................
 

URL changes in the law is very simple, but the number of the following p is not the same, and the number of page numbers is exactly the same, this is very good to do ... Write a simple loop to iterate through all the URLs.

For a in range (1,6):
  url = ' http://cd.xiaozhu.com/search-duanzufang-p{}-0/'. Format (a)
  # We're here to try 5 pages, You can write the number of crawled pages according to your own needs. 

The complete code is as follows:

From lxml import etree
Import requests
import time to

A in range (1,6):
    url = ' http://cd.xiaozhu.com/ search-duanzufang-p{}-0/'. Format (a)
    data = Requests.get (URL). Text

    S=etree. HTML (data)
    File=s.xpath ('//*[@id = ' page_list ']/ul/li ')
    time.sleep (3) for
    
    div in file:
        title= Div.xpath ("./div[2]/div/a/span/text ()") [0]
        Price=div.xpath ("./div[2]/span[1]/i/text ()") [0]
        scrible= Div.xpath ("./div[2]/div/em/text ()") [0].strip ()
        Pic=div.xpath ("./a/img/@lazy_src") [0]
            
        print ("{}   {} {   } {   }\n '. Format (title,price,scrible,pic)) 

Look at the effect of crawling down 5 pages:

I'm sure you've mastered the basics of reptiles, but you need to be familiar with it and write your own code.

Writing code is not only careful but also requires patience. A lot of people from the beginning to give up, not because of programming this matter how difficult, but a practice process, encountered a small problem.

All right, this is the section to share.