Getting started with Python crawlers | 5 Crawl Piggy short rent rental information __python

Piggy short rent is a rental site, which has a lot of quality accommodation rental information, the following we take the rental information in Chengdu as an example, to try to crawl the data.

Piggy Short rent (Chengdu) page: http://cd.xiaozhu.com/

  1. Crawl Rental title

By convention, first climb down the title and try the water, find the title, and copy the XPath.

 
Multiple copies of the title XPath for several houses are compared:

*[@id = "Page_list"]/ul/li[1]/div[2]/div/a/span
//*[@id = "Page_list"]/ul/li[2]/div[2]/div/a/span
//*[@ Id= "Page_list"]/ul/li[3]/div[2]/div/a/span

Instantly found that the XPath for the title only changes after the ordinal, so the second writes out the XPath for crawling the entire page title:

*[@id = "Page_list"]/ul/li/div[2]/div/a/span

 
Or a fixed set of routines, let's try to climb down the title of the whole page:

Piglets in the IP restrictions are more stringent, the code must add sleep () function to control the frequency of crawling

 
Okay, let's compare the XPath information.

Follow the label on the web and find the entire Housing Information tab, as compared to the XPath:

*[@id = "Page_list"]/ul/li   #整体
//*[@id = "Page_list"]/ul/li/div[2]/div/a/span   #标题

 
You should know how to change the code, write a loop:

File=s.xpath ('//*[@id = ' page_list ']/ul/li ') for
div in file:    
  title=div.xpath ("./div[2]/div/a/span/text ()") [0]

 
All right, let's try it out:

  2. Crawling information for multiple elements

Compare XPath to other elements:

*[@id = "Page_list"]/ul/li   #整体
//*[@id = "Page_list"]/ul/li/div[2]/div/a/span   #标题
//*[@id = "Page_ List "]/ul/li/div[2]/span[1]/i   #价格
//*[@id =" Page_list "]/ul/li/div[2]/div/em   #描述
//*[@id =" Page_list "]/ul/li/a/img   #图片

 
You can then write code:

File=s.xpath ("//*[@id =" page_list "]/ul/li") for
div in file:    
  title=div.xpath ("./div[2]/div/a/span/text ()") [0]          
  Price=div.xpath ("./div[2]/span[1]/i/text ()") [0]    
  Scrible=div.xpath ("./div[2]/div/em/text ()") [0].strip ()    
  Pic=div.xpath ("./a/img/@lazy_src") [0]

 
To try to run:

  3. Flip page, crawl more pages

Look at the changes to the URL when you turn the page:

http://cd.xiaozhu.com/search-duanzufang-p1-0/    #第一页
http://cd.xiaozhu.com/search-duanzufang-p2-0/    # Second page
http://cd.xiaozhu.com/search-duanzufang-p3-0/    #第三页
http://cd.xiaozhu.com/ search-duanzufang-p4-0/    #第四页 ................

 
URL changes in the law is very simple, but the number of the following p is not the same, and the number of page numbers is exactly the same, this is very good to do ... Write a simple loop to iterate through all the URLs.

For a in range (1,6):    
  url = ' http://cd.xiaozhu.com/search-duanzufang-p{}-0/'. Format (a)
  # We're here to try 5 pages, You can write the number of crawled pages according to your own needs.

 
The complete code is as follows:

From lxml import etree
Import requests
import time to

A in range (1,6):
    url = ' http://cd.xiaozhu.com/ search-duanzufang-p{}-0/'. Format (a)
    data = Requests.get (URL). Text

    S=etree. HTML (data)
    File=s.xpath ('//*[@id = ' page_list ']/ul/li ')
    time.sleep (3) for

    div in file:
        title= Div.xpath ("./div[2]/div/a/span/text ()") [0]
        Price=div.xpath ("./div[2]/span[1]/i/text ()") [0]
        scrible= Div.xpath ("./div[2]/div/em/text ()") [0].strip ()
        Pic=div.xpath ("./a/img/@lazy_src") [0]

        print ("{}   {} {   } {   }\n '. Format (title,price,scrible,pic))

 
Look at the effect of crawling down 5 pages:

 
I'm sure you've mastered the basics of reptiles, but you need to be familiar with it and write your own code.

Writing code is not only careful but also requires patience. A lot of people from the beginning to give up, not because of programming this matter how difficult, but a practice process, encountered a small problem.

 
Well, that's all for the class.

In vain ~

If you want a teacher with a professional teacher, a systematic knowledge, and real-time online answers, you can join our program Python Crawler (Introduction + Advanced).

Next section: Getting Started with Python crawler | 6 Save the crawled data to the local

Complete 7-Section course catalogue:
Getting started with Python crawlers | 1 installation of the Python environment
Getting started with Python crawlers | 2 Crawl Film Information
Getting started with Python crawlers | 3 Crawler Essentials Python Knowledge
Getting started with Python crawlers | 4 Climbing Watercress TOP250 book information
Getting started with Python crawlers | 5 Crawl Piggy short rent rental information
Getting started with Python crawlers | 6 Save the crawled data to the local
Getting started with Python crawlers | 7 category crawl film to solve dynamic loading problem