Getting started with Python crawlers | 5 Crawl Piggy short rental rental information

Piggy short rent is a rental site, which has a lot of quality homestay rental information, below we take the rental information in Chengdu as an example, to try to crawl this data.

Piggy Short rent (Chengdu) page: http://cd.xiaozhu.com/

1. Crawl Rental title

As a rule, first crawl down the title and try the water, find the title, copy the XPath.

Multiple copies of the title XPath for several houses are compared:

[@id = "Page_list"]/ul/li[1]/div[2]/div/a/span
[@id = "Page_list"]/ul/li[2]/div[2]/div/a/span
*[@id = "Page_list"]/ul/li[3]/div[2]/div/a/span
An XPath that finds the title in an instant is changed only after the <li>, so the second writes out the XPath that crawls the full page title:

*[@id = "Page_list"]/ul/li/div[2]/div/a/span
Or a fixed routine, let's try to crawl the entire page title:

Piglets are more restrictive in terms of IP restrictions, so be sure to include the sleep () function in the code to control the frequency of crawling

Okay, let's compare the XPath message:

Follow the title of the label online to find the entire House information label, the XPath comparison is as follows:

[@id = "Page_list"]/ul/li #整体
[@id = "Page_list"]/ul/li/div[2]/div/a/span #标题
You should know how to change the code, write a loop:

File=s.xpath ('//*[@id = "page_list"]/ul/li ')
For div in file:
Title=div.xpath ("./div[2]/div/a/span/text ()") [0]
Okay, let's run it. Try it:

2. Crawling information for multiple elements

To compare the XPath of other elements:

[@id = "Page_list"]/ul/li #整体
[@id = "Page_list"]/ul/li/div[2]/div/a/span #标题
[@id = "Page_list"]/ul/li/div[2]/span[1]/i #价格
[@id = "Page_list"]/ul/li/div[2]/div/em #描述
*[@id = "Page_list"]/ul/li/a/img #图片
You can then write the code:

File=s.xpath ("//*[@id =" page_list "]/ul/li")
For div in file:
Title=div.xpath ("./div[2]/div/a/span/text ()") [0]
Price=div.xpath ("./div[2]/span[1]/i/text ()") [0]
Scrible=div.xpath ("./div[2]/div/em/text ()") [0].strip ()
Pic=div.xpath ("./a/img/@lazy_src") [0]
To try to run it:

3. Page flipping, crawling more pages

Look at the changes in the URL when turning the page:

http://cd.xiaozhu.com/search-duanzufang-p1-0/#第一页
http://cd.xiaozhu.com/search-duanzufang-p2-0/#第二页
http://cd.xiaozhu.com/search-duanzufang-p3-0/#第三页
http://cd.xiaozhu.com/search-duanzufang-p4-0/#第四页
........................
The law of the URL change is very simple, just p after the number is not the same, and with the number of page numbers is exactly the same, this is very good to run ... Write a simple loop to iterate through all the URLs.

For a in range (1,6):
url = ' http://cd.xiaozhu.com/search-duanzufang-p{}-0/'. Format (a)

We try 5 pages here, you can write the number of crawled pages according to your own needs.

The complete code is as follows:

From lxml import etree
Import requests
Import time

For a in range (1,6):
url = ' http://cd.xiaozhu.com/search-duanzufang-p{}-0/'. Format (a)
data = Requests.get (URL). text

s=etree.HTML(data)file=s.xpath(‘//*[@id="page_list"]/ul/li‘)time.sleep(3)for div in file:    title=div.xpath("./div[2]/div/a/span/text()")[0]    price=div.xpath("./div[2]/span[1]/i/text()")[0]    scrible=div.xpath("./div[2]/div/em/text()")[0].strip()    pic=div.xpath("./a/img/@lazy_src")[0]    print("{}   {}   {}   

Take a look at the effect of crawling 5 pages down:

I believe you have mastered the basic reptile routines, but you still need to be familiar with, can write code independently.

Write code not only to be careful, but also to be patient. Many people go from getting started to giving up, not because of how difficult it is to program, but because of a small problem in the course of some practice.

All right, here's the lesson! Here for everyone to prepare the rookie learn Python learning Exchange groups: 639584010, learning materials, answer questions to share with you.

Getting started with Python crawlers | 5 Crawl Piggy short rental rental information