Python crawler XPath Basic use of the detailed

Source: Internet
Author: User
This article mainly describes the Python crawler XPath basic use of the details, and now share to everyone, but also to make a reference. Come and see it together.

First, Introduction

XPath is a language that looks for information in an XML document. XPath can be used to traverse elements and attributes in an XML document. XPath is the main element of the XSLT standard, and both XQuery and XPointer are built on top of the XPath expression.

Second, installation

PIP3 Install lxml

Third, the use

1. Import

From lxml import etree

2. Basic use

From lxml Import etreewb_data = "" "<p> <ul> <li class=" item-0 "><a href=" link1.html "  Rel= "External nofollow" rel= "external nofollow" rel= "external nofollow" >first item</a></li> <li class= "Item-1" ><a href= "link2.html" rel= "external nofollow" rel= "external nofollow" rel= "External nofollow" rel= "External nofollow" rel= "external nofollow" >second item</a></li> <li class= "Item-inactive" >& Lt;a href= "link3.html" rel= "external nofollow" rel= "external nofollow" rel= "external nofollow" >third item</a> </li> <li class= "item-1" ><a href= "link4.html" rel= "external nofollow" rel= "external nofollow" rel= " External nofollow ">fourth item</a></li> <li class=" item-0 "><a href=" link5.html "rel=" Exte    rnal nofollow "rel=" external nofollow "rel=" external nofollow ">fifth item</a> </ul> </p> "" "HTML = etree. HTML (Wb_data) print (HTML) result = etree.tostring (HTML) print (Result.decode ("Utf-8")) 

From the results below, our printer HTML is actually a Python object, etree.tostring (HTML) is not the whole HTML of the basic wording, complete the lack of arms and legs of the label.

 <element html at 0x39e58f0>

3, to obtain the content of a tag (basic use), note, get all the contents of a tag, a will not have to add a forward slash, otherwise error.

The wording of a

html = etree. HTML (wb_data) Html_data = Html.xpath ('/html/body/p/ul/li/a ') print (HTML) for I in Html_data:  print (I.text) < Element html at 0x12fe4b8>first itemsecond itemthird Itemfourth itemfifth Item

Two (just add a/text () directly after the label that needs to find the content)

html = etree. HTML (wb_data) Html_data = Html.xpath ('/html/body/p/ul/li/a/text () ') print (HTML) for I in Html_data:  print (i) < Element html at 0x138e4b8>first itemsecond itemthird Itemfourth itemfifth Item

4. Open read HTML file

#使用parse打开html的文件html = Etree.parse (' test.html ') Html_data = Html.xpath ('//* ') <br> #打印是一个列表, need to traverse print (Html_ data) for I in Html_data:  print (I.text)

html = etree.parse (' test.html ') Html_data = etree.tostring (html,pretty_print=true) res = Html_data.decode (' Utf-8 ') Print (res) printing:<p> <ul> <li class= "item-0" ><a href= "link1.html" rel= "external nofollow" rel= "ext Ernal nofollow "rel=" external nofollow ">first item</a></li> <li class=" item-1 "><a href=" Link2 . html "rel=" external nofollow "rel=" external nofollow "rel=" external nofollow "rel=" external nofollow "rel=" external nofollow ">second item</a></li> <li class=" item-inactive "><a href=" link3.html "rel=" external nofollow "rel=" external nofollow "rel=" external nofollow ">third item</a></li> <li class=" Item-1 " ; <a href= "link4.html" rel= "external nofollow" rel= "external nofollow" rel= "external nofollow" >fourth item</a& gt;</li> <li class= "item-0" ><a href= "link5.html" rel= "external nofollow" rel= "external nofollow" rel= "E xternal nofollow ">fifth item</a>&Lt;/li> </ul></p> 

5. Print the properties of a tag under the specified path (you can find the contents of a tag by traversing the value of the property)

html = etree. HTML (wb_data) html_data = Html.xpath ('/html/body/p/ul/li/a/@href ') for I in Html_data:  print (i)

Print:

Link1.html

Link2.html

Link3.html

Link4.html

Link5.html

6, we know that we use XPath to get all the ElementTree objects, so if you need to find content, but also need to traverse the list of data to get.

Find the absolute path under the A Tag property equals link2.html content.

html = etree. HTML (wb_data) html_data = Html.xpath ('/html/body/p/ul/li/a[@href = "link2.html" rel= "external nofollow" rel= "external nofollow "rel=" external nofollow "rel=" external nofollow "rel=" External nofollow "]/text () ') print (html_data) for I in Html_data:  print (i)

Print:

[' Second item ']

Second item

7, above we find all is absolute path (each is from the root to find), below we look for relative path, for example, find all Li tags under the a tag content.

html = etree. HTML (wb_data) html_data = Html.xpath ('//li/a/text () ') print (html_data) for I in Html_data:  print (i)

Print:

[' First item ', ' Second item ', ' Third item ', ' Fourth item ', ' Fifth item ']

First item

Second item

Third item

Fourth item

Fifth item

8, above we use absolute path, find all the A tag property equals href attribute value, take advantage of the/---absolute path, below we use relative path, look up the L relative path under the Li tag under the a tag of the HREF attribute value, note that a tag needs double//.

html = etree. HTML (wb_data) html_data = Html.xpath ('//li/a//@href ') print (html_data) for I in Html_data:  print (i)

Print:

[' link1.html ', ' link2.html ', ' link3.html ', ' link4.html ', ' link5.html ']

Link1.html

Link2.html

Link3.html

Link4.html

Link5.html

9, the relative path under the absolute path to check the specific properties of the method is similar, can also be said the same.

html = etree. HTML (wb_data) html_data = Html.xpath ('//li/a[@href = "link2.html" rel= "external nofollow" rel= "external nofollow" rel= " External nofollow "rel=" external nofollow "rel=" external nofollow "] ') print (html_data) for I in Html_data:  print ( I.text)

Print:

[<element A at 0x216e468>]

Second item

10. Find the href attribute of the A tag in the last Li tag

html = etree. HTML (wb_data) html_data = Html.xpath ('//li[last ()]/a/text () ') print (html_data) for I in Html_data:  print (i)

Print:

[' Fifth item ']

Fifth item

11. Find the href attribute of the A tag in the second-to-last Li tag

html = etree. HTML (wb_data) html_data = Html.xpath ('//li[last () -1]/a/text () ') print (html_data) for I in Html_data:  print (i)

Print:

[' Fourth item ']

Fourth item

12. If you are extracting an XPath path for a label on a page, you can:

*[@id = "KW"]

Explanation: Use a relative path to find all the tags, and the property ID equals the label of KW.

Common

#!/usr/bin/env python#-*-coding:utf-8-*-from scrapy.selector import selector, Htmlxpathselectorfrom scrapy.http Import htmlresponsehtml = "" "<! DOCTYPE html>

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.