Python crawler XPath Basic use of the detailed

This article mainly describes the Python crawler XPath basic use of the details, and now share to everyone, but also to make a reference. Come and see it together.

First, Introduction

XPath is a language that looks for information in an XML document. XPath can be used to traverse elements and attributes in an XML document. XPath is the main element of the XSLT standard, and both XQuery and XPointer are built on top of the XPath expression.

Second, installation

PIP3 Install lxml

Third, the use

1. Import

From lxml import etree

2. Basic use

From lxml Import etreewb_data = "" "<p> <ul> <li class=" item-0 "><a href=" link1.html "  Rel= "External nofollow" rel= "external nofollow" rel= "external nofollow" >first item</a></li> <li class= "Item-1" ><a href= "link2.html" rel= "external nofollow" rel= "external nofollow" rel= "External nofollow" rel= "External nofollow" rel= "external nofollow" >second item</a></li> <li class= "Item-inactive" >& Lt;a href= "link3.html" rel= "external nofollow" rel= "external nofollow" rel= "external nofollow" >third item</a> </li> <li class= "item-1" ><a href= "link4.html" rel= "external nofollow" rel= "external nofollow" rel= " External nofollow ">fourth item</a></li> <li class=" item-0 "><a href=" link5.html "rel=" Exte    rnal nofollow "rel=" external nofollow "rel=" external nofollow ">fifth item</a> </ul> </p> "" "HTML = etree. HTML (Wb_data) print (HTML) result = etree.tostring (HTML) print (Result.decode ("Utf-8")) 

From the results below, our printer HTML is actually a Python object, etree.tostring (HTML) is not the whole HTML of the basic wording, complete the lack of arms and legs of the label.

 <element html at 0x39e58f0>

3, to obtain the content of a tag (basic use), note, get all the contents of a tag, a will not have to add a forward slash, otherwise error.

The wording of a


html = etree. HTML (wb_data) Html_data = Html.xpath ('/html/body/p/ul/li/a ') print (HTML) for I in Html_data:  print (I.text) < Element html at 0x12fe4b8>first itemsecond itemthird Itemfourth itemfifth Item

Two (just add a/text () directly after the label that needs to find the content)


html = etree. HTML (wb_data) Html_data = Html.xpath ('/html/body/p/ul/li/a/text () ') print (HTML) for I in Html_data:  print (i) < Element html at 0x138e4b8>first itemsecond itemthird Itemfourth itemfifth Item

4. Open read HTML file


#使用parse打开html的文件html = Etree.parse (' test.html ') Html_data = Html.xpath ('//* ') <br> #打印是一个列表, need to traverse print (Html_ data) for I in Html_data:  print (I.text)


html = etree.parse (' test.html ') Html_data = etree.tostring (html,pretty_print=true) res = Html_data.decode (' Utf-8 ') Print (res) printing:<p> <ul> <li class= "item-0" ><a href= "link1.html" rel= "external nofollow" rel= "ext Ernal nofollow "rel=" external nofollow ">first item</a></li> <li class=" item-1 "><a href=" Link2 . html "rel=" external nofollow "rel=" external nofollow "rel=" external nofollow "rel=" external nofollow "rel=" external nofollow ">second item</a></li> <li class=" item-inactive "><a href=" link3.html "rel=" external nofollow "rel=" external nofollow "rel=" external nofollow ">third item</a></li> <li class=" Item-1 " ; <a href= "link4.html" rel= "external nofollow" rel= "external nofollow" rel= "external nofollow" >fourth item</a& gt;</li> <li class= "item-0" ><a href= "link5.html" rel= "external nofollow" rel= "external nofollow" rel= "E xternal nofollow ">fifth item</a>&Lt;/li> </ul></p> 

5. Print the properties of a tag under the specified path (you can find the contents of a tag by traversing the value of the property)


html = etree. HTML (wb_data) html_data = Html.xpath ('/html/body/p/ul/li/a/@href ') for I in Html_data:  print (i)

Print:


 

 
  
 
   
Link1.html
 
   
Link2.html
 
   
Link3.html
 
   
Link4.html
 
   
Link5.html
 
  

 
6, we know that we use XPath to get all the ElementTree objects, so if you need to find content, but also need to traverse the list of data to get.

Find the absolute path under the A Tag property equals link2.html content.


html = etree. HTML (wb_data) html_data = Html.xpath ('/html/body/p/ul/li/a[@href = "link2.html" rel= "external nofollow" rel= "external nofollow "rel=" external nofollow "rel=" external nofollow "rel=" External nofollow "]/text () ') print (html_data) for I in Html_data:  print (i)

Print:


 

 
  
 
   
[' Second item ']
 
   
Second item
 
  

 
7, above we find all is absolute path (each is from the root to find), below we look for relative path, for example, find all Li tags under the a tag content.


html = etree. HTML (wb_data) html_data = Html.xpath ('//li/a/text () ') print (html_data) for I in Html_data:  print (i)

Print:


 

 
  
 
   
[' First item ', ' Second item ', ' Third item ', ' Fourth item ', ' Fifth item ']
 
   
First item
 
   
Second item
 
   
Third item
 
   
Fourth item
 
   
Fifth item
 
  

 
8, above we use absolute path, find all the A tag property equals href attribute value, take advantage of the/---absolute path, below we use relative path, look up the L relative path under the Li tag under the a tag of the HREF attribute value, note that a tag needs double//.


html = etree. HTML (wb_data) html_data = Html.xpath ('//li/a//@href ') print (html_data) for I in Html_data:  print (i)

Print:


 

 
  
 
   
[' link1.html ', ' link2.html ', ' link3.html ', ' link4.html ', ' link5.html ']
 
   
Link1.html
 
   
Link2.html
 
   
Link3.html
 
   
Link4.html
 
   
Link5.html
 
  

 
9, the relative path under the absolute path to check the specific properties of the method is similar, can also be said the same.


html = etree. HTML (wb_data) html_data = Html.xpath ('//li/a[@href = "link2.html" rel= "external nofollow" rel= "external nofollow" rel= " External nofollow "rel=" external nofollow "rel=" external nofollow "] ') print (html_data) for I in Html_data:  print ( I.text)

Print:


 

 
  
 
   
[<element A at 0x216e468>]
 
   
Second item
 
  

 
10. Find the href attribute of the A tag in the last Li tag


html = etree. HTML (wb_data) html_data = Html.xpath ('//li[last ()]/a/text () ') print (html_data) for I in Html_data:  print (i)

Print:


 

 
  
 
   
[' Fifth item ']
 
   
Fifth item
 
  

 
11. Find the href attribute of the A tag in the second-to-last Li tag


html = etree. HTML (wb_data) html_data = Html.xpath ('//li[last () -1]/a/text () ') print (html_data) for I in Html_data:  print (i)

Print:


 

 
  
 
   
[' Fourth item ']
 
   
Fourth item
 
  

 
12. If you are extracting an XPath path for a label on a page, you can:


*[@id = "KW"]

Explanation: Use a relative path to find all the tags, and the property ID equals the label of KW.


Common


#!/usr/bin/env python#-*-coding:utf-8-*-from scrapy.selector import selector, Htmlxpathselectorfrom scrapy.http Import htmlresponsehtml = "" "<! DOCTYPE html>