1 minute to quickly generate xslt for Web content extraction, web content xslt

1 minute to quickly generate xslt for Web content extraction, web content xslt

1 minute to quickly generate xslt for Web content extraction. The specific content is as follows:

1. Project Background

In the article "Python real-time web crawler project description", we have said that a general web crawler can save programmers more than half the time. The focus is that the crawling rules used by the extraction machine need to be quickly generated. In the text of extracting Web page data using xslt in python, we have seen that this extraction rule is an xslt program. In the example program, a long segment of xslt is directly assigned to the variable, but I didn't talk about how this xslt came from.

Netizens will inevitably question: does it take a long time to compile this xslt?

In reality, this xslt is automatically generated by using the intuitive tagging function of GooSeeker's MS. If you are proficient in it, you can do it in just one minute.

2. What Can MS do?

MS has several graphical interfaces that integrate a series of html parsing tools, including:

• Automatically generate XSLT Based on Visual Annotation
• Test the correctness of XSLT in real time
• Tree-like DOM structure display
• Profiling the attributes of a DOM Node
• Generate an XPath for the DOM node. You can choose to locate the class, id, or absolute location.
• Search DOM nodes based on xpath

MS has three interfaces: DOM window, embedded browser window, And workbench. Define xslt conversion rules on the workbench.

3. Use MS to generate several XSLT

Suppose we want to capture the forum post list. The following describes the operation method step by step:
Step 1: Open GooSeeker's MS and enter the URL to be crawled
Step 2: In the Browser display window of MS mouji, select the content to be extracted, name it, and click OK.

Step 3: click the "test" button on the workbench. xslt is generated and displayed in the "data rules" window.

With the above operations, you do not need to program them. You can mark them on the page using a graphical interface to generate xslt within one minute.

4. How to Use XSLT

Using xslt to extract webpage data in python, we handed the generated xslt into the program as a string. It seems like a sudden return to prehistoric civilization, the original copy is used. Otherwise, it is just an example. The article "python real-time web crawler project: Definition of content extractors" has taken shape. There are many xslt injection methods, and the most automated method is api, I will explain it in detail in subsequent articles.

5. Document modification history

: V3.0, Chapter 2 added
: V2.0, additional text description

The above is all the content of this article. I hope it will be helpful for your learning and support for helping customers.