Basic theory of Jsoup (i.) __JS

Introduction of Jsoup

In the past, when using Java to process parsing HTML documents or fragments, we usually use Htmlparser (http://htmlparser.sourceforge.net/), the Open source class library. Now we have jsoup, the future processing of HTML content only need to use Jsoup is enough, Jsoup has a faster update, more convenient API and so on.

Jsoup is a Java HTML parser that can directly parse a URL address, HTML text content. It provides a very labor-saving API for fetching and manipulating data through dom,css and jquery-like operations, and can be seen as a Java version of jquery.

The main functions of Jsoup are as follows:

Parsing html from a URL, file, or string;

Use a DOM or CSS selector to find and retrieve data;

Can manipulate HTML elements, attributes, text;

Jsoup is based on the MIT protocol and can be safely used in commercial projects. Official website: http://jsoup.org/

Second, parse traversal HTML document

Jsoup processing HTML files is the process of converting the HTML document entered by the user, parsing it into a document object. Jsoup generally supports conversions of the following source content.

Parsing an HTML string

Parse a body fragment

Load a Document object based on a URL address

Load a Document object from a file

(i) parsing an HTML string

Processing an HTML string. We might need to parse it, extract its contents, or verify that its format is complete, or that you want to modify it. Jsoup can help us solve these problems with ease.

There is a static method Jsoup.parse (String html) in Jsoup that converts our HTML fragment to a Document object. Examples are as follows:

Help

1 2	String html = "<div><p align=\" Center\ "> This is the content of P elements </p>"; Document document = Jsoup.parse (HTML);

Using the above method, you can convert an HTML string into a Document object, and once the Document object is available, we can use the appropriate method to handle the problem as needed. We can see here that the translated HTML fragment is not a legitimate HTML fragment, the DIV tag inside is not closed. This is not a problem for jsoup, it can deal with this kind of problem very well.

(ii) Analysis of body fragments

If we now have an HTML fragment (for example, a div containing a pair of P tags; an incomplete HTML document) want to parse it. This HTML fragment can be a comment submitted by a user or edit the body section in a CMS page. We can use the Jsoup.parsebodyfragment (String html) method.

Examples are as follows:

Help

1 2	String html = "<div><p align=\" Center\ "> This is the content of P elements </p>"; Document document = Jsoup.parsebodyfragment (HTML);

There may be doubt here, this is the same as the HTML fragment above. Yes, the same thing. The Parsebodyfragment method creates an empty shell document and inserts the parsed HTML into the BODY element. If you use the normal jsoup.parse (String html) method, you can usually get the same result, but explicitly treat user input as a body fragment to ensure that any poor HTML provided by the user will be parsed into the BODY element.

The Document.body () method can get all the child elements of the BODY element of the document, the same as Doc.getelementsbytag ("body").

(iii) Loading the Document object according to a URL address

Sometimes we might want to use a URL address and then extract the contents and convert it to a document object. We may have used the HTTP client to simulate a request, then get the return content, and so on, using Jsoup to easily solve the problem. Examples are as follows:

Help

1 2 3

Document document = Jsoup.connect ("http://www.baidu.com"). get (); String title = Document.title (); String text = Document.text ();

The Connect (String URL) method creates a new Connection, and get () obtains and analyzes an HTML file. If an error occurs when getting HTML from this URL, the IOException is thrown and should be handled appropriately.

The Connection interface also provides a method chain to resolve specific requests, as follows:

Help

1	Document doc = Jsoup.connect ("http://test.com"). Data ("Query", "Java"). useragent ("Mozilla"). Cookies ("auth", "token"). Timeout (3000). Post ();

You can send the link address post parameters, set Useragent,cookie,timeout, and so on, and here is the use of the link operation is very convenient (familiar with jquery should be familiar with this link operation).

(iv) document loading from documents

Sometimes the HTML content we have to deal with may be in a file on the hard drive, we need to extract or parse something out of it, and we can do this through Jsoup. The sample code is as follows:

Help

1 2	File input =newfile ("d:/input.html"); Document doc = jsoup.parse (input, "UTF-8", "http://test.com/");

See here may have a doubt, the first parameter is a file, the second is coding, what is the third one? The third parameter is BaseURL, using him we can easily handle the relative path problem, if you do not need to pass, this is a polymorphic method, in the previous three parts, You can add one such baseurl, which will be described in detail later.

Third, data extraction

(i) traversing a document using a DOM method

In Chapter Two we can get a document object that we can use to traverse the documentation, such as:

Help

1 2 3 4 5 6 7

Document doc = jsoup.parse (input, "UTF-8", "http://test.com/"); Element content = Doc.getelementbyid ("content"); Elements links = Content.getelementsbytag ("a"); for (Element link:links) { String linkhref = link.attr ("href"); String LinkText = Link.text (); }

Here we can easily use the Doument object method to get the content. Common methods are as follows:

Find elements

getElementById (String ID)

Getelementsbytag (String tag)

Getelementsbyclass (String className)

Getelementsbyattribute (String key) (and related methods)

Element siblings:siblingelements (), firstelementsibling (), lastelementsibling (); nextelementsibling (), Previouselementsibling ()

Graph:parent (), children (), child (int index)

Element data

attr (string key) Gets the property attr (string key, String value) Set property

Attributes () Get all properties

ID (), className () and Classnames ()

Text () Get textual content (String value) to set text content

HTML () Gets the HTML content within the htmlhtml (String value) setting element within the element

outerHTML () Get HTML content outside the element

Data (for example: script and Style labels)

Tag () and TagName ()

Manipulating HTML and text

Append (string html), prepend (string html)

AppendText (string text), Prependtext (string text)

Appendelement (String tagName), Prependelement (string tagName)

HTML (String value)

(ii) Using selectors to find elements

When using jquery, we all marvel at its powerful selector, and Jsoup has the same powerful selector that allows us to process documents. The sample code is as follows:

Help

1 2 3 4 5 6

Elements links = doc.select ("a[href]"); With <span style= "text-decoration:underline;" A element of >href</span> property Elements PNGs = Doc.select ("img[src$=.png]"); The extension is. <span style= "Text-decoration:underline;" >png</span> Pictures Element masthead = Doc.select ("Div.masthead"). Class equals <span style= "text-decoration:underline;" >masthead</span> <span style= "text-decoration:underline;" >div</span> Label Elements resultlinks = Doc.select ("H3.R > A");//A element after the H3 element

The Jsoup elements object supports selector syntax similar to CSS (or jquery) to achieve very powerful and flexible lookup capabilities.

This select method can be used in document, Element, or elements objects. And is context-sensitive, you can either implement filtering for the specified element or chain-select access.

The Select method returns a elements collection and provides a set of methods to extract and process the results.

(iii) Extraction of attributes and documents from elements

The general method for extracting properties using Jsoup is as follows:

To get the value of a property, you can use the Node.attr (String key) method

For text in an element, you can use the Element.text () method

For HTML content in an element or attribute, you can use the element.html (), or the node.outerhtml () method

Examples are as follows:

Help

01 02 03 04 05 06 07 08 09 10 11

String html = "<p>an <a href= ' http://example.com/' ><b>example</b></a> link.</p>" ; Document DOC = Jsoup.parse (HTML);//Parse HTML string returns a document implementation Element link = doc.select ("a"). First ();//Find </pre> string text = Doc.body (). text ();//"An example link"//get the literal in the string String linkhref = link.attr ("href");//"http://example.com/"//Get link address String LinkText = Link.text ();//"Example"//Get text in link address </pre> String Linkouterh = link.outerhtml (); "<a href=" http://example.com "><b>example</b></a>" String Linkinnerh = link.html ();//"<b>example</b>"//Get HTML content within a link

(iv) URL processing

When dealing with HTML content, we may often encounter this problem, we need to convert the link address in the HTML page from the relative address to the absolute address, Jsoup has a method to solve this problem. The BaseURL we faced before was used to solve the problem. The sample code is as follows:

Help

1 2 3 4 5

Document doc = Jsoup.connect ("http://www.baidu.com/"). get (); Element link = doc.select ("a"). String relhref = link.attr ("href");/= = "/" String abshref = link.attr ("Abs:href"); "Http://www.baidu.com/gaoji/preferences.html"

In HTML elements, URLs often write relative paths relative to the location of the document: <a href= "/download" >...</A>. When you use the Node.attr (String key) method to obtain the href attribute of a element, it returns directly to the specified value in the HTML source code.

If you need to get an absolute path, you need to add ABS: prefix to the attribute name. This allows you to return the URL address attr ("Abs:href") that contains the root path.

Therefore, it is important to define the base URI when parsing an HTML document. If you do not want to use ABS: prefix, there is also a way to achieve the same function Node.absurl (String key).

Iv. Data modification

(i) Setting property values

When dealing with HTML, we may sometimes need to modify the attribute values inside, such as Picture address, class name, and so on.

You can use the property setting method Element.attr (String key, String value), and elements.attr (string key, String value).

If you need to modify the class attribute of an element, you can use the Element.addclass (string className) and Element.removeclass (String className) methods.

Elements provides a way to bulk manipulate element attributes and classes, such as adding a rel= "nofollow" to each of the a elements in a div.
You can use the following methods:

Help

1	Doc.select ("Div.comments a"). attr ("rel", "nofollow");

The Jsoup method here also supports linking operations, as follows:

Help

1	Doc.select ("Div.masthead"). attr ("title", "Jsoup"). AddClass ("Round-box");

(ii) Set the HTML content of the element

We need to add content such as HTML snippets to HTML to do the following:

Help

1 2 3 4 5 6 7 8 9 [

Element div = doc.select ("div").;// <div></div> Div.html (" <p>lorem ipsum</p> ");// <div><p>lorem ipsum</p></div> Div.prepend ("<p>First</p>");//Add HTML content before div Div.append ("<p>Last</p>");//Add HTML content after div //  After the result:  <div><p>first</p><p>lorem ipsum</p><p>last </p></div> Element span = doc.select ("span").// <span>one </span> Span.wrap ("<li><a href= ' http://example.com/' ></a></li>"); //The result of adding an external HTML content to the element package: //<li><a href= "http://example.com" ><span>One</span></a></li>

(iii) Set the textual content of the element

If we need to modify the text content within the element, we can do the following:

1 2 3 4 5

Element div = doc.select ("div")./<div></div> Div.text ("Five > Four");//<div>five > Four</div> Div.prepend ("a"); Div.append ("last"); Now: <div>first five > Four last</div>

Description

The text setting method is the same as the HTML setter method:

Element.text (String text) clears the internal HTML content from an element and then provides the text instead

Element.prepend (String a) and Element.append (string last) Add text nodes before and after the inner HTML of the element.

If the incoming text contains characters such as, >, it will be treated as text, not HTML.