HTML Extraction Tool jsoup

ArticleDirectory

• The main functions of jsoup are as follows:

Jsoup is a Java HTML Parser that can directly parse a URL address and HTML text content. It provides a set of very labor-saving APIs that can be used to retrieve and manipulate data through DOM, CSS, and operations similar to jquery. The current version is 1.7.1.

 

The main functions of jsoup are as follows:

1. parse HTML from a URL, file, or string;

2. Use the Dom or CSS selector to find and retrieve data;

3. HTML elements, attributes, and text can be operated;

Official jsoup Website: Alibaba /.

For the translated Chinese documents, please refer to http://www.open-open.com/jsoup/. these are all good materials for learning jsoup.

The figure shows the jsoup package structure. jsoup first parses an HTML document or HTML segment into a DOM structure. Each element in HTML corresponds to a node in the Dom. Therefore, you can use Dom, CSS, and jquery-like methods to retrieve and manipulate data.

For example, the following is an example of using jsoup to extract web pages:

 1   Import Java.net. url;  2   3   Import  Org. jsoup. jsoup;  4   Import  Org. jsoup. nodes. Document;  5   Import  Org. jsoup. nodes. element;  6   Import  Org. jsoup. Select. elements;  7   8  Public   Class  Jsouptest  9   {  10   11       Public   Static   Void Main (string [] ARGs) Throws  Exception  12   {  13 URL url = New URL ("http://www.guet.edu.cn "); //  Connect to a page  14 Document Doc = jsoup. parse (URL, 3*1000 ); //  Parse and obtain the Document Object  15   16 Elements test = Doc. Select (""); //  Get all a elements on the page  17           For  (Element: test)  18   { 19               //  Element. outerhtml () and element. tostring () have the same effect.  20 System. Out. println ("Link Source code : "+ Element. outerhtml ());  21 System. Out. println ("link address:" + element. ATTR ("href") + "link text :" 22 + Element. Text () + "next SBL:" + Element. nextsibling (). nodename ());  23   ;  24  }  25   26   }  27 }

As for how to obtain all the text content on the page of an HTML document, you can use the above example to make the following modifications:

 Import  Org. jsoup. jsoup;  Import  Org. jsoup. nodes. Document;  Import  Org. jsoup. nodes. element;  Import  Org. jsoup. nodes. node;  Import Org. jsoup. Select. elements;  Public   Class  Loaddocument {  Public   Static   Void Main (string [] ARGs) Throws  Exception {document DOC = Jsoup. Connect ("http://Jsoup.org" ). Get (); String title = Doc. Title (); system. Out. println ( "HTML content of the document :"); System.out.println(doc.html (); system. Out. println ( "Text content :" ); System. Out. println (Doc. Text ());}} 

The result is as follows:

 

Through jsoup, you can easily extract content on the page. more conveniently, use selector-syntax to find elements allows you to develop Extraction Rules to extract target information nodes on the page.

For example:

 1   Import  Java. Io. file;  2   Import  Java. Io. ioexception;  3   4   Import Org. jsoup. jsoup;  5   Import  Org. jsoup. nodes. Document;  6   Import  Org. jsoup. nodes. element;  7   Import  Org. jsoup. Select. elements;  8   9   10   Public   Class  Use_selector_syntax_find_elements { 11   12   13       Public   Static   Void Main (string [] ARGs) Throws  Ioexception  14   {  15           //  File input = new file ("  Http://www.hao123.com//index.html  ");  16 Document Doc = jsoup. Connect ("http://www.hao123.com/index.htm" ). Get ();  17           //  Document Doc = jsoup. parse (input, "gb2312 ","");  18           19 System. Out. println ("-----------------------------------------------" );  20   System. Out. println (DOC );  21 System. Out. println ("-----------------------------------------------" );  22  Paser (DOC );  23   }  24       25       Public   Static   Void  Paser (document DOC)  26   {  27 Elements links = Doc. Select ("A [href]"); //  A with href  28 Elements pngs = Doc. Select ("img?src==.png]");//  IMG with SRC ending. PNG  29 Element masthead = Doc. Select ("Div. masthead"). First (); //  Div with class = masthead  30 Elements resultlinks = Doc. Select ("h3.r> "); //  Direct A after h3  31           32   System. Err. Print (LINKS );  33 System. out. println ("++ ++");  34           35   }  36 }

The jsoup function is the same as htmlparser, but it is more convenient than htmlparser, And the API is more concise.