Htmlunit Introductory One

Htmlunit is an open Source Java page Analysis tool that allows you to effectively use Htmlunit to analyze content on a page after reading the page.

Projects can emulate the browser run, known as the Java Browser Open source implementation. is a browser with no interface.

The RHINOJS engine is used. Analog JS run.

The use of Htmlunit Crawl Web page can be divided into the following several steps:

1, define a WebClient client.

It is equivalent to defining a browser without an interface.

2. Use the WebClient client to get htmlpage from the specified URL.

HtmlPage contains all the information in the destination URL page.

3. Get the specified element we need from HtmlPage.

Here's a look at an example:

 PackageCom.fuwh;Importcom.gargoylesoftware.htmlunit.WebClient;ImportCom.gargoylesoftware.htmlunit.html.HtmlPage; Public classDemo01 { Public Static voidMain (string[] args) {WebClient WebClient=NULL; Try{webClient=NewWebClient ();//define a default WebClientHtmlPage page=webclient.getpage ("https://www.cnblogs.com/");//get htmlpage from the specified URLSystem.out.println (Page.astext ());//convert HtmlPage into a string to print out}Catch(Exception e) {//Todo:handle ExceptionE.printstacktrace (); }finally{webclient.close (); //shutting down the client        }    }}

In the above example, we create a default WebClient instance so that the Webclient#getbrowserversion () method can be seen,

The chrome version of the browser is created by default.

Of course, we can also specify the version of the browser at the time of creation.

Example:

 PackageCom.fuwh;Importcom.gargoylesoftware.htmlunit.BrowserVersion;Importcom.gargoylesoftware.htmlunit.WebClient;ImportCom.gargoylesoftware.htmlunit.html.HtmlPage; Public classDemo01 { Public Static voidMain (string[] args) {WebClient WebClient=NULL; Try{webClient=NewWebClient (browserversion.firefox_45);//Define a WebClientHtmlPage page=webclient.getpage ("https://www.cnblogs.com/");//get htmlpage from the specified URLSystem.out.println (Page.astext ());//convert HtmlPage into a string to print out}Catch(Exception e) {//Todo:handle ExceptionE.printstacktrace (); }finally{webclient.close (); //shutting down the client        }    }}

In Browserversion, many versions of browsers are defined.

After getting a htmlpage, we still prefer to find the element we want, relative to the entire page.

Htmlunit also provides rich support for finding the specified element.

Supports the use of DOM,CSS and XPath (recommended) methods.

◇ using Dom method :

 PackageCom.fuwh;Importcom.gargoylesoftware.htmlunit.BrowserVersion;Importcom.gargoylesoftware.htmlunit.WebClient;Importcom.gargoylesoftware.htmlunit.html.DomElement;Importcom.gargoylesoftware.htmlunit.html.DomNodeList;ImportCom.gargoylesoftware.htmlunit.html.HtmlAnchor;Importcom.gargoylesoftware.htmlunit.html.HtmlDivision;ImportCom.gargoylesoftware.htmlunit.html.HtmlPage; Public classDemo01 { Public Static voidMain (string[] args) {WebClient WebClient=NULL; Try{webClient=NewWebClient (BROWSERVERSION.FIREFOX_45);//Define a WebClient            FinalHtmlPage page=webclient.getpage ("https://www.cnblogs.com/");//get htmlpage from the specified URL                        /*** DomElement Subclass: HtmlElement * Htmlelemnt also has a lot of subclasses, basically covering all the HTML elements * For example: htmldivision , Htmlinput*/System.out.println ("============================================="); //gets the specified DOM element by IDHTMLDivision htmldiv= (htmldivision) Page.getelementbyid ("header");            System.out.println (Htmldiv.asxml ()); System.out.println ("============================================="); //to get the collection of elements through TagNameDomnodelist<domelement> Nodelist=page.getelementsbytagname ("a");  for(DomElement domelement:nodelist) {HtmlAnchor HtmlAnchor=(HtmlAnchor) domelement; System.out.println ("Title:" +htmlanchor.astext () + "-Address:" +htmlanchor.getattribute ("href")); }                    } Catch(Exception e) {//Todo:handle ExceptionE.printstacktrace (); }finally{webclient.close (); //shutting down the client        }    }}

◇ Use CSS method :

 PackageCom.fuwh;Importcom.gargoylesoftware.htmlunit.BrowserVersion;Importcom.gargoylesoftware.htmlunit.WebClient;Importcom.gargoylesoftware.htmlunit.html.DomElement;Importcom.gargoylesoftware.htmlunit.html.DomNodeList;ImportCom.gargoylesoftware.htmlunit.html.HtmlAnchor;Importcom.gargoylesoftware.htmlunit.html.HtmlDivision;ImportCom.gargoylesoftware.htmlunit.html.HtmlPage; Public classDemo02 { Public Static voidMain (string[] args) {WebClient WebClient=NULL; Try{webClient=NewWebClient (BROWSERVERSION.FIREFOX_45);//Define a WebClient            FinalHtmlPage page=webclient.getpage ("https://www.cnblogs.com/");//get htmlpage from the specified URLhtmldivision Htmldiv=page.queryselector ("div");//get the first divSystem.out.println (Htmldiv.asxml ()); System.out.println ("===================================="); HTMLDivision HtmlDiv2=page.queryselector ("Div#footer_bottom");//You can also specify multiple selectors, separated by ', 'System.out.println (Htmldiv2.asxml ()); } Catch(Exception e) {//Todo:handle ExceptionE.printstacktrace (); }finally{webclient.close (); //shutting down the client        }    }}

◇ Use XPath method :

 PackageCom.fuwh;Importjava.util.List;Importcom.gargoylesoftware.htmlunit.BrowserVersion;Importcom.gargoylesoftware.htmlunit.WebClient;Importcom.gargoylesoftware.htmlunit.html.HtmlDivision;ImportCom.gargoylesoftware.htmlunit.html.HtmlPage; Public classDemo03 { Public Static voidMain (string[] args) {WebClient WebClient=NULL; Try{webClient=NewWebClient (BROWSERVERSION.FIREFOX_45);//Define a WebClient            FinalHtmlPage page=webclient.getpage ("https://www.cnblogs.com/");//get htmlpage from the specified URLList<HtmlDivision> Divlist=page.getbyxpath ("//div[@id = ' cnblogs_a1 ']");  for(HTMLDivision htmldivision:divlist) {System.out.println ("***********************************************8");            System.out.println (Htmldivision.asxml ()); }                    } Catch(Exception e) {//Todo:handle ExceptionE.printstacktrace (); }finally{webclient.close (); //shutting down the client        }    }}

Htmlunit Introductory One