Htmlunit crawling Ajax dynamically generated page content

　　Htmlunit Plainly is a browser, this browser is written in Java without interface browser, because it has no interface, so the speed of execution can be dropped.

　　Htmlunit provides a range of APIs that can be used to do more functions, such as filling out forms, submitting forms, mimicking click links, and because of the built-in Rhinojs engine, so you can performJavaScript


The previous use of the time has not been able to crawl Ajax dynamically generated table and its data, Firefox and Internet Explorer to view the source code can not see the table and data, but with Firefox to view the elements can see the information

Research Htmlunit for a long time no use, and later updated the Htmlunit jar package version, incredibly can. It used to be htmlunit2.14, and now it's in version 2.20.

Next is the table data that I crawled from a website Ajax dynamically generated. The process is: fill in the relevant query keywords on the first page, and then simulate the Click Query button to get table data

 Public classcrawlerxxx {Private StaticLogger log = Logger.getlogger (crawlerxxx.class);  Public Static voidMain (string[] args) {WebClient WC=NewWebClient (browserversion.best_supported); Wc.getoptions (). SetTimeout (10000);//Set the connection timeout time, here is 10S. If it is 0, wait indefinitelyWc.getoptions (). setjavascriptenabled (true);//Enable JS interpreter, default to TrueWc.setjavascripttimeout (10000);//set the time-out for JS executionWc.getoptions (). setcssenabled (false);//Disabling CSS supportWc.getoptions (). Setthrowexceptiononscripterror (false);//JS Run error, whether to throw an exceptionWc.getoptions (). setredirectenabled (true); Wc.getoptions (). Setuseinsecuressl (true); Try{htmlpage MainPage= Wc.getpage ("xxx"); HtmlForm form= Mainpage.getformbyname ("xxx"); Htmltextinput TextField= Form.getinputbyname ("Varghost"); Htmlsubmitinput Button= Form.getinputbyname ("xxx"); Textfield.setvalueattribute ("XXX"); HtmlPage Clickedpage=Button.Click (); Try{Thread.Sleep (10000); } Catch(interruptedexception e) {e.printstacktrace (); }            if(Clickedpage.asxml (). Contains ("OK") | | mainpage.asxml (). Contains ("OK") {Document doc=Jsoup.parse (Clickedpage.asxml ()); Element Table= Doc.getelementbyid ("pingtable"); Elements TRS= Table.getelementsbytag ("tr"); intLen =trs.size ();  for(inti = 1; i < Len; i++) {Element TD= Trs.get (i). Getelementsbytag ("TD"). Last (); String XXX=Td.text (); if(Xxx.length () > 0) {System.out.println ("XXX" +i+ "=" +xxx); }                }            }        } Catch(Failinghttpstatuscodeexception |IOException E)        {E.printstacktrace (); }    }}

Htmlunit crawling Ajax dynamically generated page content