Capturing JS Dynamic Data generation

Recently in the grasp of data, general Web data capture is relatively easy, today in the TV cat's program, found that some data can not be crawled, the Java end of the HTML file does not have a piece of code, looked up a lot of information, found that JS Dynamic Data generated, can not be directly crawled, There is a workaround is to use to find the AJAX request address and parameters, re-fetch, the method has a problem, that is, when the parameters are encrypted obsolete, the method is not good use, so, here is a way to use Htmlunit to crawl (can use JSUOP to handle the HTML file), Jar Package: http://download.csdn.net/detail/joy460154849/9822849 (the jar package used by htmlunit all imported, slightly adjusted, because some have duplicate import or version problem) code is as follows:

 Public Staticstring gethtml (string url) {//Simulate a browser@SuppressWarnings ("Resource") WebClient WebClient=NewWebClient (); //webclient parameter carrierWebclientoptions clientoptions =webclient.getoptions (); //set parameters for WebClientClientoptions.setjavascriptenabled (true); Clientoptions.setcssenabled (false); Webclient.setajaxcontroller (NewNicelyresynchronizingajaxcontroller ()); Clientoptions.settimeout (35000); Clientoptions.setthrowexceptiononscripterror (false); //simulate browser to open a destination URLHtmlPage Rootpage =NULL; Try{rootpage=webclient.getpage (HTML); } Catch(failinghttpstatuscodeexception e) {//TODO auto-generated Catch blockE.printstacktrace (); } Catch(malformedurlexception e) {//TODO auto-generated Catch blockE.printstacktrace (); } Catch(IOException e) {//TODO auto-generated Catch blockE.printstacktrace (); }            //Save page//Rootpage.save (New File ("/HOME/SHARE/TEST/1")); //Body HTML InformationHtmlElement HtmlElement =Rootpage.getbody (); String xmlcontent=Htmlelement.asxml (); returnxmlcontent; }  

Capturing JS Dynamic Data generation