Java uses Htmlunit to crawl JS rendering pages

Demand:

Need to collect JS rendering of the page, some Web pages are JS rendering

Realize:

Based on the Htmlunit implementation:

2. Public static void Getajaxpage () throws exception{
4. WebClient WebClient = new WebClient ();
6. Webclient.setjavascriptenabled (true);
8. Webclient.setcssenabled (false);
10. Webclient.setajaxcontroller (new Nicelyresynchronizingajaxcontroller ());
12. Webclient.settimeout (Integer.max_value);
14. Webclient.setthrowexceptiononscripterror (false);
16. HtmlPage rootpage = webclient.getpage ("http://tt.mop.com/read_14304066_1_0.html");
20. System.out.println (Rootpage.asxml ());
22. }

Maven dependencies:

2. <dependency>
4. <groupId>net.sourceforge.htmlunit</groupId>
6. <artifactid>htmlunit-core-js</artifactid>
8. <version>2.9</version>
10. <scope>compile</scope>
12. </Dependency>
14. <dependency>
16. <groupId>net.sourceforge.htmlunit</groupId>
18. <artifactid>htmlunit</artifactid>
20. <version>2.9</version>
22. <scope>compile</scope>
24. </Dependency>

Description

Nutch plugin: nutch-htmlunit to replace the HTTP fetch component of the Nutch itself

Java uses htmlunit to crawl JS rendered pages