Java Call PHANTOMJS collection Ajax load generated Web page

Java Call Phantomjs Collection Ajax loading generated Web page a few days ago, when I put all the corresponding pages of the link to the hand, ready to start according to the link to collect (write crawler crawl) corresponding to the terminal page, found that the data obtained by the program has no corresponding content, But my browser see the content is clearly there, so the browser to view the source code also found that there is no, this time think that the page should be Ajax loaded.    Children who do not know Ajax can learn web development. There are two ways to capture Ajax-generated content. One is the request to load the page through HTTP observation, and then we imitate the request to get the corresponding content, the second is to imitate the browser behavior to render this page to get the content. I decided to use the second way here, has been playing WebKit, but has been to load the page too wasted resources, at this time learned to have a fun thing phantomjs, this is a command line to operate WebKit of the thing, Then you can also directly in the inside with the JS API to operate the page (of course, my side is more simple to bother with).DownloadAfter Phantomjs, the direct decompression can be used, and then in the path directory to join the PHANTOMJS path (so that directly on the command line can execute the PHANTOMJS command). Next to complete a code, one is to use PHANTOMJS to get the page (using JS writing behavior), one is to use Java to call PHANTOMJS to achieve the role of content, then directly paste the code.

2. Codes.js
4. System = require (' system ')
6. address = system.args[1]; //Get command line the second parameter is then used
8. Console.log (' Loading a Web page ');
10. var page = require (' webpage '). Create ();
12. var url = address;
14. Console.log (URL);
16. Page.open (URL, function (status) {
18. //page is loaded!
20. if (status!== ' success ') {
22. Console.log (' unable to post! ');
24. } Else {
26. //console.log (page.content);
28. //var title = Page.evaluate (function () {
30. //Return document.title;//demo How to use the Jsapi of the page to manipulate the www.oicqzone.com of the page
32. //  });
34. //console.log (title);
38. Console.log (page.content);
40. }
42. Phantom.exit ();
44. });

The above JS code estimates should not be able to read a few ... Next stick Java code!

2. Import Org.apache.commons.io.IOUtils;
6. Import java.io.*;
10. /**
12. * Created with IntelliJ idea.
14. * USER:LSZ
16. * Date:14-4-22
18. * Time: 1:17
20. * Utils for HTTP
22. */
24. Public class Httputils {
26. public static string getajaxcotnent (string url) throws IOException {
28. Runtime RT = Runtime.getruntime ();
30. Process p = rt.exec ("Phantomjs.exe c:/phantomjs/codes.js" +url); Here my codes.js is stored in the PHANTOMJS directory below the C drive.
32. InputStream is = P.getinputstream ();
34. BufferedReader br = new BufferedReader (new InputStreamReader (IS));
36. StringBuffer SBF = new StringBuffer ();
38. String tmp = "";
40. While ((tmp = Br.readline ()) =null) {
42. Sbf.append (TMP);
44. }
46. //system.out.println (sbf.tostring ());
48. return sbf.tostring ();
50. }
54. public static void Main (string[] args) throws IOException {
56. Getajaxcotnent ("http://www.oicqzone.com");
58. }
60. }

In fact, the principle is very simple, is through interprocess communication with Java call PHANTOMJS This component to request rendering page, but this practice because each time to restart the PHANTOMJS process, so slow, there is another kind of directly loaded with Phantomjs page, Post the content to our custom HTTP backend to receive the data is a bit faster.

Java Call PHANTOMJS collection Ajax load generated Web page