When you crawl the data of a microblog, someone else uses the Fm.view method to pass the HTML tag, so how does Jsoup parse it?

Using Jsoup, here's a little idea.

I only did my own home page and other people's micro-blog crawl other crawl did not try (sorry more lazy ...) ）

The first is to use the Jsoup to access the page to see the microblog landing form found in the Ajax way, so the code to get a cookie is a bit difficult

So I stole a lazy. Use IE developer tools to obtain the cookie to be written in the form of a map and then use the code:

Java code

1. Response res=jsoup.connect ("http://weibo.com"). Cookies (Map). Method (Method.post). Execute ();
2. String S=res.body ();

Got the next found quite a lot of:

You can write your own script to print map.put (xxx,xxx)

I use Scala to write a paragraph in Java write the same matter:

Scala code

1. S.split (";"). foreach (s = = {val X=s.split ("=");p rintln (S "" "" Map.put ("${x (0)}", "${x (1)}"); " "")});  

Finally get the body well ... It's a bunch of script tags. Top of the list is the contents of the fixed top column of the microblog (the contents of the navigation bar)

LZ tried to find what is needed is <script>fm.view in an ID of pl_content_homefeed he is the content of the home page

Then the LZ made a simple processing without using the regular because .... The amount ... Bad writing:

Java code

2. String S=res.body ();
4. //system.out.println (s);
6. String[] Ss=s.split ("<script>fm.view");
8. int i=0;
10. //pl_content_homefeed
12. for (String x:ss) {
14. System.out.println (i++ + "======================================");
16. System.out.println (x.substring (0, X.length () >100?100:x.length ()));
18. System.out.println ("===========================================");
20. //        }
22. String content=ss[8].split ("\" html\ ": \" ") [1].replaceall (" \\\\n ", " "). ReplaceAll (" \\\\t ",  " "). ReplaceAll ("\\\\", "" ");
24. Content=content.substring (0, Content.length () <=13?content.length (): Content.length ()-13);
26. SYSTEM.OUT.PRINTLN (content);

The content of the output is the micro-blog contents displayed on the homepage

However, this output Unicode is not translated into Chinese characters need to use the Native2ascii tool to find a Web:

http://soulshard.iteye.com/blog/346807

Measurements can be used:

Java code

1. System.out.println (native2asciiutils.ascii2native (content));

Note that the above code LZ is fixed on the homepage, so in the interception of the direct use of index 8

Change the Post method to get method can also get to other people's Weibo page

Then give a way to print out all of the HTML content you've got (try some of the pages available):

Java code

2. Package jsouptest;
6. Import java.io.IOException;
8. Import java.util.ArrayList;
10. Import Java.util.HashMap;
12. Import java.util.List;
14. Import Java.util.Map;
18. Import Org.jsoup.Connection.Method;
20. Import Org.jsoup.Connection.Response;
22. Import Org.jsoup.Jsoup;
26. Public class Jsouptest {
28. public static void Main (string[] args) throws IOException {
30. map<string, string> map = new hashmap<> ();
32. //map.put Please follow your Weibo cookie to get
36. Response res = jsoup.connect ("http://weibo.com/u/someone else's homepage id")
38. . Cookies (Map). Method (Method.get). Execute ();
40. String s = res.body ();
42. System.out.println (s);
44. string[] ss = S.split ("<script>fm.view");
46. int i = 0;
48. //Pl_content_homefeed
50. //Pl.content.homeFeed.index
52. list<string> list = new arraylist<> ();
54. For (String x:ss) {
56. System.out.println (i++ + "======================================");
58. System.out.println (x.substring (0,
60. X.length () > 200? 200:x.length ()));
62. System.out.println ("===========================================");
64. if (x.contains ("\" html\ ": \")) {
66. String value = gethtml (x);
68. List.add (value);
70. System.out.println (value);
72. }
76. }
78. //Content=ss[8].split ("\" html\ ": \" ") [1].replaceall (" (\\\\t|\\\\n) ",
80. //""). ReplaceAll ("\\\\\" "", "\" "). ReplaceAll (" \\\\/","/");
82. //content=content.substring (0,
84. //Content.length () <=13?content.length (): Content.length ()-13);
86. //System.out.println (native2asciiutils.ascii2native (content));
88. }
92. public static string gethtml (string s) {
94. String content = s.split ("\" html\ ": \" ") [1]
96. . ReplaceAll ("(\\\\t|\\\\n|\\\\r)", " " "). ReplaceAll (" \\\\\ "", " \" ")
98. . ReplaceAll ("\\\\/", "/");
100. Content = Content.substring (0,
102. Content.length () <= content.length ()
104. : Content.length ()- 13);
106. return native2asciiutils.ascii2native (content);
108. }
110. }

The contents of the crawl should be properly formatted before you can use Jsoup to parse

But there's nothing wrong with trying to do the parsing directly (though there are some wrong labels)

It's just a page crawl strategy. I don't want to write more about it. If you use your own Sina Weibo cookie to crawl

When you crawl the data of a microblog, someone else uses the Fm.view method to pass the HTML tag, so how does Jsoup parse it?