Use Jsoup crawl Baidu Network Disk resources sharing connection (multi-threaded)

suddenly one day want to say can use some kind of method to share the resources of Baidu network to crawl down, so the hands. It is said that the best way to know is Http://pan.baidu.com/wap Crawl, a look at the link behind the UK value is a series of numbers, thought can assemble their own links, recycling constantly to extract pages. So I analyzed the next page structure, began. (UK is a 10-digit number, for 0000000001 This UK value has not yet thought of a good solution, for 1000000000 This can be used uk++ to achieve continuous circulation.) )
From the beginning of writing, found a second to extract a page, think of the previous use of the WebMagic crawler crawl page using Java multithreading technology, so Baidu ... directly on the code.

Package Getbaiduyunurl;import Java.io.file;import Java.io.filewriter;import java.io.ioexception;import Java.io.printwriter;import Org.jsoup.jsoup;import Org.jsoup.nodes.document;import Org.jsoup.nodes.Element;import Org.jsoup.select.elements;class Mythread implements runnable{private long count= 1;public void Run () {try{for (This.cou nt=1800011339;this.count<1813454114;this.count++) {PrintWriter printwriter = new PrintWriter (New FileWriter (new File ("d:/self-taught/baiduyundata/data/url.txt"), true); String url= "http://pan.baidu.com/wap/share/home?uk=" +this.count; Document doc = jsoup.connect (URL). Timeout (60000). get (); String title = Doc.title (); System.out.print (title); Element content = Doc.body (); Elements dataems=content.select ("[data-ac=active]"); System.out.println (This.count); for (Element dataem:dataems) {Elements lists=dataem.select (". List-item"); String name=dataem.attr ("Data-fn"); if (name!= "") {System.out.println (name); printwriter.println (name); for (Element listem:lists) {String Linkhref = url+listem.attr ("href"); Printwriter.println (LINKHREF); System.out.println (LINKHREF); }}} printwriter.close (); }}catch (IOException e) {//TODO auto-generated catch Blocke.printstacktrace ();}} }public class Multithread {public static void main (String args[]) {mythread my = new Mythread (); for (int i=0;i<10;i++) {New Thread (my). Start ();}} }

Code as above, very simple, I put the crawl link in txt, the next step is to make these resources can be online search, and a lot of web search, but I am relatively low.

Use Jsoup crawl Baidu Network Disk resources sharing connection (multi-threaded)