Crawling pages with Jsoup crawlers

HttpClient and Jsoup can be implemented to simulate the browser crawl page, the former sends the request, the latter parsing the HTM tag is more powerful. This example uses Jsoup to implement requests and parsing directly.

 PackageCom.chongdong.log.test;ImportJava.io.IOException;ImportJava.util.HashMap;ImportJava.util.Map;ImportOrg.jsoup.Connection;ImportOrg.jsoup.Connection.Method;ImportOrg.jsoup.Jsoup;ImportOrg.jsoup.helper.HttpConnection.Response;ImportOrg.jsoup.nodes.Document;Importorg.jsoup.select.Elements;ImportOrg.junit.Test;/** * Class Name: Jsouptest * Class Description: Jsoup crawl Mitsuku Chat information * Created by: ZK * created in: 2015-7-20 3:52:06 * Modified by: ZK * Modified: 2015 -7-20 3:52:06 * Revision NOTES: * Development progress: * @version 1.0 * * * * Public  class jsouptest {     Public  Static  void  Postmitsuku() {map<string, string> Map =NewHashmap<string, string> ();/** Form submission parameters * Input:how old is you botid:9fa364f2fe345a10 custid:c04f62ad1e044059 FAQ:HTTP://WWW.PANDORABOTS.COM/BOTMASTER/EN/FAQ#H1 * **/Map.put ("Input","HI"); Map.put ("Botid","9FA364F2FE345A10");//open to get markers in the Jump to chat box interfaceMap.put ("CustID","c04f62ad1e044059");//When an empty request can be obtained equivalent to the markConnection conn = Jsoup.connect ("Http://fiddle.pandorabots.com/pandora/talk-xml");/* Conn.header ("(Request-line)", "Post/cgi-bin/login?lang=zh_cn http/1.1"); */Conn.header ("Accept","*/*"); Conn.header ("Accept-encoding","GZIP,DEFLATE,SDCH"); Conn.header ("Accept-language","zh-cn,zh;q=0.8");/* Conn.header ("content-length", "+"); */Conn.header ("Origin","http://www.square-bear.co.uk"); Conn.header ("Pragma","No-cache"); Conn.header ("Connection","Keep-alive");//must fill out form submissionConn.header ("Content-type","application/x-www-form-urlencoded"); Conn.header ("Host","Fiddle.pandorabots.com"); Conn.header ("Referer","http://www.square-bear.co.uk/mitsuku/mitsy_retro.swf"); Conn.header ("User-agent","mozilla/5.0 (Windows NT 6.1; WOW64) applewebkit/537.36 (khtml, like Gecko) chrome/36.0.1985.125 safari/537.36 ");Try{Response Response = (Response) conn.ignorecontenttype (true). Method (Method.post). Data (map). Execute ();//response.String Json=response.body ();        SYSTEM.OUT.PRINTLN (JSON); }Catch(IOException e) {//TODO auto-generated catch blockE.printstacktrace (); }    }/////////////////////////////////////////////////////////////////////////////////////////    //Scenario 2: By developing the capture tool, we know that the form should be submitted in an HTTP post where the Get method is inappropriate    /** * Request an English conversation page, crawl results * @param URL * @return  * *        Private StaticStringProcesslogic(String URL) {String result ="";Try{Document document = Jsoup.connect (URL). Ignorecontenttype (true). Ignorehttperrors (true). Followredirects (true). Timeout ( the). useragent ("mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; trident/4.0;) "). get (); Elements Elements = Document.select ("result that");                    result = Elements.text ();            SYSTEM.OUT.PRINTLN (result); }Catch(Exception e)                {E.printstacktrace (); result ="OK"; }returnResult } Public Static void Main(string[] args) { for(inti =0; I < -; i++) {//thread thread=new thread ();            //thread.start ();Postmitsuku (); }/ * Through the development of the clutch tool, we know that the form should be submitted in an HTTP post here for the Get method is not appropriate String URL = "Http://fiddle.pandorabots.com/pand        Ora/talk-xml?input=%s&botid=9fa364f2fe345a10&custid=bbbb30debe1bc7f7 ";        Processlogic (URL); */}}

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Crawling pages with Jsoup crawlers