Some learning experiences of httpclient

Turn: http://wallimn.iteye.com/blog/540566 recently engaged in a project to understand the next httpclient, summed up here, and everyone to learn to share, I hope that friends to put forward valuable advice.

First, introduce the background of the project:
Goal: The domestic insurance Company's "Web one account Pass" to "WAP one account Pass".
Resource: The customer does not provide any webservice interfaces.

The third party component used in this project is the Apache HttpClient, a very powerful web crawler (it may not be good to catch this word), and here
Discuss some of the common uses of the httpclient and the places to be noted.

List of resources referenced in this article:

HttpClient Getting Started: http://www.ibm.com/developerworks/cn/opensource/os-httpclient/
HttpClient Certificate Import: http://www.blogjava.net/happytian/archive/2006/12/22/89447.html
HttpClient Advanced Knowledge: http://laohuang.iteye.com/blog/55613
HttpClient Official Document: http://hc.apache.org/httpcomponents-client/index.html
HttpClient Resource shutdown: http://www.iteye.com/topic/234759


The above article is very good, after reading also know how to use httpclient this very good tool, but here or add some more important things, is also the project
Problems that are often encountered.

The first thing to pay attention to are the following points:
1, httpclient connectivity after the issue of resource release is very important, and we use the database connection to release the same resources.
2, the HTTPS website uses the SSL encryption transmission, the certificate import should notice.
3, do such a project better understand the next HTTP semantics, such as 302,301,200,404 return code meaning (this is the most basic), cookie,session mechanism.
4. The redirect State of the httpclient is automatic by default, which greatly facilitates the developer (such as some authorization to obtain cookies), but sometimes it is managed manually, such as
Sometimes a circularredirectexception exception is encountered, which occurs because the location value in the returned header file points to the previously repeated (port number can be different) address, causing a possible death
Circular recursive redirection, which can be manually turned off: Method.setfollowredirects (false)
5, some sites will first determine whether the user's request is from the browser, if not, then return the incorrect text, so use httpclient crawl information in the head to add the following information:
Header.put ("User-agent", "mozilla/4.0" (compatible; MSIE 6.0; Windows NT 5.1; SV1; Qqdownload 1.7;. NET CLR 1.1.4322; CIBA. NET CLR 2.0.50727) ");
6, when the POST request to submit data to change the default encoding, otherwise submitted to the data will appear garbled. The Setcontentcharset () method of rewriting Postmethod is OK:






A generic class is written below to handle the text returned by the request requests:
Java code   /*   * HttpRequestProxy.java   *   * created on  November 3, 2008, 9:53 AM   */      package  cn.com.mozat.net;      import java.io.bufferedreader;   import  java.io.ioexception;   import java.io.inputstream;   import  java.io.inputstreamreader;   import java.util.hashmap;   Import java.util.Iterator ;   import java.util.map;   import java.util.set;      Import  org.apache.commons.httpclient.Header;   import org.apache.commons.httpclient.httpclient;    import org.apache.commons.httpclient.httpexception;   import  org.apache.commons.httpclient.httpmethod;   import org.apache.commons.httpclient.namevaluepair;    import org.apache.commons.httpclient. simplehttpconnectionmanager;   import org.apache.commons.httpclient.methods.getmethod;    import org.apache.commons.httpclient.methods.postmethod;      import  cn.com.mozat.exception.customexception;     /**   *    * @ author bird  email:lihongfu-84@163.com   *   * 2008-11-4   09:49:48   */   public class httprequestproxy{        Timeout interval        private static int connectTimeOut = 60000;     //ConnectionManager to turn off connection        private when managing Httpclientconnection  static boolean alwaysClose = false;    //return Data encoding format         private String encoding =  "UTF-8";           &NBSp;   private final httpclient client = new httpclient (new  Simplehttpconnectionmanager (alwaysclose));           public  Httpclient gethttpclient () {           return client ;       }                 /**       *  usage:       *  Httprequestproxy hrp = new httprequestproxy ();       *  Hrp.dorequest ("http://www.163.com", Null,null, "GBK");       *         *  @param  url   Requested resources url       * @ Param postdata  post the form's encapsulated data   No newsletters null  when requested      * @ Param heHeader information (header) included with Ader   request request   No newsletters null       *  @param  encoding response return information encoding format   No newsletters null       *  @return    response back text data        *  @throws  CustomException         */       public string dorequest (string  url,map postdata,map header,string encoding)  throws CustomException{        String responseString = null;         //Header Request Information         Header[] headers = null;        if (header != null) {         set  entryset = header.entryset ();             int datAlength = entryset.size ();              headers= new header[datalength];             int i = 0;            for (Iterator  Itor = entryset.iterator () Itor.hasnext ();) {              Map.Entry entry =  (Map.entry) itor.next ();              headers[i++] = new header (Entry.getKey (). toString (), Entry.getvalue (). toString ());