Java httpclient Usage Summary

HttpClient is a project of Apache: http://hc.apache.org/

Document is better: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/

Here is not verbose, mainly in doing the demo when some of the problems encountered here to summarize:

[citation please specify source http://blog.csdn.net/bhq2010/article/details/9210007]

1. Using connection pooling

Although the HTTP protocol is not connected, but is based on TCP, the underlying still need to establish a connection with the server. For a program that needs to crawl a large number of pages from the same site, the connection pool should be used, or each crawl will establish a connection with the Web site, send requests, get responses, release the connection, on the one hand is not efficient, there is a slight careless will inadvertently neglect the release of certain resources, Causes the site to reject connections (very many sites will reject a large number of connections to the same IP, preventing Dos attacks).

The connection pool routines are as follows:

Schemeregistry schemeregistry = new Schemeregistry () Schemeregistry.register (New Scheme ("http", 80, Plainsocketfactory.getsocketfactory ())); Schemeregistry.register ("https", 443, Sslsocketfactory.getsocketfactory ())); Poolingclientconnectionmanager cm = new Poolingclientconnectionmanager (schemeregistry); cm.setmaxtotal (200); Cm.setdefaultmaxperroute (2); Httphost googleresearch = new Httphost ("research.google.com", 80); Httphost Wikipediaen = new Httphost ("en.wikipedia.org", N); Cm.setmaxperroute (new Httproute (Googleresearch), 30); Cm.setmaxperroute (New Httproute (Wikipediaen), 50);

The role of Schemaregistry is the default port number of the registration agreement. Poolingclientconnectionmanager is a pooled Connection manager, which is the connection pool, Setmaxtotal sets the maximum number of connections for a connection pool, Setdefaultmaxperroute sets each route (http:// hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html#d5e467) on the default number of connections, Setmaxperroute sets the maximum number of connections for a single Web site.

Getting the HTTP client from the connection pool is also very important:

Defaulthttpclient client = new defaulthttpclient (cm);

2. Set HttpClient number of references

HttpClient need to set the appropriate number of parameters, the ability to work better. The default number of references can handle a small amount of crawl work, but finding a suitable set of parameters can often improve the gripping effect in a particular situation. The routines for setting the parameters are as follows:

Defaulthttpclient client = new Defaulthttpclient (cm), Integer sockettimeout = 10000;integer ConnectionTimeout = 10000; Final int retrytime = 3;client.getparams (). Setparameter (Coreconnectionpnames.so_timeout, sockettimeout); Client.getparams (). Setparameter (Coreconnectionpnames.connection_timeout, ConnectionTimeout); Client.getParams (). Setparameter (Coreconnectionpnames.tcp_nodelay, false); Client.getparams (). Setparameter ( Coreconnectionpnames.socket_buffer_size, 1024 * 1024); Httprequestretryhandler Myretryhandler = new Httprequestretryhandler () {@Overridepublic Boolean retryrequest ( IOException exception, int executioncount, HttpContext context) {if (Executioncount >= retrytime) {//Do not retry if Ove R Max retry Countreturn false;} If (Exception instanceof Interruptedioexception) {//Timeoutreturn false;} If (Exception instanceof Unknownhostexception) {//Unknown Hostreturn false;} If (Exception instanceof Connectexception) {//Connection Refusedreturn false;} If (Exception instanceof Sslexception) {//SSL handshake Exceptionreturn false;} HttpRequest request = (HttpRequest) context.getattribute (Executioncontext.http_request); Boolean idempotent =! (Request instanceof Httpentityenclosingrequest); if (idempotent) {//Retry if the request is considered Idempotentreturn TR UE;} return false;}}; Client.sethttprequestretryhandler (Myretryhandler);

5, 6 lines set the maximum waiting time of the socket, the maximum waiting time of the connection (in milliseconds). Socket wait time refers to the download page and data from the site, the maximum time between two packets, over this interval, httpclient feel connected to a fault. The maximum waiting time for a connection is the maximum waiting time when the site is connected, and the site does not respond when the website is established. Line 7th sets httpclient not to use Nodelay policy. Assuming that the Nodelay policy is enabled, data transfer between HttpClient and the site will be sent as soon as possible in the send buffer, regardless of network bandwidth utilization, a strategy suitable for scenarios where real-time requirements are high. When this policy is disabled, data transfers are sent using Nagle's algorithm, which takes full account of bandwidth utilization, rather than the real-time transmission of data. Line 8th sets the size of the socket buffer (in bytes), which defaults to 8KB. Httprequestretryhandler is the interface responsible for handling request retries. Implementing the Retryrequest method in the inner class of the interface is possible. This method is called when an exception occurs after the httpclient sends the request. In this method, the number of times the request has been run, the content of the request, the exception information to infer whether to continue the retry, if you continue to retry returns True, otherwise return false. 3, set the request header set the request header is also very important, for example, set User-agent to disguise the crawler as a browser, cheat some sites to check the crawler, Setting Accept-encoding to Gzip allows the site to be recommended in compressed format for data transfer, bandwidth saving, and so on. Routines such as the following:

HttpResponse response = null; HttpGet get = new HttpGet (URL), Get.addheader ("Accept", "text/html"), Get.addheader ("Accept-charset", "utf-8"); Get.addheader ("accept-encoding", "gzip"), Get.addheader ("Accept-language", "En-us,en"); Get.addheader ("User-Agent" , "mozilla/5.0 (X11; Linux x86_64) applewebkit/537.22 (khtml, like Gecko) chrome/25.0.1364.160 safari/537.22 "); response = Client.execute (get ); httpentity entity = response.getentity ();  Header Header = Entity.getcontentencoding (); if (header! = null) {headerelement[] codecs = header.getelements (); for (int i = 0; i < codecs.length; i++) {if (Codecs[i].getname (). Equalsignorecase ("gzip")) {response.setentity (new Gzipdecompressingentity (entity));}} return response;

The meaning of each header reference http://kb.cnblogs.com/page/92320/need to be set up on the good. Suppose it takes a lot of different user-agent to take turns (the same user-agent has frequent access to a website for easy to be identified as a reptile), to be able to find it online, to be able to see it in its own Chrome browser or to grab it with the grab bag software. It is important to note that after Accept-encoding is set to gzip, the content of the reply to the site is checked for compression, assuming that it is decompressed, as seen in the code after the 9th line in the previous routine. [citation please specify source http://blog.csdn.net/bhq2010/article/details/9210007]

Java httpclient Usage Summary