Java Web Crawler connection timeout solves instance code, java Crawler

Java Web Crawler connection timeout solves instance code, java Crawler

This article focuses on the java Web Crawler connection timeout issue, as detailed below.

In web crawlers, the following errors are often reported. That is, the connection times out. The general solution to this problem is to set the connection time and request time for a long time. If the connection times out, set the number of reRequests again ].

Exception in thread "main" java.net.ConnectException: Connection timed out: connect

The following code uses httpclient to solve connection timeout. Directly upload the program.

Package daili; import java. io. IOException; import java.net. URI; import org. apache. http. httpRequest; import org. apache. http. httpResponse; import org. apache. http. client. clientProtocolException; import org. apache. http. client. methods. httpGet; import org. apache. http. client. params. cookiePolicy; import org. apache. http. client. protocol. clientContext; import org. apache. http. impl. client. basicCookieStore; import org. apache. http. impl. client. closeableHttpClient; import org. apache. http. impl. client. defaultHttpClient; import org. apache. http. impl. client. defaultHttpRequestRetryHandler; import org. apache. http. impl. client. httpClients; import org. apache. http. impl. cookie. basicClientCookie2; import org. apache. http. params. httpConnectionParams; import org. apache. http. params. httpParams; import org. apache. http. protocol. basicHttpContext; import org. apache. http. protocol. executionContext; import org. apache. http. protocol. httpContext; import org. apache. http. util. entityUtils;/** author: Qian Yang * 1563178220@qq.com */public class Test1 {public static void main (String [] args) throws ClientProtocolException, IOException, interruptedException {getRawHTML ("http://club.autohome.com.cn/bbs/forum-c-2098-1.html#pvareaid=103447");} public static String getRawHTML (String url) throws ClientProtocolException, IOException, InterruptedException {// initialize response httpclient = new response (); httpclient. getParams (). setParameter ("http. protocol. cookie-policy ", CookiePolicy. BROWSER_COMPATIBILITY); // sets the parameter HttpParams params = httpclient. getParams (); // connection time: HttpConnectionParams. setConnectionTimeout (params, 6000); HttpConnectionParams. setSoTimeout (params, 6000*20); // DefaultHttpRequestRetryHandler dhr = new timeout (5, true); HttpContext localContext = new BasicHttpContext (); HttpRequest request2 = (HttpRequest) localContext. getAttribute (ExecutionContext. HTTP_REQUEST); httpclient. setHttpRequestRetryHandler (dhr); BasicCookieStore cookieStore = new BasicCookieStore (); BasicClientCookie2 cookie = new BasicClientCookie2 ("Content-Type", "text/html; charset = UTF-8 "); basicClientCookie2 cookie1 = new BasicClientCookie2 ("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) chrome/52.0.2743.116 Safari/537.36 "); cookieStore. addCookie; cookieStore. addCookie (cookie1); localContext. setAttribute (ClientContext. COOKIE_STORE, cookieStore); HttpGet request = new HttpGet (); request. setURI (URI. create (url); HttpResponse response = null; String rawHTML = ""; response = httpclient.exe cute (request, localContext); int StatusCode = response. getStatusLine (). getStatusCode (); // get the response status code System. out. println (StatusCode); if (StatusCode = 200) {// Status Code 200 indicates the response is successful // obtain the entity content rawHTML = EntityUtils. toString (response. getEntity (); System. out. println (rawHTML); // output entity content EntityUtils. consume (response. getEntity (); // consumes entity} else {// closes the stream entity EntityUtils of HttpEntity. consume (response. getEntity (); // consumes entity Thread. sleep (20*60*1000); // if an error is reported, rest for 30 minutes.} httpclient. close (); System. out. println (rawHTML); return rawHTML ;}}

Result:

Summary

The above is all the content about the java Web Crawler connection timeout solution for the instance code. I hope it will be helpful to you. If you are interested, you can continue to refer to other related topics on this site. If you have any shortcomings, please leave a message. Thank you for your support!