Crawl Web page encountered 500 error: User Agent

When you write a spider program, you will find a lot of web pages will encounter 500 errors, this problem I first met, I was so tortured, finally solved, so simple.
When using
URL url = new URL ("http://www.google.cn/search?hl=zh-CN&newwindow=1&q= Beijing &start=0&sa=n");
Url.openstream () found 403 access denied error, is due to Google's Java program restrictions, the lack of agents.

Workaround:
URL url = new URL ("http://www.google.cn/search?hl=zh-CN&newwindow=1&q= Beijing &start=0&sa=n");
URLConnection con = url.openconnection ();
Con.setrequestproperty ("User-agent", "mozilla/4.0" (compatible; MSIE 5.0; Windows XP; Digext) ");
InputStream in = Con.getinputstream ();

private static Document Geturlcontent () throws Malformedurlexception, IOException, unsupportedencodingexception {
        Document doc = Jsoup.connect ("http://www.tudou.com/programs/view/pVploWOtCQM/")
          . Data ("Query", "Java")
          . UserAgent ("Mozilla").
          Cookie ("auth", "token").
          timeout (+)
          . Post ();
        return doc;
    }


In the writing crawler often encountered some pages to download the problem, if you put the request to simulate the same as the browser, it is sure to be able to get the page correctly, but often many parameters are not used, the real impact on a few key parameters. Here is a special summary of their own experience. 
key parameters to focus on are:
 1. URL: This is primarily a URL encoding can cause problems, the URL contains the Chinese language may appear when the
 2. User-agent: Big websites are usually optimized for different browsers, so there's a difference
 . 3. Cookies: Some websites use cookie information, such as some websites that record session information in a cookie
 4. Refer: Some websites check 5 of refer pages in order to prevent cross-site attacks
 . Accept-charset and accept-encoding: Individual websites will have special handling of the received encoding
 6. SessionID: This is sometimes set in parameters, some sites use this value, and of course other parameters may be useful. Above are the experience of experience
 
, hope to be helpful to everyone.

Original address: http://www.zhizhihu.com/html/y2011/3058.html