Use Curl to crawl Web site data, imitation IP, anti-blocking the ultimate powerful solution

Recently do some crawl other website data work, of course, others will not be a free to you scratch data, there are various anti-crawling methods. But however persuasive, outsmart, through the study are loopholes can be drilled. The following examples are written in PHP, do not use PHP to Curl child paper first learn this piece and then look down, you can refer to this article: http://blog.csdn.net/linglongwunv/article/details/8020845

Below the bottom, first say a common way, forged source IP, which is also a lot of friends to brush tickets common methods:

1, a simple point can be forged in the header x-forwarded-for, and forged Referer, the code is as follows:

[PHP]View Plaincopy

1. curl_setopt ($ch, Curlopt_httpheader, Array (' x-forwarded-for:111.222.333.4 ', ' Client-ip  : 111.222.333.4 '));
2. curl_setopt ($ch, Curlopt_referer, "http://www.test.com");

2, the above method most can fool over, but also have caught the real IP. In the use of proxies, the trouble is that you have a valid proxy IP and port number, and some also need a user name password, the code is as follows:

[PHP]View Plaincopy

1. curl_setopt ($ch, Curlopt_proxy, "http://111.222.333.4:110");

In addition, there is a situation, that is, with the browser can access, with curl is not, found that the other side checked the useragent, if not considered to be crawling and other illegal sources, then we will add useragent in the header, the code is as follows:

[PHP]View Plaincopy

1. curl_setopt ($ch, curlopt_useragent, "mozilla/5.0 (Windows NT 6.1) applewebkit/536.11 (khtml, like Gecko) chrome/ 20.0.1132.57 safari/536.11 ");

Use Curl to crawl Web site data, imitation IP, anti-blocking the ultimate powerful solution