PHP extension test tool Series-use of curl and 5 Common examples

I use php curl to capture data. of course we can use other methods to capture data, such as fsockopen and file_get_contents. However, you can only capture those pages that can be accessed directly. it is more difficult to capture pages with page access control, or to log on to pages after logon. 1. capture the file without access control & lt ;? Php $ chcurl_ini

I use php curl to capture data. of course we can use other methods to capture data, such as fsockopen and file_get_contents. However, you can only capture those pages that can be accessed directly. it is more difficult to capture pages with page access control, or to log on to pages after logon.

1. capture files without access control

2. use a proxy for crawling

Why is it necessary to use a proxy for crawling? Take google for example. if google's data is captured frequently in a short period of time, you will not be able to capture it. When google restricts your IP address, you can use another proxy to re-capture it.

 3. capture data after postLet's talk about data submission separately, because when using curl, there are often Data interactions, so it is important.
 Serialize (array ('tank', 'Zhang '), 'sex' => 1, 'birth' => '123 ') * for example, array ('name' => array ('tank', 'zhang'), 'sex' => 1, 'birth' => '123 ') */$ data = array ('name' => 'test', 'sex' => 1, 'birth' => '123 '); curl_setopt ($ ch, CURLOPT_URL, 'http: // localhost/mytest/curl/upload. php '); curl_setopt ($ ch, CURLOPT_POST, 1); curl_setopt ($ ch, CURLOPT_POSTFIELDS, $ data); curl_exec ($ ch);?>

In? In the upload. php file, print_r ($ _ POST); uses curl to capture the content output by upload. php.

Array ([name] = & gt; test [sex] = & gt; 1 [birth] = & gt; 20101010)

4. capture pages with page access control

If you use the method mentioned above, the following error will be reported:

You are not authorized to view this pageYou do not have permission to view this directory or page using the credentials that you supplied because your Web browser is sending a WWW-Authenticate header field that the Web server is not configured to accept.

At this time, we will use CURLOPT_USERPWD for verification.

5. simulate logon to sina

We need to capture data, which may be post-login content. at this time, we need to use the simulated login function of curl.

Open the cookie file under/tmp and take a look:

# Netscape HTTP Cookie File# http://curl.haxx.se/rfc/cookie_spec.html# This file was generated by libcurl! Edit at your own risk.mail.sina.com.cn?? ?FALSE?? ?/?? ?FALSE?? ?0?? ?SINAMAIL-WEBFACE-SESSID?? ?65223c4bd8900284ed463d2a3e1ac182#HttpOnly_.sina.com.cn?? ?TRUE?? ?/?? ?FALSE?? ?0?? ?SUE?? ?es%3D8d96db0820c6c79922ad57d422f575e8%26ev%3Dv0%26es2%3Dcddfb8400dc5ca95902367ddcd7f57dd.sina.com.cn?? ?TRUE?? ?/?? ?FALSE?? ?0?? ?SUP?? ?cv%3D1%26bt%3D1286900433%26et%3D1286986833%26lt%3D1%26uid%3D1445632344%26user%3D%25E5%25BC%25A0%25E6%2598%25A02001%26ag%3D2%26name%3Dzhangying20015%2540sina.com%26nick%3D%25E5%25BC%25A0%25E6%2598%25A02001%26sex%3D1%26ps%3D0%26email%3Dzhangying20015%2540sina.com%26dob%3D1982-07-18#HttpOnly_.sina.com.cn?? ?TRUE?? ?/?? ?FALSE?? ?0?? ?SID?? ?BihcallomxMx-QZxzGrOlcSQx%2F0B%2F0cmr.NyQ%2F0B%2FcmGGalmarlmcHrcGlSmrmxmfxal_CBZ%2F_afugCmmGirBYHm0Bc%40fr5ciZiGG5i#HttpOnly_.sina.com.cn?? ?TRUE?? ?/?? ?FALSE?? ?0?? ?SPRIAL?? ?bfb4102951fd5892a3fd5b42d442cd26#HttpOnly_.sina.com.cn?? ?TRUE?? ?/?? ?FALSE?? ?0?? ?SINA_USER?? ?%D5%C5%D2001