Phpcurl website Collection Implementation Program

Most of the website collection functions are implemented using the software of the locomotive. However, for some scheduled or small collections, we can use a program to implement the function. in php, curl is a function selected, let's take a look at the curl website Collection implementation program. select curl... most of the website collection functions are implemented using the software of the locomotive. However, for some scheduled or small collections, we can use a program to implement the function. in php, curl is a function selected, let's take a look at the curl website Collection implementation program.

Reason for curl selection

For curl and file_get_contents, the following is an easy-to-understand comparison:

File_get_contents is actually a combined version of a bunch of built-in file operation functions, such as file_exists, fopen, fread, and fclose. it is designed for lazy users and is mainly used to deal with local files, but it is also because of laziness, and added support for network files.

Curl is a library dedicated for network interaction. m provides a bunch of custom options. m is more stable than file_get_contents to cope with different environments,

Usage

1. enable curl support

Since curl is not supported by the php environment by default after installation, you need to modify the php. ini file and find it; extension = php_curl.dll, remove the colon, and restart the service.

2. use curl to capture data

// Initialize a cURL object $ curl = curl_init (); // Set the URL you want to crawl curl_setopt ($ curl, CURLOPT_URL, 'http: // www.phprm.com '); // Set the header curl_setopt ($ curl, CURLOPT_HEADER, 1); // sets the cURL parameter and requires that the result be saved to the string or output to the screen. Curl_setopt ($ curl, CURLOPT_RETURNTRANSFER, 1); // run cURL, request webpage $ data = curl_exec ($ curl); // close URL request curl_close ($ curl );

3. find key data through regular expression matching.

// $ Data is the value returned by curl_exec, that is, the collected target content preg_match_all ("/(.*?) /", $ Data, $ out, PREG_SET_ORDER); foreach ($ out as $ key => $ value) {// Here $ value is an array, at the same time, the record finds the entire sentence with matching characters and the individually matched character echo 'matched entire sentence :'. $ value [0]. ''; echo 'matched separately :'. $ value [1]. '';}

Tips

1. timeout settings

You can use curl_setopt ($ ch, opt) to set some timeout settings, including:

CURLOPT_TIMEOUT sets the maximum number of seconds that cURL can be executed.

CURLOPT_TIMEOUT_MS sets the maximum number of milliseconds that cURL can execute. (added in cURL 7.16.2, which can be used from PHP 5.2.3)

CURLOPT_CONNECTTIMEOUT is the waiting time before the connection is initiated. if it is set to 0, it will be infinitely waiting.

CURLOPT_CONNECTTIMEOUT_MS indicates the connection wait time, in milliseconds. if it is set to 0, the connection waits infinitely. it is added to cURL 7.16.2 and is available starting from PHP 5.2.3.

CURLOPT_DNS_CACHE_TIMEOUT sets the time for saving DNS information in the memory. the default value is 120 seconds.

Curl_setopt ($ ch, CURLOPT_TIMEOUT, 60); // you only need to set the number of seconds to curl_setopt ($ ch, CURLOPT_NOSIGNAL, 1); // note, you must set this curl_setopt ($ ch, CURLOPT_TIMEOUT_MS, 200) for millisecond timeout; // timeout in milliseconds, added to cURL 7.16.2. Available from PHP 5.2.3

2. submit data through post to retain cookies

// Here is an example to learn from: // Curl simulated login discuz program, suitable for DZ7.0! Extension_loaded ('curl') & die ('The curl extension is not loaded. '); $ discuz_url = 'http: // www.phprm.com'; // forum address $ login_url = $ discuz_url. '/logging. php? Action = login '; // logon page address $ get_url = $ discuz_url.'/my. php? Item = threads '; // my post $ post_fields = array (); // You do not need to modify the following two items: $ post_fields ['loginfield'] = 'username '; $ post_fields ['loginsubmit '] = 'true'; // username and password, which must be set to $ post_fields ['username'] = 'lxvoip '; $ post_fields ['password'] = '000000'; // security question $ post_fields ['questionid'] = 0; $ post_fields ['ancer'] = ''; // @ todo verification code $ post_fields ['seccodeverify '] = ''; // Obtain the form FORMHASH $ ch = curl_init ($ login_url); curl_setopt ($ ch , CURLOPT_HEADER, 0); curl_setopt ($ ch, CURLOPT_RETURNTRANSFER, 1); $ contents = curl_exec ($ ch); curl_close ($ ch); preg_match ('// I ', $ contents, $ matches); if (! Emptyempty ($ matches) {$ formhash = $ matches [1];} else {die ('not found the forumhash .');}

// POST the data to obtain the COOKIE

$ Cookie_file = dirname (_ FILE __). '/cookie.txt'; // $ cookie_file = tempnam ('/tmp'); $ ch = curl_init ($ login_url); curl_setopt ($ ch, CURLOPT_HEADER, 0 ); curl_setopt ($ ch, batch, 1); curl_setopt ($ ch, CURLOPT_POST, 1); curl_setopt ($ ch, CURLOPT_POSTFIELDS, $ post_fields); curl_setopt ($ ch, CURLOPT_COOKIEJAR, $ cookie_file); curl_exec ($ ch); curl_close ($ ch ); // Obtain the page content that requires logon with the COOKIE obtained above $ ch = curl_init ($ get_url); curl_setopt ($ ch, CURLOPT_HEADER, 0); curl_setopt ($ ch, CURLOPT_RETURNTRANSFER, 0); curl_setopt ($ ch, CURLOPT_COOKIEFILE, $ cookie_file); $ contents = curl_exec ($ ch); curl_close ($ ch); var_dump ($ contents );

I have some experiences in using curl to share with you.

Encoding conversion

First, check the source code to find the encoding used by the collected website and use the mb_convert_encoding function for transcoding;

Usage:

// The Source character is $ str

// The following known original encoding is GBK and converted to UTF-8

Mb_convert_encoding ($ str, "UTF-8", "GBK ");

// The following unknown original encoding. after auto detection, the conversion encoding is UTF-8

Mb_convert_encoding ($ str, "UTF-8", "auto ");

3. in order to better avoid the obstacles of uncertain factors such as line breaks and spaces, it is necessary to clear the line breaks, space characters, and tabs in the collected source code.

// Method 1, replace with str_replace

$ Contents = str_replace ("\ r \ n", '', $ contents); // clear the linefeed $ contents = str_replace (" \ n ",'', $ contents ); // clear the line break $ contents = str_replace ("\ t", '', $ contents); // clear the TAB $ contents = str_replace (" ",'', $ contents ); // clear space characters

// Method 2, replace with a regular expression

$ Contents = preg_replace ("/([\ r \ n | \ t |] +)/", '', $ contents );

4. find the code segment to be obtained through regular expression matching, and use preg_match_all to implement the matching.

Function explanation:

Int preg_match_all (string pattern, string subject, array matches [, int flags])

Pattern is a regular expression.

Subject is the original text to be searched.

Matches is an array used to store output results.

Flags is the storage mode, including:

PREG_PATTERN_ORDER; // the entire array is a two-dimensional array. $ arr1 [0] is an array consisting of the matched strings composed of the boundary. $ arr1 [1] removes the array of matched strings composed of the boundary.

PREG_SET_ORDER; // the entire array is a two-dimensional array. $ arr2 [0] [0] is the first matching string consisting of the boundary, $ arr2 [0] [1] is the first matching string consisting of removing the boundary, followed by the array and so on.

PREG_OFFSET_CAPTURE; // the entire array is a three-dimensional array. $ arr3 [0] [0] [0] is the first matching string consisting of the boundary, $ arr3 [0] [0] [1] is the offset to reach the boundary of the first matching string (the boundary is not counted), and so on, $ arr2 [1] [0] [0] is the first matching string consisting of the boundary, $ arr3 [1] [0] [1] is the offset to reach the boundary of the first matching string (the boundary is included );

// Practical application

Preg_match_all ('/

$ Out gets all matching elements

$ Out [0] [0] will include

Including full-range characters

$ Out [0] [1] will only include (.*?) Character segments matching in parentheses

// Similarly, the nth matched field can be obtained using the following method:

$ Out [n-1] [1]

// If the regular expression contains many parentheses, the method for getting the m matching point in the sentence is

$ Out [n-1] [m]

5. after obtaining the characters to be found, to remove the html tag, use the strip_tags function provided by PHP to conveniently implement it.

Example: $ result = strip_tags ($ out [0] [1]);

The above is just to download the data collection, of course, we 'd better put $ contents content into the database for processing, here is a simple php Data Query and storage function, very simple.