Phpcurl for out-of-site collection (recommended)

Source: Internet
Author: User
Curl is a library dedicated for network interaction. It provides a bunch of custom options to deal with different environments. the stability is naturally greater than file_get_contents curl, which is a library dedicated for network interaction, A bunch of custom options are provided to cope with different environments. the stability is naturally higher than file_get_contents.

Reason for curl selection

For curl and file_get_contents, the following is an easy-to-understand comparison:
File_get_contents is actually a combined version of a bunch of built-in file operation functions, such as file_exists, fopen, fread, and fclose. it is designed for lazy users and is mainly used to deal with local files, but it is also because of laziness, and added support for network files;
Curl is a library dedicated for network interaction. It provides a bunch of custom options to deal with different environments. the stability is naturally greater than file_get_contents.

Usage

1. enable curl support

After the php environment is installed, curl is not enabled by default. you need to modify the php. ini file and find it. extension = php_curl.dll, remove the colon, and restart the service;

2. use curl to capture data

The code is as follows:

// Initialize a cURL object $ curl = curl_init (); // Set the URL curl_setopt ($ curl, CURLOPT_URL, 'http: // www.cmx8.cn') to be crawled '); // Set the header curl_setopt ($ curl, CURLOPT_HEADER, 1); // sets the cURL parameter and requires that the result be saved to the string or output to the screen. Curl_setopt ($ curl, CURLOPT_RETURNTRANSFER, 1); // run cURL, request webpage $ data = curl_exec ($ curl); // close URL request curl_close ($ curl );

3. find key data through regular expression matching

The code is as follows:

// $ Data is the value returned by curl_exec, that is, the collected target content preg_match_all ("/
  • (.*?) <\/Li>/", $ data, $ out, PREG_SET_ORDER); foreach ($ out as $ key => $ value) {// Here $ value is an array, at the same time, the record finds the entire sentence with matching characters and the individually matched character echo 'matched entire sentence :'. $ value [0]. ''; echo 'matched separately :'. $ value [1]. '';}
  • Tips

    1. timeout settings

    You can use curl_setopt ($ ch, opt) to set some timeout settings, including:

    CURLOPT_TIMEOUT sets the maximum number of seconds that cURL can be executed.
    CURLOPT_TIMEOUT_MS sets the maximum number of milliseconds that cURL can be executed. (Added in cURL 7.16.2. It can be used from PHP 5.2.3. )
    CURLOPT_CONNECTTIMEOUT is the waiting time before the connection is initiated. if it is set to 0, it will wait infinitely.
    CURLOPT_CONNECTTIMEOUT_MS indicates the waiting time for the connection attempt, in milliseconds. If it is set to 0, the system waits for no limit. Added to cURL 7.16.2. Available from PHP 5.2.3.
    CURLOPT_DNS_CACHE_TIMEOUT sets the time for saving DNS information in the memory. the default value is 120 seconds.

    The code is as follows:

    Curl_setopt ($ ch, CURLOPT_TIMEOUT, 60); // you only need to set the number of seconds to curl_setopt ($ ch, CURLOPT_NOSIGNAL, 1); // note, you must set this curl_setopt ($ ch, CURLOPT_TIMEOUT_MS, 200) for millisecond timeout; // timeout in milliseconds, added to cURL 7.16.2. Available from PHP 5.2.3

    2. submit data through post to retain cookies

    The code is as follows:

    // Here is an example to learn from: // Curl simulated login discuz program, suitable for DZ7.0! Extension_loaded ('curl') & die ('The curl extension is not loaded. '); $ discuz_url = 'http: // www.lxvoip.com'; // forum address $ login_url = $ discuz_url. '/logging. php? Action = login '; // logon page address $ get_url = $ discuz_url.'/my. php? Item = threads '; // my post $ post_fields = array (); // You do not need to modify the following two items: $ post_fields ['loginfield'] = 'username '; $ post_fields ['loginsubmit '] = 'true'; // username and password, which must be set to $ post_fields ['username'] = 'lxvoip '; $ post_fields ['password'] = '000000'; // security question $ post_fields ['questionid'] = 0; $ post_fields ['ancer'] = ''; // @ todo verification code $ post_fields ['seccodeverify '] = ''; // Obtain the form FORMHASH $ ch = curl_init ($ login_url); curl_setopt ($ ch, CURLOPT_HEADER, 0 ); curl_setopt ($ ch, CURLOPT_RETURNTRANSFER, 1); $ contents = curl_exec ($ ch); curl_close ($ ch); preg_match ('/
     /I ', $ contents, $ matches); if (! Empty ($ matches) {$ formhash = $ matches [1];} else {die ('Not found the forumhash. ');} // POST the data to obtain the COOKIE $ cookie_file = dirname (_ FILE __). '/cookie.txt'; // $ cookie_file = tempnam ('/tmp'); $ ch = curl_init ($ login_url); curl_setopt ($ ch, CURLOPT_HEADER, 0 ); curl_setopt ($ ch, batch, 1); curl_setopt ($ ch, CURLOPT_POST, 1); curl_setopt ($ ch, CURLOPT_POSTFIELDS, $ post_fields); curl_setopt ($ ch, CURLOPT_COOKIEJAR, $ cookie_file); curl_exec ($ ch); curl_close ($ ch ); // Obtain the page content that requires logon with the COOKIE obtained above $ ch = curl_init ($ get_url); curl_setopt ($ ch, CURLOPT_HEADER, 0); curl_setopt ($ ch, CURLOPT_RETURNTRANSFER, 0); curl_setopt ($ ch, CURLOPT_COOKIEFILE, $ cookie_file); $ contents = curl_exec ($ ch); curl_close ($ ch); var_dump ($ contents );

    The above is the details of the php curl implementation of out-of-site collection (recommended). For more information, see other related articles in the first PHP community!

    Contact Us

    The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

    If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

    A Free Trial That Lets You Build Big!

    Start building with 50+ products and up to 12 months usage for Elastic Compute Service

    • Sales Support

      1 on 1 presale consultation

    • After-Sales Support

      24/7 Technical Support 6 Free Tickets per Quarter Faster Response

    • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.