Example of how PHP can use curl to download a specified size file

Using the Libcurl-based Curl function in PHP, you can initiate an HTTP request to the destination URL and get the returned response content. The usual way to request it is similar to the following code:

Public Function CallFunction ($url, $postData, $method, header= ") {    $maxRetryTimes = 3;    $curl = Curl_init ();    /****** Initialization request Parameter start******/    if (Strtoupper ($method)!== ' GET ' && $postData) {        curl_setopt ($curl, Curlopt_postfields, Json_encode ($postData));    } ElseIf (Strtoupper ($method) = = = ' GET ' && $postData) {            $url. = '? '. Http_build_query ($postData);    }    /****** Initialization request parameter end******/    Curl_setopt_array ($curl, array (        Curlopt_url = = $url,        curlopt_timeout = >,        curlopt_nobody = 0,        curlopt_returntransfer = 1    ));    if (method = = ' POST ') {        curl_setopt ($curl, Curlopt_post, True);    }    if (false = = Empty ()) {        curl_setopt ($curl, Curlopt_httpheader, $header);    }    $response = false;    while ($response = = = False) && (--$maxRetryTimes > 0)) {            $response = Trim (curl_exec ($curl));    }    return $response;}

The $response in the above code is the data that Curl initiated this HTTP request from the line, and if you do not specify the size to download in $header by range, regardless of how large the resource is, the full content of the URI is requested and returned. It is usually only by using curl to request some interfaces or remotely calling a function to fetch data, so it is important to curlopt_timeout this parameter in this scenario.

The usage scenario for Curl does not only access the data interface, but also detects any URL resource to provide the correct HTTP service. When the user fills in the URL is a resource file, such as a PDF or PPT, and so on, if the network condition is poor, using curl to request a larger resource, will inevitably occur time-out or consume more network resources. The previous policy is a full download (Curl will be downloaded in memory), the content is checked after the request is finished, and the monitored task is paused when the target value is exceeded. In this case after the limitation of the fact that the palliative, and finally the customer put forward a new demand, can not stop the task download only the specified size of the file and return MD5 value by the customer to verify the correctness.

After some attempts to solve the problem, the recording process is as follows.

1, try to use curlopt_maxfilesize.

PHP and Libcurl version of the version requirements, complete pre-processing, when the discovery target is greater than the set, the direct return to the size limit of errors and not to download the target, not meet the requirements.

2. Use the callback function of the curl download process.

Reference to the http://php.net/manual/en/function.curl-setopt-array.php, the final use of the Curlopt_writefunction parameter set the On_curl_write, The function will be recalled 1 times in 1s.

$ch = Curl_init (); $options = Array (curlopt_url =        ' http://www.php.net/', Curlopt_header =        false, Curlopt_headerfunction    = ' On_curl_header ', curlopt_writefunction    = ' on_curl_write ');

The final fragment of my implementation:

function On_curl_write ($ch, $data) {    $pid = Getmypid ();    $downloadSizeRecorder = Downloadsizerecorder::getinstance ($pid);    $bytes = strlen ($data);    $downloadSizeRecorder->downloaddata. = $data;    $downloadSizeRecorder->downloadedfilesize + = $bytes;//    Error_log (' On_curl_write '. $downloadSizeRecorder- >downloadedfilesize. "> {$downloadSizeRecorder->maxsize} \ n", 3, '/tmp/hyb.log ');    Make sure that the content you have downloaded is slightly larger than the maximum limit    if ($downloadSizeRecorder->downloadedfilesize-$bytes) > $downloadSizeRecorder MaxSize) {        return false;    }    return $bytes;  This incorrect return, will be error, interrupted download "errno": "ErrMsg": "Failed writing body (0! = 16384)"}

Downloadsizerecorder is a singleton pattern class, curl downloads the record size, implements the MD5 to return the downloaded content, and so on.

Class downloadsizerecorder{Const error_failed_writing =//failed WRITING body public $downloadedFileSize;        Public $maxSize;        Public $pid;        Public $hasOverMaxSize;        Public $fileFullName;        Public $downloadData;        private static $selfInstanceList = Array (); public static function getinstance ($pid) {if (!isset (self:: $selfInstanceList [$pid])} {self:: $selfInst        ancelist[$pid] = new self ($pid);    } return Self:: $selfInstanceList [$pid];            } Private Function __construct ($pid) {$this->pid = $pid;            $this->downloadedfilesize = 0;            $this->filefullname = ";            $this->hasovermaxsize = false;    $this->downloaddata = ";                }/** * Save file */Public Function savemaxsizedata2file () {if (Empty ($resp _data)) {        $resp _data = $this->downloaddata; } $fileFullName = '/tmp/http_ '. $this->pId. ' _ '. Time (). "                _{$this->maxsize}.download "; if ($resp _data && strlen ($resp _data) >0) {list ($headerOnly, $bodyOnly) = Explode ("\ r \ n                    \ r \ n ", $resp _data, 2); $saveDataLenth = ($this->downloadedfilesize < $this->maxsize)?                    $this->downloadedfilesize: $this->maxsize;                    $needSaveData = substr ($bodyOnly, 0, $saveDataLenth);            if (empty ($needSaveData)) {return;                        } file_put_contents ($fileFullName, $needSaveData);            if (file_exists ($fileFullName)) {$this->filefullname = $fileFullName;            }}}/** * Returns the file's MD5 * @return String */Public Function returnFileMd5 () {            $MD 5 = ";        if (file_exists ($this->filefullname)) {$MD 5 = md5_file ($this->filefullname);    } return $MD 5;  }      /** * Returns the downloaded size * @return int */Public Function returnsize () {return ($this->dow Nloadedfilesize < $this->maxsize)?    $this->downloadedfilesize: $this->maxsize; /** * Delete downloaded file */Public Function DeleteFile () {if (file_exists ($this->filefullname))        {unlink ($this->filefullname); }    }}

Curl request in code instance, implement limit download size

... curl_setopt ($ch, curlopt_writefunction, ' on_curl_write ');//Set callback function ... $pid = Getmypid (); $downloadSizeRecorder = Downloadsizerecorder::getinstance ($pid); $downloadSizeRecorder->maxsize = $size _limit;......//initiates a curl request $response = Curl_exec ($ch);.//Save file, return Md5$downloadsizerecorder->savemaxsizedata2file ();  Save $downloadfilemd5 = $downloadSizeRecorder->returnfilemd5 (); $downloadedfile _size = $downloadSizeRecorder Returnsize (); $downloadSizeRecorder->deletefile ();

Here, stepping on a hole. When the On_curl_write is added, the $response returns True, causing the exception to be returned when the contents are taken back. Fortunately, the size of the download has been limited in real time, using Downloaddata to record what has been downloaded, can be used directly.

if ($response = = = True) {    $response = $downloadSizeRecorder->downloaddata;}