PHP uses curl to download files of the specified size

Using the Libcurl-based Curl function in PHP, you can initiate an HTTP request to the destination URL and get the returned response content. The usual way to request it is similar to the following code:

 Public functionCallFunction ($url,$postData,$method,Header= ' '){    $maxRetryTimes= 3; $curl=Curl_init (); /** * * * Initialize request parameter start******/    if(Strtoupper($method)!== ' GET ' &&$postData) {curl_setopt ($curl, Curlopt_postfields, Json_encode ($postData)); }ElseIf(Strtoupper($method) = = = ' GET ' &&$postData){        $url. = '? '.Http_build_query($postData); }    /** * * * Initialize request parameter end******/Curl_setopt_array ($curl,Array(Curlopt_url=$url,Curlopt_timeout= 10,Curlopt_nobody= 0,Curlopt_returntransfer= 1    )); if(method = = ' POST ') {curl_setopt ($curl, Curlopt_post,true); }    if(false==Empty()) {curl_setopt ($curl, Curlopt_httpheader,$header); }    $response=false;  while(($response===false) && (--$maxRetryTimes> 0)){        $response=Trim(Curl_exec ($curl)); }    return $response;}

The $response in the above code is the data that Curl initiated this HTTP request from the line, and if you do not specify the size to download in $header by range, regardless of how large the resource is, the full content of the URI is requested and returned. It is usually only by using curl to request some interfaces or remotely calling a function to fetch data, so it is important to curlopt_timeout this parameter in this scenario.

The usage scenario for Curl does not only access the data interface, but also detects any URL resource to provide the correct HTTP service. When the user fills in the URL is a resource file, such as a PDF or PPT, and so on, if the network condition is poor, using curl to request a larger resource, will inevitably occur time-out or consume more network resources. The previous policy is a full download (Curl will be downloaded in memory), the content is checked after the request is finished, and the monitored task is paused when the target value is exceeded. In this case after the limitation of the fact that the palliative, and finally the customer put forward a new demand, can not stop the task download only the specified size of the file and return MD5 value by the customer to verify the correctness.

After some attempts to solve the problem, the recording process is as follows.

1, try to use curlopt_maxfilesize.

PHP and Libcurl version of the version requirements, complete pre-processing, when the discovery target is greater than the set, the direct return to the size limit of errors and not to download the target, not meet the requirements.

2. Use the callback function of the curl download process.

Reference to the http://php.net/manual/en/function.curl-setopt-array.php, the final use of the Curlopt_writefunction parameter set the On_curl_write, The function will be recalled 1 times in 1s.

$ch = Curl_init ();
$options = Array (curlopt_url = ' http://www.php.net/',
Curlopt_header = False,
curlopt_headerfunction = ' On_curl_header ',
curlopt_writefunction = ' On_curl_write '
);

The final fragment of my implementation:

functionOn_curl_write ($ch,$data){    $pid=Getmypid(); $downloadSizeRecorder= Downloadsizerecorder::getinstance ($pid); $bytes=strlen($data); $downloadSizeRecorder->downloaddata. =$data; $downloadSizeRecorder->downloadedfilesize + =$bytes;//error_log (' On_curl_write '. $downloadSizeRecorder->downloadedfilesize. > {$downloadSizeRecorder    MaxSize} \ n ", 3, '/tmp/hyb.log '); Make sure the downloaded content is slightly larger than the maximum limit    if(($downloadSizeRecorder->downloadedfilesize-$bytes) >$downloadSizeRecorder-maxSize) {        return false; }    return $bytes;//this incorrect return, will be error, interrupted download "errno": "ErrMsg": "Failed writing body (0! = 16384)"}

Downloadsizerecorder is a singleton pattern class, curl downloads the record size, implements the MD5 to return the downloaded content, and so on.

classdownloadsizerecorder{Consterror_failed_writing = 23;//Failed Writing Body     Public $downloadedFileSize;  Public $maxSize;  Public $pid;  Public $hasOverMaxSize;  Public $fileFullName;  Public $downloadData; Private Static $selfInstanceList=Array();  Public Static functionGetInstance ($pid)    {        if(!isset(Self::$selfInstanceList[$pid]) { self::$selfInstanceList[$pid] =NewSelf$pid); }        returnSelf::$selfInstanceList[$pid]; }    Private function__construct ($pid)    {        $this->pid =$pid; $this->downloadedfilesize = 0; $this->filefullname = ' '; $this->hasovermaxsize =false; $this->downloaddata = ' '; }    /** * Save file*/     Public functionSavemaxsizedata2file () {if(Empty($resp _data)){            $resp _data=$this-Downloaddata; }        $fileFullName= '/tmp/http_ '.$this->pid. ' _‘. Time()." _{$this->maxsize}.download "; if($resp _data&&strlen($resp _data) >0)        {            List($headerOnly,$bodyOnly) =Explode("\r\n\r\n",$resp _data, 2); $saveDataLenth= ($this->downloadedfilesize <$this->maxsize)?$this->downloadedfilesize:$this-maxSize; $needSaveData=substr($bodyOnly, 0,$saveDataLenth); if(Empty($needSaveData)){                return; }            file_put_contents($fileFullName,$needSaveData); if(file_exists($fileFullName)){                $this->filefullname =$fileFullName; }        }    }    /** * Returns the MD5 of the file * @return string*/     Public functionReturnFileMd5 () {$MD 5= ' '; if(file_exists($this-filefullname)) {            $MD 5=Md5_file($this-filefullname); }        return $MD 5; }    /** * Returns the downloaded size * @return int*/     Public functionreturnsize () {return($this->downloadedfilesize <$this->maxsize)?$this->downloadedfilesize:$this-maxSize; }    /** * Delete the downloaded file*/     Public functionDeleteFile () {if(file_exists($this-filefullname)) {            unlink($this-filefullname); }    }}

Curl request in code instance, implement limit download size

... curl_setopt ($ch, Curlopt_writefunction, ' on_curl_write ');//Setting the callback function...$pid=Getmypid();$downloadSizeRecorder= Downloadsizerecorder::getinstance ($pid);$downloadSizeRecorder->maxsize =$size _limit; ......//initiate a Curl request$response= Curl_exec ($ch); ......//save file, return MD5$downloadSizeRecorder->savemaxsizedata2file ();//Save$downloadFileMd 5=$downloadSizeRecorder-returnFileMd5 ();$downloadedfile _size=$downloadSizeRecorder-returnsize ();$downloadSizeRecorder->deletefile ();

Here, stepping on a hole. When the On_curl_write is added, the $response returns True, causing the exception to be returned when the contents are taken back. Fortunately, the size of the download has been limited in real time, using Downloaddata to record what has been downloaded, can be used directly.

if ($response = = = True) {
$response = $downloadSizeRecorder->downloaddata;
}

PHP uses curl to download files of the specified size