How to use Phpcurl to implement multi-process download file classes

Bulk download files are generally used in a circular manner to perform the download individually. However, with the bandwidth and server performance allowed, using multi-process to download can greatly improve the efficiency of the download. This paper introduces the multi-process request method of PHP using Curl, which realizes the simultaneous downloading of files.

Principle:

Use Curl's batch method to turn on multi-process, implement bulk download file.

Main methods:

Curl_multi_init
Returns a new Curl batch handle

Curl_multi_add_handle
To add a separate curl handle to a curl batch session

curl_multi_exec
Child connection that runs the current CURL handle

curl_multi_getcontent
Returns the text stream of the obtained output if Curlopt_returntransfer is set

Curl_multi_remove_handle
Remove a handle resource from the Curl batch handle resource

Curl_multi_close
Close a set of curl handles

The complete code is as follows:

BatchDownLoad.class.php

<?php/** * Multi-process bulk download file (implemented using PHP curl_multi_exec) * date:2017-07-16 * author:fdipzone * version:1.0 * * Func * publi c Download Download processing * Public process multi-process Download * Private To_log Write execution result to log file */class batchdownload {//download file set private $    Download_config = Array ();    Maximum number of open processes private $max _process_num = 10;    Timeout number of seconds private $timeout = 10;    Log file private $logfile = null; /** * Initialize * @param Array $download _config downloaded file settings * @param Int $max _process_num Maximum number of open processes * @par Am INT $timeout Timeout seconds * @param String $logfile log file path */Public function __construct ($d        Ownload_config, $max _process_num=10, $timeout =10, $logfile = ") {$this->download_config = $download _config;        $this->max_process_num = $max _process_num;        $this->timeout = $timeout;        Log file if ($logfile) {$this->logfile = $logfile; }else{$this->logfile = dirname (__file__). ' /batch_doWnload_ '. Date (' Ymd ').        Log ';        }}/** * Execute download * @result Int */Public Function download () {//processed quantity $handle _num = 0; Unhandled finish while (count ($this->download_config) >0) {//required to be processed is greater than the maximum number of processes if (count ($this-            >download_config) > $this->max_process_num) {$process _num = $this->max_process_num;            Less than the maximum number of processes to process}else{$process _num = count ($this->download_config);            }//Extract the specified quantity for download $tmp _download_config = Array_splice ($this->download_config, 0, $process _num);            Perform download $result = $this->process ($tmp _download_config);            Write Log $this->to_log ($tmp _download_config, $result);        Record the number of processed $handle _num + = count ($result);    } return $handle _num; }/** * Multi-process Download file * @param array $download _config settings for this download * @return Array */public funCtion process ($download _config) {//File resource $fp = Array ();        Curl Session $ch = array ();        Execution result $result = Array ();        Create Curl Handle $MH = Curl_multi_init ();            Loop Set number foreach ($download _config as $k + = $config) {$ch [$k] = Curl_init ();            $fp [$k] = fopen ($config [1], ' a ');            curl_setopt ($ch [$k], Curlopt_url, $config [0]);            curl_setopt ($ch [$k], Curlopt_file, $fp [$k]);            curl_setopt ($ch [$k], Curlopt_header, 0);            curl_setopt ($ch [$k], Curlopt_returntransfer, true); curl_setopt ($ch [$k], Curlopt_useragent, ' mozilla/4.0 (compatible; MSIE 7.0;            Windows NT 6.0) ');        Join the processing curl_multi_add_handle ($MH, $ch [$k]);        } $active = null;        do{$MRC = curl_multi_exec ($MH, $active);        } while ($active);        Get Data foreach ($fp as $k = = $v) {fwrite ($v, Curl_multi_getcontent ($ch [$k])); }//Turn off CURL handle with File resource foreach ($download _config as $k + = $config) {curl_multi_remove_handle ($MH, $ch [$k]);            Fclose ($fp [$k]);            Check if the download succeeds if (File_exists ($config [1])) {$result [$k] = true;            }else{$result [$k] = false;        }} curl_multi_close ($MH);    return $result; /** * Write log * @param array $data download file data * @param array $flag download file status data */Private function To_log ($d        ATA, $flag) {//temporary log data $tmp _log = '; foreach ($data as $k = + $v) {$tmp _log. = ' ['. Date (' y-m-d h:i:s '). '] URL: '. $v [0]. ' File: '. $v [1]. ' Status: '. $flag [$k].        Php_eol; }//Create log directory if (!is_dir (dirname ($this->logfile)) {mkdir (dirname ($this->logfile), 0777, True        );    }//write log file file_put_contents ($this->logfile, $tmp _log, file_append); }}?>

demo.php

<?phprequire ' BatchDownLoad.class.php '; $base _path = DirName (__file__). ' /photo '; $download _config = Array (    ' http://www.example.com/p1.jpg ', $base _path. ' /p1.jpg '),    array (' http://www.example.com/p2.jpg ', $base _path. ' /p2.jpg '),    array (' http://www.example.com/p3.jpg ', $base _path. ' /p3.jpg '),    array (' http://www.example.com/p4.jpg ', $base _path. ' /p4.jpg '),    array (' http://www.example.com/p5.jpg ', $base _path. ' /p5.jpg '), $obj = new Batchdownload ($download _config, 2, $handle _num = $obj->download (); echo ' Download num: '. $ Handle_num. Php_eol;? >

Post-execution log output

[2017-07-16 18:04:21] url:http://www.example.com/p1.jpg file:/home/fdipzone/photo/p1.jpg status:1[2017-07-16 18:04:21] url:http://www.example.com/p2.jpg file:/home/fdipzone/photo/p2.jpg status:1[2017-07-16 18:04:21] Url:http ://www.example.com/p3.jpg file:/home/fdipzone/photo/p3.jpg status:1[2017-07-16 18:04:21] url:http:// Www.example.com/p4.jpg file:/home/fdipzone/photo/p4.jpg status:1[2017-07-16 18:04:21] url:http://www.example.com/ P5.jpg file:/home/fdipzone/photo/p5.jpg status:1

This article explains some of the ways to download files in bulk, more about the content of the PHP Chinese web.