RollingcURL: PHP concurrency best practices

In actual projects or self-compiled gadgets (such as news aggregation, product price monitoring, and price comparison), data is usually obtained from the 3rd-party website or API, when you need to process one URL queue, you can use the curl_multi _ * family function provided by cURL to improve the performance... "> <LINKhref =" http://www.php100.com//stati

In actual projects or self-compiled gadgets (such as news aggregation, product price monitoring, and price comparison), data is usually obtained from the 3rd-party website or API, to improve the performance of a URL queue, you can use the curl_multi _ * family function provided by cURL to implement simple concurrency.

This article will discuss two specific implementation methods and compare the performance of different methods.

1. typical cURL concurrency mechanism and its problems

The Classic cURL implementation mechanism can be easily found online. for example, refer to the PHP online manual for the following implementation methods:

Function classic_curl ($ urls, $ delay ){
$ Queue = curl_multi_init ();
$ Map = array ();

Foreach ($ urls as $ url ){
// Create cURL resources
$ Ch = curl_init ();

// Set URL and other appropriate options
Curl_setopt ($ ch, CURLOPT_URL, $ url );

Curl_setopt ($ ch, CURLOPT_TIMEOUT, 1 );
Curl_setopt ($ ch, CURLOPT_RETURNTRANSFER, 1 );
Curl_setopt ($ ch, CURLOPT_HEADER, 0 );
Curl_setopt ($ ch, CURLOPT_NOSIGNAL, true );

// Add handle
Curl_multi_add_handle ($ queue, $ ch );
$ Map [$ url] = $ ch;
}

$ Active = null;

// Execute the handles
Do {
$ Mrc = curl_multi_exec ($ queue, $ active );
} While ($ mrc = CURLM_CALL_MULTI_PERFORM );

While ($ active> 0 & $ mrc = CURLM_ OK ){
If (curl_multi_select ($ queue, 0.5 )! =-1 ){
Do {
$ Mrc = curl_multi_exec ($ queue, $ active );
} While ($ mrc = CURLM_CALL_MULTI_PERFORM );
}
}

$ Responses = array ();
Foreach ($ map as $ url => $ ch ){
$ Responses [$ url] = callback (curl_multi_getcontent ($ ch), $ delay );
Curl_multi_remove_handle ($ queue, $ ch );
Curl_close ($ ch );
}

Curl_multi_close ($ queue );
Return $ responses;
}

 

First, all the URLs are pushed into the concurrent queue, and then the concurrent process is executed. after all the requests are received, the data will be parsed and processed. in actual processing, due to the impact of network transmission, the content of some URLs will take precedence over those returned by other URLs, but the classic cURL concurrency must wait for the slowest URL to return before processing, waiting means that the CPU is idle and wasted. if the URL queue is short, this idle and waste is still in the acceptable range, but if the queue is long, this waiting and waste will become unacceptable.

2. improved Rolling cURL concurrency

After careful analysis, it is not difficult to find that there is still room for optimization for the concurrency of the classic cURL. in the optimization mode, when a URL request is completed, process it as quickly as possible and wait for other URLs to return, instead of waiting for the slowest interface to return to start processing and other work, so as to avoid idle CPU and waste. I will not talk much about it. the specific implementation is shown below:

Function rolling_curl ($ urls, $ delay ){
$ Queue = curl_multi_init ();
$ Map = array ();

Foreach ($ urls as $ url ){
$ Ch = curl_init ();

Curl_setopt ($ ch, CURLOPT_URL, $ url );
Curl_setopt ($ ch, CURLOPT_TIMEOUT, 1 );
Curl_setopt ($ ch, CURLOPT_RETURNTRANSFER, 1 );
Curl_setopt ($ ch, CURLOPT_HEADER, 0 );
Curl_setopt ($ ch, CURLOPT_NOSIGNAL, true );

Curl_multi_add_handle ($ queue, $ ch );
$ Map [(string) $ ch] = $ url;
}

$ Responses = array ();
Do {
While ($ code = curl_multi_exec ($ queue, $ active) = CURLM_CALL_MULTI_PERFORM );

If ($ code! = CURLM_ OK) {break ;}

// A request was just completed -- find out which one
While ($ done = curl_multi_info_read ($ queue )){

// Get the info and content returned on the request
$ Info = curl_getinfo ($ done ['handle']);
$ Error = curl_error ($ done ['handle']);
$ Results = callback (curl_multi_getcontent ($ done ['handle']), $ delay );
$ Responses [$ map [(string) $ done ['handle'] = compact ('info', 'error', 'results ');

// Remove the curl handle that just completed
Curl_multi_remove_handle ($ queue, $ done ['handle']);
Curl_close ($ done ['handle']);
}

// Block for data in/output; error handling is done by curl_multi_exec
If ($ active> 0 ){
Curl_multi_select ($ queue, 0.5 );
}

} While ($ active );

Curl_multi_close ($ queue );
Return $ responses;
}

 

3. Performance comparison of two concurrent implementations

The performance comparison test before and after improvement is performed on the LINUX host. The Concurrent Queue used during the test is as follows:

Http://item.taobao.com/item.htm? Id = 14392877692

Http://item.taobao.com/item.htm? Id = 16231676302

Http://item.taobao.com/item.htm? Id = 17037160462

Http://item.taobao.com/item.htm? Id = 5522416710

Http://item.taobao.com/item.htm? Id = 16551116403

Http://item.taobao.com/item.htm? Id = 14088310973

Briefly describe the principles of the experiment design and the format of the performance test results: to ensure the reliability of the results, each group of experiments repeats 20 times, in a single experiment, the same interface URL set is given, measure the time consumption (in seconds) of the two concurrency mechanisms, Classic (the Classic concurrency mechanism) and Rolling (the improved concurrency mechanism), respectively ), calculate the time saved (Excellence, in seconds) and the performance improvement ratio (Excel. % ). in order to keep the actual request as close as possible while keeping the experiment simple, only a simple regular expression matching is performed on the processing of the returned results, without other complicated operations. in addition, to determine the impact of result processing callback on performance comparison test results, you can use usleep to simulate the data processing logic (such as extraction, word segmentation, writing files or databases) that is more appropriate in reality ).

The callback functions used in performance testing are:

Function callback ($ data, $ delay ){
Preg_match_all ('/(. +) <\/h3>/iU', $ data, $ matches );
Usleep ($ delay );
Return compact ('data', 'matches ');
}

 

When data processing callback is not delayed: Rolling Curl is slightly better, but the performance improvement is not obvious.

------------------------------------------------------------------------------------------------Delay: 0 micro seconds, equals to 0 milli seconds------------------------------------------------------------------------------------------------Counter         Classic         Rolling         Winner          Excellence      Excel. %------------------------------------------------------------------------------------------------1               0.1193          0.0390          Rolling         0.0803          67.31%2               0.0556          0.0477          Rolling         0.0079          14.21%3               0.0461          0.0588          Classic         -0.0127         -21.6%4               0.0464          0.0385          Rolling         0.0079          17.03%5               0.0534          0.0448          Rolling         0.0086          16.1%6               0.0540          0.0714          Classic         -0.0174         -24.37%7               0.0386          0.0416          Classic         -0.0030         -7.21%8               0.0357          0.0398          Classic         -0.0041         -10.3%9               0.0437          0.0442          Classic         -0.0005         -1.13%10              0.0319          0.0348          Classic         -0.0029         -8.33%11              0.0529          0.0430          Rolling         0.0099          18.71%12              0.0503          0.0581          Classic         -0.0078         -13.43%13              0.0344          0.0225          Rolling         0.0119          34.59%14              0.0397          0.0643          Classic         -0.0246         -38.26%15              0.0368          0.0489          Classic         -0.0121         -24.74%16              0.0502          0.0394          Rolling         0.0108          21.51%17              0.0592          0.0383          Rolling         0.0209          35.3%18              0.0302          0.0285          Rolling         0.0017          5.63%19              0.0248          0.0553          Classic         -0.0305         -55.15%20              0.0137          0.0131          Rolling         0.0006          4.38%------------------------------------------------------------------------------------------------Average         0.0458          0.0436          Rolling         0.0022          4.8%------------------------------------------------------------------------------------------------Summary: Classic wins 10 times, while Rolling wins 10 times

 

Data processing callback latency of 5 ms: Rolling Curl wins, performance is improved by about 40%.

------------------------------------------------------------------------------------------------Delay: 5000 micro seconds, equals to 5 milli seconds------------------------------------------------------------------------------------------------Counter         Classic         Rolling         Winner          Excellence      Excel. %------------------------------------------------------------------------------------------------1               0.0658          0.0352          Rolling         0.0306          46.5%2               0.0728          0.0367          Rolling         0.0361          49.59%3               0.0732          0.0387          Rolling         0.0345          47.13%4               0.0783          0.0347          Rolling         0.0436          55.68%5               0.0658          0.0286          Rolling         0.0372          56.53%6               0.0687          0.0362          Rolling         0.0325          47.31%7               0.0787          0.0337          Rolling         0.0450          57.18%8               0.0676          0.0391          Rolling         0.0285          42.16%9               0.0668          0.0351          Rolling         0.0317          47.46%10              0.0603          0.0317          Rolling         0.0286          47.43%11              0.0714          0.0350          Rolling         0.0364          50.98%12              0.0627          0.0215          Rolling         0.0412          65.71%13              0.0617          0.0401          Rolling         0.0216          35.01%14              0.0721          0.0226          Rolling         0.0495          68.65%15              0.0701          0.0428          Rolling         0.0273          38.94%16              0.0674          0.0352          Rolling         0.0322          47.77%17              0.0452          0.0425          Rolling         0.0027          5.97%18              0.0596          0.0366          Rolling         0.0230          38.59%19              0.0679          0.0480          Rolling         0.0199          29.31%20              0.0657          0.0338          Rolling         0.0319          48.55%------------------------------------------------------------------------------------------------Average         0.0671          0.0354          Rolling         0.0317          47.24%------------------------------------------------------------------------------------------------Summary: Classic wins 0 times, while Rolling wins 20 times

Based on the above performance comparison, Rolling cURL should be a more choice in the application scenario for processing URL queue concurrency, when the concurrency is very large (1000 +, you can control the maximum length of a concurrent queue, for example, 20. add one unrequested URL to the queue immediately after one URL is returned and processed, in this way, the written code will be more robust and won't be stuck or crashed due to the large number of concurrent jobs. for detailed implementation, see: http://code.google.com/p/rolling-curl/