Introduction to the PHP concurrent multi-process processing tool Gearman and the tool gearman

Introduction to the PHP concurrent multi-process processing tool Gearman and the tool gearman

At work, we sometimes encounter the need to simultaneously publish data to multiple servers, or simultaneously process multiple tasks. The PHP curl_multi method can be used to concurrently process requests. However, the response time of concurrent processing is slow due to network and data, various servers, and so on, because the process of Concurrent Requests also includes logging, processing data, and other logic, waiting for processing results and returning, it cannot meet the background operation experience.

There is another solution that leverages Gearman to implement concurrency requirements. The Client sends requests to Jobs of Gearman, and then performs curl_multi, data processing, logs, and other operations in each Work. At the same time, supervisor is used to monitor the processes of Gearman and Works, in this way, a parallel multi-process and load balancing solution can be implemented.

What can Gearman do:

Asynchronous processing: image processing, order processing, batch mail/notification, etc.
High CPU or memory processing: large-capacity data processing, MapReduce operations, log aggregation, and video encoding
Distributed and Parallel Processing
Scheduled processing: incremental update and Data Replication
Maximum rate of FIFO Processing
Distributed System monitoring tasks

How Gearman works:
Applications using Gearman generally consist of three parts: a Client, a Worker, and a Task Server. The role of the Client is to submit a Job task to the Job Server. The Job Server will find a suitable Worker to complete this task. The Worker executes the Job sent by the Client and returns the result to the Client through the Job Server. Gearman provides Client and Worker APIs, which can be used to communicate with Gearman Job Server. The communication between the Client and Worker in Gearman is performed through a TCP connection.

Gearman can share the workload to different machines.

Installation:

Copy codeThe Code is as follows:
Rpm-ivh http://dl.iuscommunity.org/pub/ius/stable/Redhat/6/x86_64/epel-release-6-5.noarch.rpm
Yum install-y gearmand

Start:
Gearmand-d

Install PHP Gearman Extension
I use pcel for installation. You can also download the source package to compile and install it. But remember to install libgearman and re2c first. Otherwise, the extended compilation and installation will fail.

Pecl install gearman # not successful and prompt version problems can try pecl install gearman-1.0.3, the default seems to be 1.1.2
Compilation and installation are also simple
Copy codeThe Code is as follows:
Http://pecl.php.net/get/gearman-1.1.1.tgz wget-c
Tar zxvf gearman-1.1.1.tgz
Phpize
./Configure
Make & make install
Echo "extension = gearman. so">/etc/php. ini

PHP interface functions
Gearman provides many complete extension functions, including GearmanClient, GearmanJob, GearmanTask, and GearmanWorker. For details, refer to the official PHP manual.
This is one of the official Example, equivalent to an Example of concurrent distribution task processing.

<?php$client = new GearmanClient();$client->addServer();// initialize the results of our 3 "query results" here$userInfo = $friends = $posts = null;// This sets up what gearman will callback to as tasks are returned to us.// The $context helps us know which function is being returned so we can// handle it correctly.$client->setCompleteCallback(function(GearmanTask $task, $context) use (&$userInfo, &$friends, &$posts) {switch ($context){case 'lookup_user':$userInfo = $task->data();break;case 'baconate':$friends = $task->data();break;case 'get_latest_posts_by':$posts = $task->data();break;}});// Here we queue up multiple tasks to be execute in *as much* parallelism as gearmand can give us$client->addTask('lookup_user', 'joe@joe.com', 'lookup_user');$client->addTask('baconate', 'joe@joe.com', 'baconate');$client->addTask('get_latest_posts_by', 'joe@joe.com', 'get_latest_posts_by');echo "Fetching...\n";$start = microtime(true);$client->runTasks();$totaltime = number_format(microtime(true) - $start, 2);echo "Got user info in: $totaltime seconds:\n";var_dump($userInfo, $friends, $posts);

Gearman_work.php

<?php$worker = new GearmanWorker();$worker->addServer();$worker->addFunction('lookup_user', function(GearmanJob $job) {// normally you'd so some very safe type checking and query binding to a database here.// ...and we're gonna fake that.sleep(3);return 'The user requested (' . $job->workload() . ') is 7 feet tall and awesome';});$worker->addFunction('baconate', function(GearmanJob $job) {sleep(3);return 'The user (' . $job->workload() . ') is 1 degree away from Kevin Bacon';});$worker->addFunction('get_latest_posts_by', function(GearmanJob $job) {sleep(3);return 'The user (' . $job->workload() . ') has no posts, sorry!';});while ($worker->work());

I have executed gearman_work.php in all three terminals.

ryan@ryan-lamp:~$ ps aux | grep gearman* | grep -v grepgearman 1504 0.0 0.1 60536 1264 ? Ssl 11:06 0:00 /usr/sbin/gearmand --pid-file=/var/run/gearman/gearmand.pid --user=gearman --daemon --log-file=/var/log/gearman-job-server/gearman.log --listen=127.0.0.1ryan 2992 0.0 0.8 43340 9036 pts/0 S+ 14:05 0:00 php /var/www/gearmand_work.phpryan 3713 0.0 0.8 43340 9036 pts/1 S+ 14:05 0:00 php /var/www/gearmand_work.phpryan 3715 0.0 0.8 43340 9036 pts/2 S+ 14:05 0:00 php /var/www/gearmand_work.php

To view the result shell of running gearman_work.php.

Copy codeThe Code is as follows:
Fetching...
Got user info in: 3.03 seconds:
String (59) "The user requested (joe@joe.com) is 7 feet tall and awesome"
String (56) "The user (joe@joe.com) is 1 degree away from Kevin Bacon"
String (43) "The user (joe@joe.com) has no posts, sorry! "

Seeing the above 3.03 seconds, it means that the client request's previous tasks have been distributed and executed in parallel.
In the actual production environment, we can use the Supervisor tool to monitor the gearmand and work processes without being unexpectedly exited.