Distributing workloads for PHP applications with Gearman

While much of the content of a WEB application is related to presentation, its value and competitive advantage may be reflected in a number of proprietary services or algorithms. If such processing is too complex or tedious, it is best to execute asynchronously so that the WEB server does not respond to incoming requests. In fact, it is better to put a computationally intensive or specialized feature on one or more separate dedicated servers.

PHP's Gearman library can distribute work to a group of machines. Gearman queues up jobs and dispatches jobs in small amounts, and distributes those complex tasks to machines reserved for this task. This library is available to Perl, Ruby, C Python, and PHP developers, and can also run on any UNIX®-like platform, including Mac OS X, Linux®, and Sun Solaris.

Adding Gearman to a PHP application is straightforward. Assuming you have a PHP application hosted on a typical LAMP configuration, Gearman will need an additional daemon and a php extension. As of November 2009, the latest version of the Gearman daemon is 0.10, and two PHP extensions are available-one with PHP wrapped in Gearman C Library and the other in pure PHP. We are going to use the former. Its latest version is 0.6.0, which can get its source code from PECL or Github (see Resources).

Note: for this article,producer refers to the machine that generated the work request;consumer is the machine that performs the work, while the agent is the connection producer with the appropriate The intermediary of consumer.

Installing Gearman

Adding Gearman to a machine takes two steps: The first step is to build and launch the daemon, and the second step is to build a PHP extension that matches the PHP version. This daemon package includes all the libraries needed to build this extension.

First, download the latest source code for the Gearman daemon, gearmand unzip the tarball, build and install this code (installation requires Superuser privileges, root user rights).

$ wget http://launchpad.net/gearmand/trunk/\ 0.10/+download/gearmand-0.10.tar.gz $ tar xvzf gearmand-0.10.tar.gz $ CD gearmand-0.10 $./configure $ make $ sudo make install

gearmandafter installation, build the PHP extension. You can get this tarball from PECL, or you can copy the repository from Github.

$ wget http://pecl.php.net/get/gearman-0.6.0.tgz $ CD Pecl-gearman # # or # $ git clone git://github.com/php/pecl-gearman.git $ CD Pecl-gearman

With this code, you can start building the extension:

$ phpize $./configure $ make $ sudo make install

This Gearman daemon is usually installed in/usr/sbin. You can start the daemon directly from the command line, or you can add the daemon to the boot configuration so that the daemon can be started each time the machine restarts.

Next, you need to install the Gearman extension. Open the php.ini file (you can php --ini quickly find the file by command) and add a line of code extension = gearman.so :

$ PHP--ini Loaded Configuration File: /etc/php/php.ini ... Extension = gearman.so

Save the file. To verify that the extension is enabled, run and php --info then look for Gearman:

$ PHP--info | grep "Gearman Support" Gearman Gearman support = Enabled Libgearman Version = 0.10

In addition, you can use a PHP snippet to verify that the build and installation are appropriate. Save this small application to verify_gearman.php:

<?php Print gearman_version (). "\ n"; ?>

Next, run the program from the command line:

$ PHP verify_gearman.php 0.10

If this version number matches the version number of the Gearman library that was previously built and installed, then the system is ready.

Run Gearman

As we mentioned earlier, a Gearman configuration has three roles:

• One or more producer generate a work request. Each work request names the function that it wants, for example, email_all or analyze .
• One or more consumer complete the request. Each consumer names one or more of the functions it provides and registers these functions with the agent. A consumer can also be called a worker.
• The agent centralizes all the services provided by the consumer that connect with it. It links the producer with the appropriate consumer.

With the following command line, you can now experience Gearman:

2. Start the agent, the Gearman daemon:

   $ Sudo/usr/sbin/gearmand--daemon

4. Run a worker with the command-line utility gearman . This worker needs a name and can run any command-line utility. For example, you can create a worker to list the contents of a directory. -fparameter names the function provided by the worker:

   $ gearman-w-F LS--LS-LH

5. The last block of code is a producer, or a job used to generate a lookup request. can also be usedgearmanGenerates a request. Similarly, with-foption to specify the service you want to get help from:
   
   
   
   
   
   $ gearman-f ls </dev/null[email protected] Supergiantrobot staff 1.4K Nov 15:07 gearman-0.6.0[email protected] 1 supergiant Robot staff 29K Oct 1 04:44 gearman-0.6.0.tgz[email protected] 1 Supergiantrobot St AFF 5.8K Nov 15:32 gearman.html[email protected] Supergiantrobot staff 1.1K Nov 14:04 gearmand-0.10[email protected] 1 Supergiantrobot staff 5.3K Jan 1 1970 PAC KAGE.XMLdrwxr-xr-x Supergiantrobot staff 1.6K Nov 14:45 pecl-gearman

Using Gearman from PHP

Using Gearman from PHP is similar to the previous example, the only difference being that this is the creation of producer and consumer within PHP. Each consumer work is encapsulated within one or more PHP functions.

Listing 1 shows a Gearman worker written in PHP. Save the code in a file named worker.php .

Listing 1. worker.php

<?php $worker = new Gearmanworker (); $worker->addserver (); $worker->addfunction ("title", "Title_function"); while ($worker->work ()); function Title_function ($job) { Return Ucwords (Strtolower ($job->workload ())); } ?>

Listing 2 shows a producer, or client, written in PHP. Save this code in a file named client.php .

Listing 2. client.php

<?php $client = new Gearmanclient (); $client->addserver (); Print $client->do ("title", "All the world ' s STagE"); print "\ n"; ?>

Now, you can connect the client and worker with the following command line:

$ PHP worker.php & $ PHP client.php All the world ' s A Stage $ jobs [3]+ Running php worker.php &

The worker application continues to run and is ready to serve another client.

Advanced Features of Gearman

Gearman may be used in many places within a WEB application. You can import large amounts of data, send many e-mails, encode video files, dig data, and build a central logging facility-all without compromising the experience and responsiveness of your site. Data can be processed in parallel. Also, because the Gearman protocol is language-and platform-independent, you can mix programming languages in your solution. For example, you can write a producer in PHP with a C worker in Ruby or any other language that supports the Gearman library.

A Gearman network that connects the client to the worker can actually use any structure you can imagine. Many configurations can run multiple agents and assign workers to many machines. Load balancing is implicit: each actionable worker (possibly with more than one worker per worker host) pulls the job out of the queue. A job can run synchronously or asynchronously and has a priority.

The latest version of Gearman has extended the system features to include durable job queues and a new protocol to submit work requests over HTTP. For the former, the Gearman Task Force column is saved in memory and stored in a relational database for backup. This way, if the Gearman daemon fails, it can recreate the task queue after a reboot. Another recent improvement is to increase queue persistence through a memcached cluster. Memcached storage is also dependent on memory, but is dispersed over several machines to avoid a single point of failure.

Gearman is a just-starting but very powerful work distribution system. According to Gearman's author, Eric day, Yahoo! uses Gearman on 60 or more servers to process 6 million jobs daily. News aggregator Digg has built a Gearman network of the same size and can handle 400,000 jobs a day. A good example of Gearman can be found in Narada, an open source search engine (see Resources).

Future versions of Gearman will collect and report statistics, provide advanced monitoring and caching job results, and more. To track this Gearman project, you can subscribe to its Google group, or access its IRC channel on Freenode #gearman .

Distributing workloads for PHP applications with Gearman