Distributed Task Distribution framework-Gearman

Distributed Task Distribution framework-Gearman

Official documentation: http://gearman.org/getting-started/ installation methods and examples are available, you can take a look.

Gearman is a program framework for task distribution. It can be used in various scenarios. Compared with Hadoop, Gearman prefers task distribution. Its task distribution is very simple, which can be simply completed by using scripts. Gearman was originally used for the image resize function of LiveJournal. Because the image resize consumes a lot of computing resources, it needs to be scheduled to run on multiple backend servers. After the task is completed, it returns to the front end and then presents it to the interface.

What can Gearman do?

• Asynchronous processing: image processing, order processing, batch mail/notification, etc.
• High CPU or memory processing: large-capacity data processing, MapReduce operations, log aggregation, and video encoding
• Distributed and Parallel Processing
• Scheduled processing: incremental update and Data Replication
• Maximum rate of FIFO Processing
• Distributed System monitoring tasks

1. Installation Method

# Install gearmanyum install gearmand # install libgearmanyum install libgearman-devel # install gearman php extension pecl install gearman # Add gearman. so to php. iniecho "extension = gearman. so ">/etc/php. ini

Install it without pecl

yum install php-pear

If pecl cannot be installed, you can directly download the source code and use phpize to compile and install it.

 

cd ~/softwarewget http://pecl.php.net/get/gearman-1.1.2.tgzcd gearman-1.1.2phpize./configuremake && make install 

2. Simple Example: (we use Gearman to asynchronously process Baidu cloud PUSH Service)

First check the client. php registration event, client. php

<? Php // create a Gearman object $ client = new GearmanClient (); // The default value of addServer is localhost and the default value of port is 4730. If not, adjust $ client-> addServer (); // $ client-> addServer ("192.168.0.0", 4730); echo "Sending job \ n"; $ username = "test"; $ message = "message "; $ data = array ("username" => $ username, "message" => $ message,); // register events and PASS Parameters, multiple parameters can be converted using json_encode // the task can be run in a blocking manner, and the priority can be specified; of course, it can also be non-blocking (running does not wait for the result) run // you can refer to php Gearman api documentation: doNornal , DoHigh, doLow, doBackground $ result = $ client-> doBackground ("testFunction", json_encode ($ data); if ($ result) {echo "Success: $ result \ n ";}?>

Let's look at worker's processing. In fact, it is to register the event processing function, worker. php

<? Php $ worker = new GearmanWorker (); $ worker-> addServer (); // registers events and event processing functions $ worker-> addFunction ("testFunction", "handler "); // run workerwhile ($ worker-> work (); function handler (GearmanJob $ job) {$ workload = json_decode ($ job-> workload (); echo "received: ". print_r ($ workload, 1) ;}?>

You can run php client on the command line. php, php worker. php. it is best to start worker first to handle the event in a timely manner. You can also start the client first to queue the event in Gearman and wait for worker to process it;

You can also start multiple workers. Gearman automatically performs load balancing and assigns it to different workers for processing.

You can explore more application scenarios...

For example, use Gearman to synchronize data from MySQL to Redis (asynchronous replication)

Gearman monitoring:

You can use a supervisor or gearman manager.

For examples of using supervisor to monitor Gearman tasks, see:

This article permanently updates the link address: