Gearman distributed asynchronous Task distribution framework

What is Gearman?

Gearman provides a generic application framework to farm off work to other machines or processes that is better suited to Do the work. It allows your to do work in parallel, to the load balance processing, and to the call functions between languages. It can be used in a variety of applications, from high-availability Web sites to the transport of database replication Eve Nts. In other words, it's the nervous system for how distributed processing communicates. A few strong points about Gearman:

• Open Source It ' s free! (in both meanings of the word) Gearman has a active open source community that's easy-to-get involved with if you need help or want to contribute. Worried about licensing? Gearman is BSD.
• Multi-Language -there is interfaces for a number of languages, and the this list is growing. You also has the option to write heterogeneous applications with clients submitting work in one language and workers perf Orming that work in another.
• flexible -tied to any specific design pattern. You can quickly put together distributed applications using any model you choose, one of the those options being map/reduce.
• Fast -Gearman have a simple protocol and interface with an optimized, and threaded, server written in C + + to mi Nimize your application overhead.
• embeddable -Since Gearman is fast and lightweight, it's great for applications of all sizes. It's also easy-to-introduce into existing applications with minimal overhead.
• No Single point of failure-Gearman can not be only to help scale systems, but can do it in a fault tolerant.
• No limits on message size -Gearman supports single messages up to 4gig in size. Need to do something bigger? No problem Gearman can chunk messages.
• worried about scaling? -Don ' t worry about it with Gearman. Craig's List, Tumblr, Yelp, Etsy,... Discover what others has known for years.

Content is being updated regularly, so please check back often. Also want to check out other forms of communication if you would like to learn more or get involved!

How does Does Gearman work?

A Gearman powered application consists of three parts:a client, A worker, and a job server. The client is responsible for creating a job to being run and sending it to a job server. The job server would find a suitable worker that can run the job and forwards the job on. The worker performs the requested by the client and sends a response to the client through the job server. Gearman provides client and worker APIs that your applications call-to-talk with the Gearman job server (also known as GEA Rmand) so you don ' t need to deal with networking or mapping of jobs. Internally, the Gearman client and worker APIs communicate with the job server using TCP sockets. To explain how Gearman works in more detail, lets look at a simple application that'll reverse the order of characters I n a string. The example is given in PHP, although other APIs would look quite similar.

We start off by writing a client application this is responsible for sending off the job and waiting for the result so It can print it out. It does the Gearman client API to send some data associated with a function name, the the function& nbsp reverse . The code for here (with error handling omitted for brevity):

 <?php//Reverse Client code $client Span class= "o" >= new gearmanclient ();  $client ->addserver ();  $client ->do (" Hello world! "               

This code initializes a client class, configures it to use a job server with add_server (no arguments means 127.0.0.1 Default port), and then tells the client API to run of reverse the function with the workload "Hello world!". The function name and arguments is completely arbitrary as far as Gearman be concerned, so if could send any data struct Ure that's appropriate for your application (text or binary). At the Gearman client API would package up the job into a Gearman protocol packet and send it to the job server To find a appropriate worker that can run the reverse function. Let's now look at the worker code:

<?phpReverse Worker Code$worker = new gearmanworker();  $worker-Addserver();  $worker-addfunction("reverse", function ($job) { return strrev($job-Workload());});  while ($worker, work());           

This code defines a function  my_reverse_function  that takes a string and returns the reverse of That string. It is used by a worker object to register a function named  reverse  after It was setup to connect to The same local job server as the client. When the job server receives the job is run, it looks at the list of workers who has registered the function name  ; Reverse  and forwards the job on to one of the free workers. The Gearman worker API then takes this request, runs the function  my_reverse_function , and sends the RES Ult of this function back through the job server to the client.

As can see, the client and worker APIs (along with the job server) deal with the job management and network Communicat Ion so can focus on the application parts. There a few different ways you can run jobs in Gearman, including background for asynchronous processing and prioritized J Obs. See the documentation available for the various APIs for details.

How is Gearman useful?

The reverse example above seems like a lot of work to run a function, but there is a number of ways this can is useful . The simplest answer is so you can use Gearman as an interface between a client and a worker written in different Languag Es. If you want your PHP Web application to call a function written in C and you could use the PHP client API with the C worker a PI, and stick a job server in the middle. Of course, there is more efficient ways of doing the (like writing a php extension in C), but could you want a PHP client And a Python worker, or perhaps a MySQL client and a PERL worker. You can mix and match any of the supported language interfaces easily, you just need all applications to being able to unders Tand the workload being sent. Is your favorite language not supported yet? Get involved with the project, it's probably fairly easy for either you or one of the existing Gearman developers to Put a language wrapper on top of the C library.

The next-to-Gearman can useful is-put the worker code on a separate machine (or cluster of machines) that's Better suited to doing the work. Say your PHP Web application wants to does image conversion, but this is too much processing to run it on the Web server mac Hines. Could instead ship the image off to a separate set of workers machines to does the conversion, this is the load does not Impact the performance of your Web server and other PHP scripts. By doing this, you also get a natural form of load balancing since the job server only sends new jobs to idle workers. If all of the workers running on a given machine is busy, you don ' t need to worry about new jobs being sent there. This makes scale-out with multi-core servers quite simple:do do you have the cores on a worker machine? Start up to instances of your worker (or perhaps more if they is not CPU bound). It is also seamless to add new machines to expand your worker pool, just boot them up, install the worker code, andThere are them connect to the existing job server.

Now you ' re probably asking what if the job server dies? You is able to run multiple job servers and has the clients and workers connect to the first available job server they a Re configured with. This is the if one job server dies, clients and workers automatically fail over to another job server. You probably don ' t want to run too many job servers, but have both or three is a good idea for redundancy. The diagram to the left shows how a simple Gearman cluster.

From here the can scale out your clients and workers as needed. The job servers can easily handle has hundreds of clients and workers connected at once. You can draw your own physical (or virtual) machine lines where capacity allows, potentially distributing load to any numb Er of machines. For more details on specific uses and installations, see the sections on examples.

Support multiple languages, client can be PHP, worker can be any language;

Gearman distributed asynchronous task distribution framework