Installation and Use of Gearman in Linux, distributed Message Queue (CentOS-6.5: gearmand-1.1.12)
1 Gearman Introduction
1.1 Overview Gearman is a machine used to delegate jobs to other machines. Distributed calls are more suitable for jobs and concurrent jobs to achieve load balancing among multiple calls, or a system used to call functions in other languages.
1.2 Gearman is a program architecture for dispatching tasks. It consists of three parts:
1) Gearman client: Provides the gearman client API for application calls. APIS can use C, PHP, PERL, and MYSQL udfs as the request initiator.
2) Gearman job server: distributes client requests to the schedulers of each gearman worker, which is equivalent to a central controller, but does not process specific business logic.
3) Gearman worker: Provides the gearman worker API for application calls. It is responsible for client requests and returns the processing results to the client.
1.3 applications
The core of Mogilefs's distributed file system is implemented using gearman.
This software has many application scenarios, such as video processing, distributed log processing, email processing, file synchronization processing, and image processing for video websites, A scenario that does not affect the experience and response. It is okay for programs that require parallel computing and processing. Yahoo uses gearman to process 6 million jobs per day on 60 or more servers. The news aggregators digg builds a gearman network of the same size and can process 400000 jobs every day.
Gearman can be used not only for task distribution, but also for application load balancing. You can place worker on different servers or start multiple cores of the same cpu. For example, if you do not want a web server to process video format conversion, You can distribute tasks on the server and load worker to process the video format, the external web server will not be affected by the video conversion process. In addition, it is convenient to add a server to the task scheduling center and register it as a worker. At this time, the job server sends the request to the idle worker when the request arrives. You can also run multiple job servers to form an ha architecture. If a job server fails, the client and worker will automatically migrate to another job server.
1.4 Working principle diagram
2. Running Process
The processing of a Gearman request involves three roles: Client-> Job-> Worker.
Client: The request initiator, which can be C, PHP, Perl, MySQL UDF, etc.
Job: The request scheduler, used to coordinate the forwarding of requests sent by the Client to appropriate Work.
Worker: The request processor, which can be C, PHP, Perl, and so on.
Because the Client and Worker do not limit the use of the same language, it is conducive to the integration between multiple systems in multiple languages.
Even by adding more workers, we can easily implement the Distributed Load Balancing architecture of applications.
3 download Gearman 1) Official Website
Http://gearman.org/
2) download from the official website
Https://launchpad.net/gearmand
3) Official Website user guide
Http://gearman.org/getting-started/
3) all the software used in this installation (the installation environment is CentOS-6.5)
Http://download.csdn.net/detail/clevercode/8698699
4 install Gearman
4.1 install a common library required for linux
Installation of common support libraries in Linux: http://blog.csdn.net/clevercode/article/details/45438401
4.2 install the library on which gearmand depends # yum install-y boost-devel gperf libevent-devel libuuid-devel
4.3 install the gearmand Service 1) decompress
# Cd/usr/local/src/gearman
# Tar xzf gearmand-1.1.12.tar.gz
2) Configuration
# Cd gearmand-1.1.12
#./Configure
3) Compile
# Make
4) Installation
# Make install
5) install successfully. Enter
# Gearman
5 install php extension 1) install phpize
# Yum install-y php-devel
2) decompress
# Cd/usr/local/src/gearman
# Tar xzf gearman-1.1.2.tgz
3) Configuration
# Cd gearman-1.1.2
# Phpize
#./Configure
4) Compile
# Make
5) Installation
# Make install
6) Installation successful
If "Installing shared extensions:/usr/lib64/php/modules/" is displayed, the installation is successful./usr/lib64/php/modules/is the directory of the gearman. so extension.
7) configuration (add extension)
# Vi/usr/local/php5/etc/php. ini
Extension = "gearman. so"
8) check whether the configuration is successful
# Vi test. php
Print gearman_version (). "\ n ";
?>
After php test. php is executed, 1.1.12 indicates that the installation is successful.
# Php test. php
1.1.12
6. Start and Stop Gearman
1) Create logs/data0/logs/gearmand. log
# Touch/data0/logs/gearmand. log
2) Start
#/Usr/local/sbin/gearmand-d-u root-L 192.168.142.130 -- log-file =/data0/logs/gearmand. log
3) Parameter Details
-B, -- backlog = number of reserved listening connections
-D, -- daemon running in the background
-F, -- file-descriptors = number of file descriptors
-H, -- help
-J, -- job-retries = number of times the job runs before the ob server removes the unavailable job to prevent other available worker from crashing due to continuous running. No limit by default
-L,-log-file = location where the log file is stored (the simplest log is recorded by default)
-L, -- listen = listener IP address, all accepted by default
-P, -- port = specifies the listening port
-P, -- pid-file = specifies the write location of the process ID
-R, -- protocol = load protocol module
-Q, -- queue-type = Specify the persistent queue
-T, -- threads = the number of I/9 threads used. The default value is 0.
-U, -- user = after startup, switch to the specified user
-V, -- verbose adds Level 1 Details
-V, -- version: displays version information.
4) Check for running
# Ps axu | grep gearmand
5) view the listening port
# Netstat-anp | grep 4730
6) stop and kill the process directly.
7. Use Gearman
7.1 create a Worker to create worker. php and a Worker to send emails. The Code is as follows:
AddServer ('2017. 168.142.130 ', '123'); $ worker-> addFunction ("sendMail", "my_sendmail_function"); while ($ worker-> work (); function my_sendmail_function ($ job) {// receive data $ tmp = $ job-> workload (); $ receiveArr = unserialize ($ tmp); $ from = $ receiveArr ['from']; $ to = $ receiveArr ['to']; $ subject = $ receiveArr ['subobject']; $ content = $ receiveArr ['content']; // send an email //.... return $ subject. 'sendmail OK ';}?>
7.2 start Worker
If a large amount of data is processed, run the following Script Multiple times to start multiple Worker ends.
# Nohup php worker. php> tmp.txt &
7.3 create a Client (blocking mode, which ends only after the returned results) and create a client. php. The do () method is the blocking mode. You must wait for the worker to return the result before the program can stop. (Return: hello Gearman sendmail OK)
AddServer ('2017. 168.142.130 ', '123'); $ job = array (); $ job ['from'] = 'clevercode'; $ job ['to'] = 'gearman '; $ job ['object'] = 'Hello Gearman '; $ job ['content'] = 'Hello Gearman: this is from gearmanclient'; $ job = serialize ($ job ); // wait until the worker side returns the result. $ Ret = $ client-> do ("sendMail", $ job); echo $ ret. "\ r \ n";?>
7.4 create a Client (non-blocking, no need to wait for the result) to create a client2.php. DoBackground () does not have to wait for the worker side to return results, and the program ends.
AddServer ('2017. 168.142.130 ', '123'); $ job = array (); $ job ['from'] = 'clevercode'; $ job ['to'] = 'gearman '; $ job ['object'] = 'Hello Gearman '; $ job ['content'] = 'Hello Gearman: this is from gearmanclient'; $ job = serialize ($ job ); // if you do not wait for the returned result, $ ret = $ client-> doBackground ("sendMail", $ job); echo $ ret. "\ r \ n";?>
8. Run the following command on Gearman management to check port 4730.
# (Echo "status"; sleep 2) | telnet 192.168.142.130 4730
1) Field Description: "known registered tasks" "running tasks" "tasks in the queue" "available Worker ".
2) sendMail 0 0 1. The registered Task Name is sendMail. 0 tasks are running normally, the queue is empty, and there is an available Worker.
