Laravel Queue-Let daemons process time-consuming tasks

Issues to be solved

Recently, a Web project that is managing a server cluster needs to handle some extremely time-consuming operations, such as disk-formatted partitions. The first idea for this requirement is that, in order to make the RPC (Remote Procedure Call) service side of the node as simple as possible (simply to popen execute an instruction, there is time for me to write a blog about how RPC is implemented for this project), We chose to let the Web side directly wait for processing results, then the problem, how to ensure that users do not have to wait, but also to ensure that the task of accurate execution?

A simple RPC structure such as

In the past, to deal with some slightly time-consuming operations, you can optimize the code structure, optimize the number of database operations, some threads to handle some simple such as email, generate large compressed files, extract video thumbnail, inter-server visits and so on, to avoid users in the Web page waiting.

But now this operation is obviously not possible with the previous methods, because the current operation even if only one time, it is very time-consuming, not to mention may need to deal with hundreds of thousands of servers. This is hard to do at the thread level, knowing that in response to a request for a thread in the web process, after the disconnection of the TCP connection, the process is likely to be killed, like Apache, of course, can be configured to change the behavior of Apache, but obviously not very reliable.

A better way to do this is to start a daemon on the Web server, so the question is how to create the daemon, and fortunately Laravel helps us think about it.

Queue of Laravel

Laravel's queue defaults to handling multiple tasks in sync (synchronous), which is obviously not what we want. Since this project is using the laravel4.1 version, I chose BEANSTALKD to implement asynchronous processing of multiple tasks.

Where BEANSTALKD is a more professional queue service driver and is a resident backend service, we can submit the task to it through the interface it provides, and the daemon that it creates executes the queue.

Configuring the queue Execution Environment 1. Installing the BEANSTALKD Service

I developed a computer for the CentOS5.4, the version is relatively low, so the process of installation still encountered some trouble

First execute the following instruction

1 wget FTP://fr2.rpmfind.net/linux/epel/5/ppc/epel-release-5-4.noarch.rpm2 RPM-IVH epel-release-5-4noarch.rpm3yum  makecache4 Yum Search BEANSTALKD

But finally found that the software could not find, so the source of the Yum to replace 163

1 cd/etc/yum. REPOS.D2wget http://Mirrors.163.com/.help /centos5-base-163.repo3mv Centos-base.repo Centos-base.repo.bak

Once again Makecache && install is OK, after installation, start the BEANSTALKD service

1 Service BEANSTALKD start

In addition, you can search the BEANSTALKD configuration file under Sysconfig

Add BEANSTALKD driver for Laravel

BEANSTALKD's PHP driver package is Pda/pheanstalk

Enter Laravel's protected directory, Composer.json in this directory

Perform

1 2. *

The following error occurred

，

You can see the mirror address response 502, so you need to find a usable image for composer http://www.phpcomposer.com/

Modify ~/.composer/config.json as follows

Then go back to the protected directory, and then perform the installation of the previously installed driver command, this time there is a different error

The above PHP package, the first is to Phpstorm to laravel better support, the latter Symfony/yaml has been installed, do not need to upgrade, so modify the Composer.json, directly delete the two items on the line

After you delete, install the installation command again

Can see finally success, you can see which packages are installed through composer Show-i

Test

Add an action to TestController

1 classTestControllerextendsBasecontroller2 {3   4      Public functionGetqueue () {5         //6         Log:: Info ("Add a column task");7Queue::p ush (' SendEmail ',Array(' message ' = ' haha '));8         Log:: Info (' task added complete '));9         Exit;Ten     } One  A}

Create a new tasks directory under the app directory, and modify Protected/composer.json and app/global.php to add this directory to the class loading path

Modify global.php

1 classloader::adddirectories (array(23    App_path (). /commands ',4    App_path (). ' /controllers ',5    App_path (). ' /models ',6    App_path (). ' /database/seeds ',7    App_path (). ' /library ',8    App_path (). ' /tasks ',9 ));

Modify Composer.json

1 "Classmap": [2   "App/commands",3   "App/controllers",  4   "App/models",5   "App/database/migrations",6    "App/database/seeds",7   "App/tasks",8   "app/ tests/testcase.php "9 ]

The code for future time-consuming scheduling tasks is placed under this directory.

First create a new basetask.php

1 /**2 * Created by Phpstorm.3 * User:administrator4 * DATE:2015/8/19 00195 * time:11:556  */7 Abstract classBasetask8 {9      Public Abstract  functionFire ($job,$data);Ten}

Then create a new sendmail.php

1 /**2 * Created by Phpstorm.3 * User:administrator4 * DATE:2015/8/19 00195 * time:11:506  */7 classSendEmailextendsBasetask8 {9 Ten      Public functionFire ($job,$data) One     { A         //todo:implement Fire () method. -         Log:: Info ("Perform on column Tasks". Json_encode ($data)." Time: ". Time()); -         Sleep(30); the         Log:: Info ("Execute on column Tasks".) Time()); -         //flush a task from a queue -         $job-Delete (); -         //to return a task to a queue +  - //$job->release (); +  A     } at}

Finally, to modify the configuration file queue.php file in the Config directory, change to

1 ' default '   = ' beanstalkd ',

1 Array (2    ' driver ' = ' beanstalkd ',3    ' host '   = ' localhost ',4    ' queue '  = ' default ',5    ' ttr '   =    6 ,

Explanation about TTR There's a http://stackoverflow.com/questions/25791633/change-beanstalkd-default-ttrhttp://in355hz.iteye.com here. The/blog/1395727 represents time to run, which overrides the default parameters so that BEANSTALKD detects whether the configuration and write code are completed in this timeframe, executed in the shell, and in the protected directory, Artisan files in this directory

1 php artisan queue:work2 PHP artisan Queue:listen

Then visit Url,http://192.168.1.10/ssanlv/test/queue, you can find that the request is complete immediately, the page does not wait

View Protected/app/storage/logs/laravel.log to see the following content

508-478 just 30 seconds.

The following test a practical problem, the impression that the Apache server and the client after the request has been disconnected will kill the processing of the httpd process, only the configuration of the keep-alive parameter in the process will be retained in the Apache process pool, but after the user requests a time-consuming operation, When the browser is closed, will the daemon that handles the time-consuming task be killed too? Of course, in fact, a little more worried, when the response is completed after the TCP link has been disconnected, if the process will be killed, then kill off, and your browser close should not have much relationship, or try it, practice is the hard truth

This will change the sleep time in sendmial to 600 seconds.

Finally found not finished, you can see the listen reported abnormal

Obviously the execution timeout, which appears to be the TTR problem set earlier

The TTR comment out or modify the higher value, found or not, finally in a careful look at the error message, found

So change the way the command executes

1 php artisan queue:listen--timeout=

The final command completes the successful execution of the task

can see 1353-753 = 600 just fine

In addition, it seems that the task to the column should be saved, when I did not start listen, the task will not be processed, but I start listen, the previous task will be executed immediately

But there's still a problem. this is the processing of the column form, to start the next pair of column tasks, you must wait for the last column task to complete , but I have seen before, a work for a task queue, then I can complete a number of task queue, a bit of multi-core CPU scheduling OH.

Better way.

Finally, a discussion with one of the great gods explored another better approach, although it would add to the complexity of the RPC service Code on the node, but it would not be cumbersome.

This is the callback, the management of the cluster's Web server can not wait, just the following steps,

• An instruction is sent to the RPC service on the node where the time-consuming operation is performed through the RPC client on the Web server.
• After the RPC service on the node receives the instruction, instead of executing the instruction, it immediately returns a task ID to the Web server, the RPC client.
• The Web server saves this ID as a task record.
• RPC Service processing instructions on the node, as for the processing instructions, we also discuss on the node in a separate process P to handle, because the RPC service also can not let RPC client silly wait
• After processing P has finished processing, the execution result and task ID are used as parameters to callback the Web server's web interface
• After the Web server receives a callback from the RPC service, it finds the task by ID, updates the execution status of the task, updates the data

Obviously, this way more reliable, but also greatly reduce the burden of the Web server, to know that the number of threads of the Linux system is limited, but it takes a lot of time, such as the server to wait, no matter what strategy is probably the server destroy.

Laravel Queue-Let daemons process time-consuming tasks