Introduction to the Laravel queue system

This article mainly introduces the content is about the laravel of the queue system Introduction, has a certain reference value, now share to everyone, the need for friends can refer to

The Laravel queue provides a unified API for different background queue services, such as Beanstalk,amazon SQS, Redis, and even other relational database-based queues. The goal of the queue is to delay processing time-consuming tasks, such as sending messages, to drastically shorten Web requests and the corresponding time.

The queue configuration file is stored in the config/queue.php . Each queue-driven configuration can be found in the file, including databases, BEANSTALKD, Amazon SQS, Redis, and synchronous (local-use) drivers. It also contains a null queue driver for those tasks that abandon the queue.

Connect Vs. Queue

It is important to understand the difference between "connection" and "queue" before starting to use the Laravel queue. In your config/queue.php configuration file, there is a connections configuration option. This option defines a unique connection for Amazon SQS, Beanstalk, or a back-end service such as Redis. Either way, a given connection may have multiple "queues", while a "queue" can be considered to be a different stack or a large number of queue tasks.

Note that queue a property is included in the configuration example for each connection in the configuration file queue . This is the default queue, which is distributed to the queue when the task is sent to the specified connection. In other words, if you do not explicitly define a queue when you distribute the task, it is placed queue in the queue defined by the properties in the connection configuration:

This task will be distributed to the default queue ... dispatch (new job);//This task will be sent to the "emails" queue ... dispatch ((New Job)->onqueue (' emails '));

Some applications may not need to send tasks to different queues, but only to a simple queue. However, it is still very useful to push tasks to different queues because the Laravel queue processor allows you to define the priority of the queue, so you can prioritize different queues or distinguish different ways of handling different tasks. For example, if you push the task into the high queue, you can have the queue processor prioritize the tasks:

PHP Artisan queue:work--queue=high,default

Required settings for the drive

Database

To use database This queue driver, you need to create a data table to store the task, and you can use queue:table this Artisan command to create a migration of this data table. Once the migration is created, you can use migrate this command to create a data table:

PHP Artisan queue:tablephp Artisan Migrate

Redis

In order to use the redis queue driver, you need to config/database.php configure the Redis database connection in your configuration file

If your Redis queue connection is using a Redis cluster, your queue name must contain a key hash tag. This is to ensure that all redis keys are placed in the same hash for a given queue:

' Redis ' = [    ' driver ' = ' redis ',    ' connection ' = ' default ',    ' queue ' = ' = ' {default} ',    ' Retry_after ' = 90,],

Other queue-driven dependency expansion packs

Before you use the queue service in the list, you must install the following dependent expansion packs:

• Amazon SQS:aws/aws-sdk-php ~3.0

• BEANSTALKD:pda/pheanstalk ~3.0

• Redis:predis/predis ~1.0

Create a task

Build Task Class

In your application, the queue's task class is placed by default in the app/Jobs directory, and if the directory does not exist, the directory will be created automatically when you run the make:job artisan command. You can use the following Artisan command to generate a new queue task:

PHP Artisan make:job Sendreminderemail

The generated class implements the Illuminate\Contracts\Queue\ShouldQueue interface, which means that the task will be pushed into the queue instead of being executed synchronously.

Task class Structure

The structure of a task class is simple and generally contains only one method that the queue uses to invoke this task handle . Let's look at an example of the task class, in this example, let's say we manage a podcast publishing service that needs to process the propagated guest files before publishing:

<?phpnamespace app\jobs;use app\podcast;use App\AudioProcessor;use Illuminate\bus\queueable;use Illuminate\queue\serializesmodels;use Illuminate\queue\interactswithqueue;use Illuminate\contracts\queue\shouldqueue;class Processpodcast implements shouldqueue{use Interactswithqueue,    Queueable, Serializesmodels;    protected $podcast;     /** * Create a new task instance. * * @param podcast $podcast * @return void */Public Function __construct (podcast $podcast) {$    This->podcast = $podcast;     }/** * Run the task.    * * @param audioprocessor $processor * @return void */Public function handle (audioprocessor $processor) {//Process uploaded podcast ...}} 

Note that in this example, we pass a eloquent model directly in the constructor of the task class. Because we quoted this in the task class. SerializesModels
, so that the eloquent model can be gracefully serialized and deserialized when it comes to processing tasks. If your queue task class receives a eloquent model in the constructor, only the attributes that recognize the model are serialized into the queue. When the task is actually running, the queue system will automatically retrieve the complete model from the database. This whole process is completely transparent to your application, which avoids some of the problems of serializing a complete eloquent pattern instance.

The method is called when the queue processes the task, handle and here we can also let the Laravel service container automatically inject the dependent object through the handle method's parameter type hint.

{Note} This binary data, like picture content, must be converted using a method before it is put into a queue task base64_encode . Otherwise, when this task is placed in the queue, it may not be serialized correctly as JSON.

Distributing tasks

Once you've written the task class, you'll be able dispatch to distribute it through auxiliary functions. The only argument that needs to dispatch be passed is an instance of this task class:

<?phpnamespace app\http\controllers;use app\jobs\processpodcast;use illuminate\http\request;use App\Http\ Controllers\controller;class Podcastcontroller extends controller{    /**     * Save podcasts.     *     * @param  request  $request     * @return Response     *    /Public function store (request $ Request)    {        //create podcast        ... Dispatch (New Processpodcast ($podcast));}    }

{Tip} dispatch Provides a simple, globally available function that is also very easy to test. Check out the Laravel test documentation below to learn more.

Deferred distribution

If you want to delay the execution of a task in a queue, you can use the method of the task instance delay . This method is Illuminate\Bus\Queueable provided by trait, and this trait is loaded by default in all auto-generated task classes. For deferred tasks we can give an example, for example, to specify a task to be performed after 10 minutes of distribution:

<?phpnamespace app\http\controllers;use carbon\carbon;use app\jobs\processpodcast;use Illuminate\Http\Request; Use App\http\controllers\controller;class Podcastcontroller extends controller{    /**     * Save a new podcast.     *     * @param  request  $request     * @return Response     *    /Public function store (request $ Request)    {        //create podcast        ... $job = (new Processpodcast ($podcast))                    ->delay (Carbon::now ()->addminutes ());        Dispatch ($job);    }}

{Note} The Amazon SQS Queue service has a maximum delay of 15 minutes.

customizing Queues & Connections

Distributing tasks to a specified queue

By pushing tasks to different queues, you can categorize queue tasks and even control how many tasks are assigned to different queues. Remember, this is not about pushing tasks into different "connections" in the queue configuration file, but instead pushing them to a different queue in a connection. To specify a queue, you invoke the method of the task instance onQueue :

<?phpnamespace app\http\controllers;use app\jobs\processpodcast;use illuminate\http\request;use App\Http\ Controllers\controller;class Podcastcontroller extends controller{    /**     * Save a new podcast.     *     * @param  request  $request     * @return Response     *    /Public function store (request $ Request)    {        //create podcast        ... $job = (new Processpodcast ($podcast))->onqueue (' processing ');        Dispatch ($job);    }}

Distributing tasks to a specified connection

If you use multiple queue connections, you can push the task to the specified connection. To specify a connection, you can invoke the method of the task instance onConnection :

<?phpnamespace app\http\controllers;use app\jobs\processpodcast;use illuminate\http\request;use App\Http\ Controllers\controller;class Podcastcontroller extends controller{    /**     * Save a new podcast.     *     * @param  request  $request     * @return Response     *    /Public function store (request $ Request)    {        //create podcast        ... $job = (new Processpodcast ($podcast))->onconnection (' Sqs ');        Dispatch ($job);    }}

Of course, you can chain-call onConnection and onQueue to specify both the connection and the queue for the task:

$job = (new Processpodcast ($podcast))                ->onconnection (' Sqs ')                ->onqueue (' processing ');

Specify the maximum number of retries/timeout values for a task

Maximum number of attempts

Specify the maximum number of attempts in a task that you can try to set by using the Artisan command line --tries :

PHP Artisan queue:work--tries=3

However, you can take a more refined approach to doing this, such as defining the maximum number of attempts in a task class. If the maximum number of attempts is defined in both the class and the command line, Laravel will take precedence over the values in the task class:

<?phpnamespace App\jobs;class Processpodcast implements shouldqueue{    /**     * Task Max Retries     *     * @var int     */Public    $tries = 5;}

Timeout

Similarly, the maximum number of seconds a task can run can be specified using the switch on the Artisan command line --timeout :

PHP Artisan queue:work--timeout=30

However, you can also define a variable in the task class to set the maximum description that can be run, and if the maximum number of attempts is defined on both the class and the command line, Laravel will take precedence over the values in the task class:

<?phpnamespace App\jobs;class Processpodcast implements shouldqueue{    /**     * Time-out for tasks to run.     *     * @var int */public    $timeout = 120;}

Error handling

If an exception is thrown when the task runs, the task is automatically released back into the queue so that it can be re-run again. If you continue to throw an exception, the task will continue to be freed back to the queue until the number of retries reaches the maximum number of times your app allows. The maximum number of times is queue:work defined by parameters when invoking the Artisan command --tries . More information on the queue processor can be seen below.

Running the queue processor

The Laravel contains a queue processor that can handle these tasks when a new task is pushed into the queue. You can run the processor with the Queue:work Artisan command. Note that once the queue:work command starts, it will run until you manually stop or you close the console:

PHP Artisan Queue:work

{tip} to keep queue:work the process running in the background permanently, you should use process monitoring tools, such as Supervisor to ensure that the queue processor does not stop running.

Be sure to remember that the queue processor is a long-running process and holds the application state that has started in memory. The result is that if you modify the code after the processor is running, those changes will not be applied to the processor. So in your redeployment process, be sure to restart the queue processor.

Specify Connections & Queues

You can specify the connection that the queue processor uses. You config/queue.php define multiple connections in a configuration file, and work the connection name you pass to the command is at least consistent with one of them:

PHP Artisan queue:work Redis

You can customize the queue processor by processing a specific queue for a given connection. For example, if all your messages are processed in a redis queue in the connection emails , you can start a queue processor that only processes that particular queue with the following command:

PHP Artisan queue:work Redis--queue=emails

Resource considerations

The daemon queue does not "restart" the framework before each job is processed. Therefore, after each task is completed, you should release any resources that are too large to occupy. For example, if you use the GD library for image processing, you should use the imagedestroy free memory after completion.

Queue priority

Sometimes you want to set the priority of the processing queue. For example config/queue.php , you might have set the redis default queue priority in the connection low , but you might occasionally want to push a task into the high priority queue, like this:

Dispatch (New Job)->onqueue (' High ');

To verify that the tasks in the high queue are low processed before the tasks in the queue, you start a queue processor, pass the list of queue names to it and comma, interval:

PHP Artisan queue:work--queue=high,low

Queue Processor & Deployment

Because the queue processor is a long-lived process, if the code changes and the queue processor does not restart, they cannot apply the new code. So the simplest way is to restart the queue processor during the redeployment process. You can only enter it gracefully queue:restart to restart all queue processors.

PHP Artisan Queue:restart

This command will tell all queue processors to end the process after the current task has been completed so that no task is lost. Because the queue processor is executing the queue:restart command against the end process, you should run a process manager, such as Supervisor from the restart queue processor.

Task Expiration & Timeout

Task expires

config/queue.phpIn the configuration file, each queue connection defines an retry_after option. This option specifies how many seconds the task will be processed and is retried as a failure. For example, if this option is set to 90 , then it will be released back to the queue when the task continues to execute for a 90 second without being deleted. Typically, you should retry_after set the time that corresponds to the longest-consuming task.

{Note} The only connection that does not have an retry_after option is Amazon SQS. When using Amazon SQS, you must configure this retry threshold through the Amazon command line.

Queue Processor Timeout

queue:workThe Artisan command has an external --timeout option. This option specifies Laravel how long the queue processor should be shut down after the maximum execution time. Sometimes a queue's subprocess is zombie for a number of reasons, such as an external HTTP request that is not responding. This --timeout option removes the zombie process that exceeds the specified event limit.

PHP Artisan queue:work--timeout=60

retry_afterConfiguration options and --timeout command-line options are different, but you can work at the same time to ensure that tasks are not lost and do not repeat.

{Note} --timeout Should always be retry_after at least a few seconds longer than the short. This ensures that the task process can always be killed before the failure is retried. If your --timeout option is greater than the retry_after configuration option, your task may be executed two times.

Queue Process Sleep Time

When a queue needs to process a task, the process continues to process the task with no delay between them. However, if no new work is available, the sleep parameters determine how long the worker process will "sleep":

PHP Artisan queue:work--sleep=3

Supervisor Configuration

Installing Supervisor

Supervisor is a process monitoring software on a Linux operating system that automatically restarts queue:listen queue:work after or after a command failure. To install Supervisor in Ubuntu, you can use the following command:

sudo apt-get install Supervisor

{Tip} If you manually configure Supervisor to sound a little difficult to cope with, consider using Laravel Forge, which can automatically install and configure Supervisor for your Laravel project.

Configure Supervisor

Supervisor profiles are typically placed in a /etc/supervisor/conf.d directory where you can create any number of profiles that require Supervisor to monitor your process. For example we create one laravel-worker.conf to start and monitor a queue:work process:

[program:laravel-worker]process_name=% (Program_name) s_% (process_num) 02dcommand=php/home/forge/app.com/artisan Queue:work SQS--sleep=3--tries=3autostart=trueautorestart=trueuser=forgenumprocs=8redirect_stderr=truestdout_ Logfile=/home/forge/app.com/worker.log

The commands in this example numprocs require Supervisor to run and monitor 8 queue:work processes, and restart after they fail to run. Of course, you have to change the command command queue:work sqs to show the queue driver of your choice.

Start Supervisor

When this configuration file is created, you need to update the Supervisor configuration and start the process with the following command:

sudo supervisorctl rereadsudo supervisorctl updatesudo supervisorctl start laravel-worker:*

For more information on setting up and using Supervisor, please refer to the Supervisor official documentation.

Handle a failed task

Sometimes the tasks in your queue will fail. Don't worry, things will not go smoothly. The Laravel has a convenient way to specify the maximum number of times a task retries. When a task exceeds this retry count, it is inserted into the failed_jobs data table. To create failed_jobs a table, you can use the queue:failed-table command:

PHP Artisan queue:failed-tablephp Artisan Migrate

Then run the queue processor, and you should --tries specify the maximum number of retries for the task by parameter when calling the Queue:work command. If not specified, the task will be retried permanently:

PHP Artisan queue:work Redis--tries=3

Purge failed tasks

You can define the method directly in the task class failed , which can run the purge logic of the task when the task fails. This place is great for sending a warning to a user or resetting a task to perform. The exception information that causes the task to fail is passed to the failed method:

<?phpnamespace app\jobs;use exception;use app\podcast;use app\audioprocessor;use Illuminate\Bus\Queueable;use Illuminate\queue\serializesmodels;use Illuminate\queue\interactswithqueue;use Illuminate\Contracts\Queue\ Shouldqueue;class Processpodcast implements shouldqueue{use    interactswithqueue, queueable, serializesmodels;    protected $podcast;    /**     * Create a new task instance.     *     * @param  podcast  $podcast     * @return void     *    /Public Function __construct (podcast $ Podcast)    {        $this->podcast = $podcast;    }    /**     * Perform tasks.     *     * @param  audioprocessor  $processor     * @return void     *    /Public Function handle ( Audioprocessor $processor)    {        //processing on the spread of the guest ...    }    /**     * Failed task to process.     *     * @param  Exception  $exception     * @return void     *    /Public Function failed ( Exception $exception)    {        //Send failure notification to user, etc..    }}

Task Failure Events

If you want to register an event that will be called when a queue task fails, you can use the Queue::failing method. This will give you the opportunity to notify your team by e-mail or hipchat through this event. For example, we can AppServiceProvider attach a callback function to this event in the Laravel built-in:

<?phpnamespace App\providers;use Illuminate\support\facades\queue;use Illuminate\queue\events\jobfailed;use Illuminate\support\serviceprovider;class Appserviceprovider extends serviceprovider{    /**     * Start Services for any application.     *     * @return void */public    function boot ()    {        queue::failing (jobfailed $event) {            //$event->connectionname            //$event->job            //$event->exception        });    }    /**     * Registered service provider.     *     * @return void     *    /Public Function register ()    {        //    }}

Retry failed Task

To see failed_jobs all of your failed tasks in the datasheet, you can use queue:failed this Artisan command:

PHP Artisan queue:failed

queue:failedThe command lists the ID, connection, queue, and failure time of all tasks, and the task ID can be used to retry the failed task. For example, to retry a 5 failed task with an ID, the command is as follows:

PHP Artisan queue:retry 5

To retry all failed tasks, you can use queue:retry and use all as ID:

PHP Artisan Queue:retry All

If you want to remove a failed task, you can use the queue:forget command:

PHP Artisan queue:forget 5

queue:flushThe command allows you to remove all failed tasks:

PHP Artisan Queue:flush

Task events

Using the queue before and after methods, you can specify the callback processing before and after the task is processed. In these callbacks, it is a good time to implement additional logging or to increase the statistical data. Typically, you should call these methods in the service container. For example, we use the Appserviceprovider in Laravel:

<?phpnamespace App\providers;use Illuminate\support\facades\queue;use Illuminate\support\serviceprovider;use Illuminate\queue\events\jobprocessed;use Illuminate\queue\events\jobprocessing;class AppServiceProvider extends serviceprovider{    /**     * Start any service.     *     * @return void */public    function boot ()    {        queue::before (jobprocessing $event) {            //$event->connectionname            //$event->job            //$event->job->payload ()        });        Queue::after (function (jobprocessed $event) {            //$event->connectionname            //$event->job            //$ Event->job->payload ()        });    }    /**     * Registered service provider.     *     * @return void     *    /Public Function register ()    {        //    }}

队列using the method in facade looping , you can try to execute the specified callback method before the queue gets the task. For example, you can use a closure to roll back a transaction that has failed a task before.

queue::looping (function () {while (Db::transactionlevel () > 0) {db::r    Ollback (); }});