Source code analysis: Laravel queue

Source code analysis: Laravel queue

Preface

Laravel's queue service provides unified APIs for various background queue services. The queue allows you to delay the execution of time-consuming tasks, such as sending an email. This effectively reduces the request response time.

Problems Found

In Laravel, Redis is used to process queue tasks. The framework provides powerful functions. However, a problem occurs recently, that is, it is found that a task is executed multiple times. Why?

Reasons:

In Laravel, if the execution time of a queue (task) is greater than 60 seconds, the execution will be considered failed and re-added to the queue. This will lead to repeated execution of the same task.

The logic of this task is to push the content to the user. the user needs to be retrieved and traversed Based on the queue content, and sent by requesting the backend HTTP interface. For example, if there are 10000 users, the execution time will certainly be greater than 60 seconds if there are too many users or the interface processing speed is not so fast, so this task will be re-added to the queue. Even worse, if the previous tasks are not completed in 60 seconds, they will be re-added to the queue, so that the same task will be executed more than once, but multiple times.

Find the culprit from the Laravel source code.

Source code file: vendor/laravel/framework/src/Illuminate/Queue/RedisQueue. php

/** * The expiration time of a job. * * @var int|null */protected $expire = 60;

The $ expire member variable is a fixed value. Laravel considers that a queue should be executed in 60 seconds. Queue fetch method:

public function pop($queue = null){ $original = $queue ?: $this->default;  $queue = $this->getQueue($queue);  $this->migrateExpiredJobs($queue.':delayed', $queue);  if (! is_null($this->expire)) {  $this->migrateExpiredJobs($queue.':reserved', $queue); }  list($job, $reserved) = $this->getConnection()->eval(  LuaScripts::pop(), 2, $queue, $queue.':reserved', $this->getTime() + $this->expire );  if ($reserved) {  return new RedisJob($this->container, $this, $job, $reserved, $original); }}

There are several steps to get the queue, because the queue fails to be executed, or the execution time-out will be saved in another set for retry. The process is as follows:

1. Re-rpush the failed queue from the delayed set to the currently executed queue.

2. Re-rpush the queue due to execution timeout from the reserved set to the currently executed queue.

3. Then, the task is retrieved from the queue and executed. At the same time, the queue is put into the sorted set of reserved.

The eval command is used to execute this process and several lua scripts are used.

Retrieve tasks from the queue to be executed:

local job = redis.call('lpop', KEYS[1])local reserved = falseif(job ~= false) then reserved = cjson.decode(job) reserved['attempts'] = reserved['attempts'] + 1 reserved = cjson.encode(reserved) redis.call('zadd', KEYS[2], ARGV[1], reserved)endreturn {job, reserved}

It can be seen that Laravel puts a copy of the queue to be executed by Redis in an ordered collection and uses the expiration timestamp as the score.

Only after the task is completed will the task be removed from the sorted set. The code for removing a queue from this sorted set is omitted. Let's take a look at how Laravel processes a queue whose execution time is greater than 60 seconds.

That is, the operation performed by this lua script:

local val = redis.call('zrangebyscore', KEYS[1], '-inf', ARGV[1])if(next(val) ~= nil) then redis.call('zremrangebyrank', KEYS[1], 0, #val - 1) for i = 1, #val, 100 do  redis.call('rpush', KEYS[2], unpack(val, i, math.min(i+99, #val))) endendreturn true

Here, zrangebyscore finds the element whose score ranges from infinitely small to the current timestamp, that is, the task that was added to the set 60 seconds ago, and then removes these elements from the set through zremrangebyrank and rpush to the queue.

We should have suddenly realized this.

If a queue is not completed in 60 seconds, the process re-rpws these tasks from the reserved set to the queue when taking the queue.

Summary

The above is all the content of this article. I hope the content of this article has some reference and learning value for everyone's learning or work. If you have any questions, please leave a message to us, thank you for your support.