Queue Application Scenario, self-implemented Queue (1)

I. Queue Application Scenario: Why do teams need to be used?

In web development, we often encounter the need to process Batch Tasks. These Batch Tasks may be submitted by users, it may also be that batch processing is required when the system is triggered by an event. In the face of such a task, if it is a batch task submitted by the user, the junior programmer can only ask the user to trigger the submission action, when the server completes processing and returns the result to the browser, the user cannot close the browser window. If the data is large or the processing speed is slow, the user experience will be directly affected. However, when we use an email address of a certain message or a certain wave, after clicking send mail in a group, we only need to wait for a short time. When the browser prompts that the message is submitted successfully and is being sent, users can turn off the browser. Later, the mailbox in the address bar will receive emails from this group one after another, such as sending scheduled emails in a group, and when customers place orders, customers, customer service, warehouse and other related personnel receive the order email information. In this case, the application scope of queues is so wide.

 

2. General engineer's solution and architect's Solution

Solution 1: Create a table to store emails and messages, and use a timer program to retrieve and send messages.

 

Solution 2: abstract to a higher layer and develop a set of general asynchronous processing queues suitable for any complicated business logic

As an architect, using the queue method, the abstraction layer and the business layer are separated to ensure good scalability and maintainability. In comparison, it is much better. Next we will introduce the Implementation ideas and methods of custom queues.

 

Iii. Overall queue Design

 

1: A queue program is required to provide interfaces for adding and retrieving queues.

2: You need to store queues, files, or databases.

3: The scheduled program needs to retrieve and execute the queue.

4. Other extended functions: priority, log, timing, etc.

 

The directory structure of the Code is as follows. The functions of each file are marked with // annotations.

| -- Addtask. php // example of adding a task to a queue

| -- Cronmission. PHP // scheduled task scheduling program. For example, in a file directly called by crontab in Linux, business logic engineers can flexibly define their own queue tasks in this file, therefore, you do not need to modify crontab on the server for each queue task, which greatly improves security and convenience.

| -- DB. php // database operations

| -- DB. SQL // basic table structure required to create a queue

| -- Doqueue. php // executes the queue task

| -- Queue. Class. php // core queue business defined here, including adding tasks to the queue, reading the queue, and changing the queue task status

| -- Sendmsg. PHP // queue implements the business interface of a specific task, such as the message sending interface of the existing system. In this example, the queue program is integrated with the existing system and written logs are used for demonstration.

 

Iv. Queue Implementation 1: create a task storage table

1:

First, let's look at the most basic:

CREATE TABLE `queue` (id int(11) NOTNULL auto_increment primary key,taskphpvarchar(128) NOT NULL default ‘‘,param text notnull default ‘‘,status tinyintnot null default 0,ctime timestampNOT NULL default CURRENT_TIMESTAMP,KEY (ctime)) ENGINE=InnoDB DEFAULT CHARSET=utf8;

 

Field explanation:

Taskphp: interface file for processing services

Param: parameters to be received in the interface file for processing services

Status: the task processing status. If the value 0 is not processed, the status is changed to 1 after processing.

 

5. Queue implementation 2: Define the call interface

<? PHP/** Youcai network example code * task queue implementation * @ author Youcai network full stack engineer Teaching and Research Group Zhang Youlin @ see http://www.ucai.cn */include_once ('db. PHP '); Class queue {/** throw the task to the queue ** @ Param string $ taskphp program for executing the task * @ Param string $ Param parameter used by the Program for executing the task, for example, send Group messages to the queue: $ arr = array ("uid" => 4, // uid of the sender "UIDs" => array (6,234, 2355 ), // uid "content" => 'xxxxx', // information content); $ cqueue = new Queue (); $ cqueue-> Add ("/APP/send_msg.php", serialize ($ ARR); **/Public functio Nadd ($ taskphp, $ PARAM) {$ taskphp = mysql_real_escape_string ($ taskphp); // $ Param = mysql_real_escape_string ($ PARAM); $ Param = $ Param; $ SQL = "insert into queue (taskphp, Param) values ('". $ taskphp. "','". $ Param. "')"; $ Re = execute ($ SQL); if ($ re) {$ pid = mysql_insert_id (); return $ PID;} else {return false ;}} /** read task queue ** @ Param string $ how many limit records can be retrieved at a time */Public functiongetqueuetask ($ Limit = 1000) {$ Limit = (INT) $ limit; $ SQL = "Select ID, taskphp, Param from queue where status = 0 order by id asc"; $ Re = query ($ SQL); return $ re ;} /** update the task status ** @ Param string $ how many items can be retrieved by limit at a time */Public functionupdatetaskbyid ($ id) {$ id = (INT) $ ID; $ mtime = Time (); $ SQL = "Update queue set status = 1, mtime = ". $ mtime. "Where id = ". $ ID; $ Re = execute ($ SQL); return $ re;} public staticfunction a2s ($ ARR) {$ STR = ""; foreach ($ arr as $ key => $ value) {If (is_array ($ Val UE) {foreach ($ valueas $ value2) {$ Str. = urlencode ($ key ). "[] = ". urlencode ($ value2 ). "&" ;}} else {$ Str. = urlencode ($ key ). "= ". urlencode ($ value ). "&" ;}} return $ STR;} public staticfunction s2a ($ Str) {$ arr = array (); parse_str ($ STR, $ ARR ); return $ arr ;}}?> $ Cqueue = new Queue ();

 

1. Queue addition Interface

// $ Param1 is the program that executes the task. $ param2 is the program parameter and can be serialized data.

$ Cqueue-> Add ($ param1, $ param2 );

 

2. Read queue Interface

$ Tasks = $ cqueue-> getqueuetask ($ Limit = 1000 );

 

3. Update task status

$ Cqueue-> updatetaskstatus ($ id );

 

4. a2s is a custom String Conversion Method for arrays. Here, json_encode is not used, which is prone to problems. Similarly, s2a is used to retrieve and convert data from the database to an array.

$ Re = $ cqueue-> Add ("sendmsg. php", queue: a2s ($ ARR ));

6. Queue Implementation 3: Write an execution queue Program

According to the design, the program file for executing the queue is do_queue.php. Its main function is to extract tasks from the queue table and execute them in the background.

 

Do_queue.php code: $ phpcmd = exec ("which PHP"); // find the PHP installation location $ cqueue = new Queue (); $ tasks = $ cqueue-> getqueuetask (200); foreach ($ tasks as $ t) {$ taskphp = $ t ['taskphp']; $ Param = $ t ['param']; $ job = $ phpcmd. "". escapeshellarg ($ taskphp ). "". escapeshellarg ($ PARAM); system ($ job );}

 

VII. Business implementation of specific tasks

Let's take group sending as an example. We need to write a program for Group Sending. This program receives pre-defined parameters and then calls the message sending interface based on the parameters to send messages.

This is generally implemented by engineers who provide business functions. However, architects have to write document examples in advance to teach others to use them.

 

send_msg.php： $para = $argv[1];$arr = unserialize($para);$cmessage = new Message();foreach($arr[‘uids‘] as $touid){$cmessage->send($arr[‘uid‘],$touid, $arr[‘content‘]);}

 

8. Server deployment 1: Configure crontab

We have written all the programs that run the queue. How can we trigger this program? Of course, we need to use Linux scheduled tasks to execute do_queue.php once every certain time. However, instead of calling do_queue.php directly, we need to add a scheduler cron_mission.php and call do_queue.php in cron_mission.php.

 

Configure the scheduled task crontab:

 

Crontab-e

 

* *** CD/ucai/schedule; PHP cron_mission.php> cron_mission.log

 

# You can first use crontab-L to view the scheduled tasks used by the Local Machine

9. Server deployment 2: Write a scheduled Task Scheduler

Idea: Write the scheduled task to cron_mission.php, which allows you to flexibly control the queue task in cron_mission.php. Compared with controlling doqueue. php directly through crontab, it avoids frequent modification of crontab on the server, which is the best strategy for security and maintenance.

 

Cron_mission.php example:

 

if ($minute % 5 == 0){if(chdir($site_dir."app/")){$cmd = "$phpcmddo_queue.php > do_queue.log &";echo ‘[‘ , $ymd, ‘ ‘ , $hour , ‘:‘ , $minute , ‘] ‘ , $cmd , "\n";system($cmd);}}

 

10. Enable multi-process concurrent execution queue

Train of Thought: Number the task sequence, during database execution

Add ID % To The where condition, total number of tasks to be executed in each queue = queue ID

This avoids repeated processing.

 

For example, if each process executes 10 tasks, the modifications are as follows:

 

1. Modify scheduled tasks

Before modification:

if ($minute % 5 == 0){if(chdir($site_dir."app/")){$cmd ="$phpcmd do_queue.php > do_queue.log &";echo ‘[‘ , $ymd, ‘ ‘ , $hour , ‘:‘ , $minute , ‘] ‘ , $cmd , "\n";system($cmd);}}

 

After modification:

if ($minute % 5 == 0){for ($i=0; $i < 10; $i++) { $cmd ="$phpcmd doQueue.php 10 $i>>doQueueMission".date(‘Y-m-d‘).".log ";echo  date("Y-m-d H:i:s") . "\t :" .$cmd."\n";system($cmd);}}

// Execute 10 processes each time. $ I indicates the current process.

 

2. Queue execution program modification

Before modification:

$phpcmd =‘D:\work\wamp\bin\php\php5.3.10\php ‘;$cqueue = new Queue();$tasks = $cqueue->getQueueTask(200);

After modification:

$phpcmd =‘D:\work\wamp\bin\php\php5.3.10\php ‘;$total=$argv[1];$i=$argb[2];$cqueue = new Queue();$tasks = $cqueue->getQueueTask($total,$i,200);

3. Modify the get queue Interface

Before modification:

public function getQueueTask($limit = 1000){$limit =(int)$limit;$sql ="select id, taskphp, param from queue where status = 0 order by id asc";$re =query($sql);return $re;}

After modification:

public function getQueueTask($total,$i,$limit = 1000){$limit =(int)$limit;$sql ="select id, taskphp, param from queue where status = 0 and id%$total=$i orderby id asc";$re =query($sql);return $re;}

4. Pay attention to server pressure

The number of processes depends on the server pressure.

11. task priority

1. Add priority field to the task storage table

Add a priority field to the data table to distinguish the priority based on the value of the field value.

2. Modify the interface for retrieving queue tasks by priority

Add order by in SQL statements.

12. Record queue logs

1. Add echo to the key

 

2. Roles of shell scripts> and>

 

Summary:

The queue here is implemented by using the scheduled tasks of the server. For example, crontab in Linux is a program in the Linux system. We can also use it for scheduled execution. sh script, for example, to package the database backup and transmit it to the specified server through FTP. This function does not need to be directly used using PHP scripts. sh script. Here we skillfully combine crontab and PHP script, and use crontab to continuously call a queue scheduling interface cronmission. PHP, and then through cronmission. PHP directly controls when or when a queue task is executed.

Here are a few notes

1. When accessing data in the database, do not directly use json_encode or json_decode, which may cause some unexpected problems. In the code, we define two methods: a2s and s2a, the processing array is converted into a string, and the reading string from the database is converted into an array.

2. When the task volume is large and the server load is not fully utilized, you can use multi-process concurrent processing. When concurrent processing, you must consider how to avoid duplication, here we used to mark the queue tasks and read a batch of tasks that a process needs to process from the database each time, the Database ID and the batch ID are differentiated by the method with the remainder equal to 0 to avoid queues of different batches and repeat the same task. (The specific implementation is shown in Step 10 above)