PHP (MYSQL/REDIS) Message Queue Introduction and application scenario case--Reprint

Solemn hint: This blog is reproduced from Friends blog, personally feel write very good so without consent forcibly reproduced, the original blog connection http://www.cnblogs.com/wt645631686/p/8243438.html Welcome to visit

In the design of the site, sometimes encountered to send users a large number of text messages, or the order system has a large number of logs need to record, as well as the design of the second kill, the server can not withstand the pressure of this moment, can not be handled normally, how can we ensure that the system normal and effective operation? At this point we need to refer to the message queue to achieve this kind of demand, this time requires an intermediary system to shunt and decompress. Message Queuing is a middleware that needs to be used in conjunction with other reasonable uses.

Concepts, principles, and scenarios for Message Queuing

Essentially, the message queue structure is a queue structure of the middleware
, that is, the message and content into the container, you can directly return, do not need to wait for its post-processing results, there will be a program will read the data, and in order to do one-by-two processing, That is, according to a very large part of the concurrency, at the same time you do not need to immediately get the results of this link, then use Message Queuing can be better to solve the problem. A classic Message Queuing result should be a process in which the
is queued by a business system, inserts the message into the message queue one after another, and then returns the successful result directly after the insert succeeds, and then a message processing system, which takes the records from the message queue and processes them one after the other. Perform the operation of the team.
Message System-appropriate scenario
Redundancy
First data needs redundancy, such as the regular order system, the subsequent need for strict conversion and recording, Message Queuing can be persisted in the queue of data, and then by the Order Processing program to obtain, after the completion of the subsequent processing to delete the record, Ensure that each record can be processed and completed.
Decoupling
Message Queuing separates two sets of systems and solves the problem of deep coupling between two sets of systems. With the use of Message Queuing, the queue system and the system is not directly related to the team, the queue system and the system when one of the system crashes, will not affect the normal operation of another system.
Traffic Clipping
This scenario is the most classic is the second kill snapping, this situation will be a lot of traffic spikes, a lot of demand concentrated in just a few seconds, the instantaneous pressure on the server is very large, and we cooperate with the cache using Message Queuing can be very effective to withstand the instantaneous traffic, to prevent the server to collapse. The
Asynchronous communication
Message itself enables the queued system to return directly, so the asynchronous operation of the program is implemented, so as long as the scenario that is appropriate for Asynchrony can be implemented using Message Queuing.
Extensibility
For example, there may be a financial system to process after the order is queued, but later I want to add a delivery system, I just need to get the distribution system to subscribe to the message queue, so it is easy to expand.
Sort Guarantee
This situation refers to the fact that the order of data processing is very important in some scenarios, which is well suited for queue processing, because the queue itself can be made into single-input single-pass systems, ensuring that the data is processed in order.

Advantages and disadvantages of common Message Queuing implementations
What are the queue media?
Mysql: High reliability, easy to implement, slow, such as the table can be.
Redis: Fast, low efficiency with a single big message packet. Redis provides a list that is suitable for Message Queuing, but Redis has a problem, when the message packet is too large, the efficiency is slow, the general single content is small
Messaging system: Highly professional, reliable, but high learning costs, such as RABBITMQ

Three triggering mechanisms for message processing
Dead loop mode read processing: Let a dead loop program continuously read a queue, and post-processing, this method of failure is relatively strong, because this program constantly scan message queue, so once the message queue has data, it can be processed later. But this will cause the server pressure, the most critical is also will not know when the program will hang, once there is a failure, no way to recover in time, this situation is more suitable for the second kill, because the seconds to kill the time point of comparison concentration, once the second kill can be processed immediately.
Timed tasks: Every few seconds or a few minutes to execute once, the biggest advantage of doing this is to separate the pressure, regardless of the queue system at which point in the queue of the peak is how uneven, but because the system is scheduled to execute, so the pressure will be divided evenly, each time point of pressure is not too much, so it is more popular, In particular, the order system and logistics distribution system, such as the order system will write to the queue, the user can see my order in the logistics distribution, so that the logistics system will be scheduled to summarize the order processing, so the pressure will not be too large, the only drawback is timing and spacing and quantity to grasp, do not
When the last scheduled task is not finished, the next scheduled task starts again, which is prone to unpredictable problems.
Daemon: Similar to the PHP-FPM and php-cgi processes, requires the Shell Foundation of Linux.

Decoupling case: Queue processing order system and distribution system
In the online shopping, after submitting the order, see their orders in the delivery of goods, so participate in a system is the distribution system, if we do the structure, the order system and distribution system design together, there will be problems. First for the order system, the order system processing pressure, for the distribution system does not need to reflect on these pressures, we do not need in order system problems, and distribution system problems, this time will affect the operation of the two systems, we can decouple the solution. After the two systems are separated, we can communicate the two systems through a queue table. First, the order system will receive the user's orders, the processing of orders, the order will be written into the queue table, the queue table is the key to communicate the two systems, the distribution system in the timing of the execution of the program to read the queue table for processing, after the distribution system processing, the processed records will be marked, this is the process.
Specific details are designed as follows (MySQL queue example):

First, the order.php file receives the user's order, then generates the order number and processes the order, and the order system will add the data required by the distribution system to the queue table, and the queue table can be designed so that approximately six fields, ORDER_ID (order primary key ID), status ( Order status), mobile (subscriber phone number), address (harvest address), CREATED_AT (Time of Creation), Updated_at (post-delivery system processing time), and then have a timed script that starts the distribution handler every minute, Distribution Handler This goods.php is used to process the data in the queue table, and when the processing is complete, the field status in the queue table is changed to the processing completion so that the entire process ends.

Create Order Form

CREATE TABLE `order_queue` (
   `id` int (11) unsigned NOT NULL AUTO_INCREMENT COMMENT‘ the id number of the order ’,
   `order_id` int (11) NOT NULL,
   `mobile` varchar (20) NOT NULL COMMENT‘ user ’s mobile number ’,
   `address` varchar (100) NOT NULL COMMENT‘ user ’s address ’,
   `created_at` datetime NOT NULL DEFAULT‘ 0000-00-00 00:00:00 ’COMMENT‘ time of order creation ’,
   `updated_at` datetime NOT NULL DEFAULT‘ 0000-00-00 00:00:00 ’COMMENT‘ time of logistics system processing ’,
   `status` tinyint (2) NOT NULL COMMENT‘ current status, 0 unprocessed, 1 processed, 2 processing ’,
   PRIMARY KEY (`id`)
) ENGINE = InnoDB DEFAULT CHARSET = utf8;

order.php file for processing orders

<? php
include ‘class / db.php’;
  
if (! empty ($ _ GET [‘mobile‘])) {
     $ order_id = rand (10000,99999);
     $ insert_data = array (
         ‘Order_id’ => $ order_id,
         ‘Mobile’ => $ _ GET [‘mobile’], // Remember to filter
         ‘Created_at’ => date (‘Y-m-d H: i: s’, time ()),
         ‘Order_id’ => $ order_id,
         ‘Status’ => 0, // 0, unprocessed status
     );
     $ db = DB :: getIntance ();
// Put the data into the queue list
     $ res = $ db-> insert (‘order_queue’, $ insert_data);
     if ($ res) {
         echo $ insert_data [‘order_id’]. "Saved successfully";
     } else {
         echo "Save failed";
     }
} else {
     echo "1";
}
?>

Shipping system processing Order file goods.php

<? php
// The delivery system processes the order and marks it
include ‘class / db.php’;
$ db = DB :: getIntance ();
// 1: First change the status of the data to be processed to pending
$ waiting = array (‘status‘ => 0,);
$ lock = array (‘status’ => 2,);
$ res_lock = $ db-> update (‘order_queue’, $ lock, $ waiting, 2);
// 2: Select the data that was just updated, and then process it by the distribution system
if ($ res_lock) {
     // Select the order content to be processed
     $ res = $ db-> selectAll (‘order_queue’, $ lock);
      // then handled by the distribution system ...
     // 3: change processed to processed state
     $ success = array (
         ‘Status’ => 1,
         ‘Updated_at’ => date (‘Y-m-d H; i: s’, time ()),
     );
     $ res_last = $ db-> update (‘order_queue’, $ success, $ lock);
     if ($ res_last) {
        echo "Success:". $ res_last;
     } else {
         echo "Fail:". $ res_last;
     }
} else {
     echo "ALL Finished";
}
?>

Timed execution of script goods.sh, executed once per minute

#! / bin / bash
date "+% G-% m-% d% H:% M:% S" // Current year, month, and day
cd / data / wwwroot / default / mq /
php goods.php

The crontab task then executes the script periodically, creates the log file, and then specifies the output format

*/1 * * * */data/wwwroot/default/mq/good.sh >>/data/wwwroot/default/mq/log.log 2>&1//Specify script directory and format output// Of course, to create a log.log file

Tail-f log.log  //monitoring log

So the order system and distribution system are independent of each other and do not affect the normal operation of the other system.

To give an example of MySQL Message Queuing, send a text message as an example:

<? php
$ db = new Db ();
$ sms = new Sms ();
while (true) {
     $ item = $ db-> getFirstRecord (); // Get the first record in the data table
     if (! $ item) {
         // If there is no data in the queue, end the timer
         break;
     }
     $ res = $ sms-> send ($ item [‘phone’], $ item [‘content‘]); // Send SMS
     if ($ res) {
         $ db-> deleteFristRecord (); // Delete successfully sent records
         echo $ item [‘phone’]. ‘Send successfully’;
     } else {
         echo $ item [‘phone’]. ‘Send failed, continue to try later’;
     }
     sleep (10); // loop every ten seconds
}
 
echo ‘Send it! ‘;
?>

Save the code as timer_sms.php, open the command line, and execute the timer:

The PHP timer will automatically complete the task of sending text messages based on the set interval (10 seconds). When the task is completed, the timer is automatically exited and server resources are no longer occupied.

According to my test, the PHP timer takes up a lot of resources and does not cause any stress to the server. It also accesses the database asynchronously and does not affect the operation of the database.

The advantages of this approach are:

1, the backstage operation, the front desk does not need to wait

2, the success rate is high, the record of failure will automatically re-send, until successful

Flow peaking case: Redis's list type for second kill

Why use Redis instead of MySQL? Because Redis is based on memory, the speed is much faster, and MySQL needs to write to the hard disk, because the other business to use MySQL, if the second kill using MySQL, will be the MySQL resources to consume light, Such other business in reading MySQL must be problematic. In addition, Redis has a persistent effect on the data, which is better than memcache, and the data type is more, this time to use the Redis list, you can head or tail to the Redis linked list to add elements, so redis in the implementation of a lightweight queue is very advantageous.
Lpush/lpushx:lpush is to insert the value into the head of the list, LPUSHX is to detect whether the list exists, if it exists, it will be inserted into the head, if not present will ignore this data
RPUSH/RPUSHX: Inserts a value into the tail of the list. Ibid., opposite position
Lpop: Removes and obtains the first element in a list.
Rpop: Removes and obtains the last element in the list.
LTRIM: Retains elements within the specified interval.
Llen: Gets the length of the linked list.
LSET: Sets the value of the linked list element with an index.
LINDEX: Gets the elements in the linked list by index.
Lrange: Gets the elements within the specified range of the list.

"Seconds to kill business program" record which user participated in the second kill, while recording time, so that the convenience of subsequent processing, the user's ID will be stored in the "Redis" list to queue, such as the intention to let the first 10 seconds to kill the success, the latter of the second kill failed, so that the length of the Redis linked list is maintained to 10 10 later if you request append data to Redis, then the program rejects the request, after Redis access, the subsequent program will value Redis, because the data can not be placed in the cache for a long time, followed by a program to traverse the value of Redis, put into the database for permanent storage, Because the second kill is not too long, you can use the script to cycle through the scan.
Detailed Description:
First, the Redis program will put the user's request data into Redis, mainly the UID and microsecond timestamp, and then check the length of the Redis linked list, discard processing beyond the length, the dead loop data read the contents of the Redis linked list, storage.

Database code:

 CREATE TABLE `redis_queue` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `uid` int(11) NOT NULL DEFAULT ‘0‘,
  `time_stamp` varchar(24) NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=88 DEFAULT CHARSET=utf8;

The other two programs, one is to receive the user's request to write to Redis program, and the other is to take redis data to write to the database program.

The program that receives the user request:

<? php

$ redis = new Redis ();
$ redis-> connect (‘127.0.0.1’, 6379);
$ redis-_name = ‘miaosha’;


// Seckill users flood into simulation, 100 users
for ($ i = 0; $ i <100; $ i ++) {
     $ uid = rand (1000000,99999999);
}
// Check the length of the redis linked list (the existing number)
$ num = 10;
if ($ redis-> lLen ($ redis_name) <10) {
     // Join the end of the linked list
     $ redis-> rPush ($ redis_name, $ uid. ‘%‘. microtime ());
} else {// if it reaches 10
     // End of spike
}
$ redis-> close ();

Handlers (Redis data written to MySQL)

<? php
// Read a value from the head of the queue to determine whether this value exists. If it exists, the time and uid are cut out and saved to the database. (For redis, if this value is taken from redis, then this value is not in the redis queue. If a problem fails, then we need a mechanism to put the failed data back into the redis linked list)
$ redis = new Redis ();
$ redis-> connect (‘127.0.0.1’, 6379);
$ redis-_name = ‘miaosha’;

// Endless loop detection redis queue
while (1) {
    $ user = $ redis-> lpop ($ redis_name);
    if (! $ user || $ user == ‘null’) {// If no data exits the loop
        // If it is always executed and the speed is very fast, then the server is under high pressure, here once every 2 seconds
        sleep (2);
        // out of the loop
        continue;
    }
    // Take out the microsecond timestamp and uid
    $ user_arr = explode (‘%‘, $ user);
    $ insert_data = array (
        ‘Uid’ => $ user_arr [0];
        ‘Time_stamp’ => $ user_arr [1];
    );
    $ res = $ db-> insert (‘redis_queue’, $ insert_data);
    // if insert fails
    if (! $ res) {
        // From which direction to take out and from which direction to insert back
        $ redis-> lpush ($ redis_name, $ user);
        sleep (2);
    }
}
$ redis-> close (); 

Test, you can perform a loop detection script, and then execute the second kill script to start testing, monitoring MySQL database changes.

RabbitMQ: A more professional message system implementation scenario

Can see my blog current column to understand

Http://www.cnblogs.com/wt645631686/category/1171220.html

PHP (MYSQL/REDIS) Message Queue Introduction and application scenario case--Reprint