Implementation and application of PHP message queue

Well-known in the design of the site, will encounter to the user "mass SMS", "Order system has a large number of logs", "Seconds to kill design" and so on, the server can not handle this instantaneous burst of pressure, this situation to ensure the normal and effective use of the system, you need "Message Queuing" help. This article is mainly through the idea of Message Queuing to learn.

The following knowledge is the main understanding:

1. What is a queue, what can he do?

2. What are the application scenarios for columns?

3. How do I use a queue to decouple my business?

4, how to use Redis queue to eliminate high pressure?

5, professional RABBITMQ How to use the system?

Summarize the following main contents

@ Message Queuing concepts, principles and scenarios

@ Decoupling case: Queue processing order system and distribution system

@ Flow peaking case: Redis's list type for second kill

@RabbitMQ: A more professional messaging system implementation scenario

I. Understanding Message Queuing

1.1 Message-to-column concepts

In essence, the message-to-column is a queue structure of the middleware, that is, the message placed in the middleware can be returned directly, do not need the system to process immediately, but also a program to read the data, and sequentially processed successively.

This means that when you encounter a concurrency that is particularly large and takes a long time to return to the processing results, using Message Queuing can solve such problems.

1.2 Core structure

Queued by a business system, the message is inserted into the message queue one after another, after the success of the successful insert, followed by a message processing system, the system will be in the message system to take the records of successive extraction and processing, complete a team out of the process.

1.3 Application Scenarios

Data redundancy: such as the order system, the subsequent need for strict data conversion and recording, Message Queuing can be persisted in the storage of the data in the queue, and then there are orders, the subsequent processing procedures to obtain, after the subsequent processing after the deletion of this record to ensure that each record can be processed to complete.

System decoupling: After using the message system, the queue system and the system is separate, it is said that as long as the day crashes, will not affect the normal operation of another system.

Traffic clipping: For example, seconds and snapping, we can use the cache with the message queue, can effectively withstand the instantaneous traffic, to prevent the server cannot withstand the crash.

Asynchronous communication: The message itself can be returned directly after it is queued.

Extensibility: For example, the order queue, not only can process orders, but also to other business use.

Sort Guarantee: Some scenarios need to be processed according to the order of the products such as single-input and single-out to ensure that the data in a certain order processing, using Message Queuing is possible.

These are common usage scenarios for Message Queuing, and of course Message Queuing is just a middleware that can be used with other products.

1.4 Common queue implementations pros and cons

Queue Media

1, database, such as MySQL (high reliability, easy to implement, slow speed)

2. Caching, such as Redis (fast, low efficiency when a single message packet is too large)

3, message system, such as RABBITMQ (strong professional, reliable, high learning costs)

Message processing trigger mechanism

1, Dead Cycle mode read: Easy to achieve, failure can not be restored in time; (more suitable for the second kill, relatively centralized, operation and maintenance centralized)

2, timed tasks: Pressure sharing, there is a processing limit, the current more popular processing trigger mechanism. (The only drawback is that the interval and data need to be noted, don't wait for the previous task to finish the next task and start again)

3. Daemon: Similar to PHP-FPM and PHP-CG, requires Shell Foundation

Second, the decoupling case: queue processing " Order system " and " Distribution system "

Simply say the program decoupling: program decoupling is to avoid your wife and your mother at the same time fall into the water first to save who the problem (steal laugh ing)

For the order process, we can design two systems, one is "order system" another is "distribution system", in the online shopping when we should have seen, when I submitted an order, I can see in the background my goods are in delivery. This is the time to get involved in a "distribution system".

If we are in the framework of the "Order System" and "distribution system" design together, there will be some problems, first of all for the order system, because the system pressure will be relatively large, but "distribution system" does not need to do some immediate response to these pressures.

The second we also do not want to cause the delivery system failure after the order system fails, this time will affect the normal operation of the two systems. So we want to decouple the two systems. After the two systems are separated, we can communicate these two systems through an intermediate "queue table".

2.1 Architecture Design

1, the first order system will receive the user's orders, and then the processing of orders.

2. The order information is then written to the queue table, which is the key to communicating these two systems.

3, by the distribution system timed execution of a program to read the queue table for processing.

4, after the distribution system processing, the processed records will be marked.

2.2 Program Flow

Third, the flow peak case: Redis list type to achieve the second kill

Redis is based on memory, it's very fast, and Redis has a very good complement to the database because it is durable, Redis periodically writes the data to the hard drive, so it doesn't have to worry about power outages, which in turn is more advantageous than the other cache memcache, and Redis provides five data types (string, doubly linked list, hash, set, ordered set)

In general, do the second kill case, snapping, instant high than you, need to queue the case of Redis is a good choice.

3.1 The list type in the Redis data type

The Redis list is a doubly linked list that can append data from the head or tail.

* LPUSH/LPUSHX: Inserts a value into the (/existing) list header

* RPUSH/RPUSHX: Inserts a value into the (/existing) list trailer

* Lpop: Remove and get the first element of a list

* Rpop: Remove and get the last element of the list

* LTRIM: Preserves elements within a specified interval

* Llen: Get list length

* LSET: Sets the value of the list element by index

* LINDEX: Get the elements in the list by index

* Lrange: Gets the element within the specified range of the list

3.2 Architecture Design

A simple structure of the second-kill program.

1, the first record is which user participated in the second kill simultaneously recorded his time.

2. Save the user's ID to the Redis list and let it queue up. If only the first 10 users can participate in the success, if the number in the list is enough, it will not let it continue to append data. In this way, the list length of Redis will only be 10

3, finally in the Redis data is slowly written to the database, to reduce the pressure of data

3.3 Code-level design

1. When the user starts the second kill, writes the request of the second Kill program to Redis (UID, time_stamp).

2, if only 10 people can be killed successfully, check Redis has stored the length of the data, exceeding the upper limit directly discard the completion of the second kill instructions.

3, finally in the death cycle processing into the Redis 10 data, and then slowly fetch the data and deposited into the MySQL database.

Killing this piece in seconds is particularly stressful for the database, and if we don't have this design, it can cause MySQL to write to the bottleneck. We use a list of Redis, and then the second kill request into the Redis inside, finally through the storage program, the data slowly written to the database, so that the flow can be achieved, the balance of the MySQL will not cause too much pressure.

Iv. RabbitMQ

Here to explain some of the use of RABBITMQ, first of all, we talked about the second kill case of the lock mechanism, to prevent other programs to process the same record, if our system architecture is very complex, there are many programs to read a queue in real-time, or I have multiple senders, at the same time to operate one or more queues, Even I think that these programs are distributed on different machines, in which case the Redis queue is a bit of a struggle. What to do at this time, we need to introduce some more professional Message Queuing system, these systems can be better to solve the problem.

Architecture and principles of 4.1 rabbitmq

　　

Features: Full implementation of AMQP, cluster simplification, persistence, cross-platform

Rabbitmqs use

1, RABBITMQ installation (Rabbitmq-server, php-amqplib)

2. The producer sends messages to the message channel

3, the consumer processing messages

Work queue

　　　　

Thought: Sent by the producer to the message system, the message system encapsulates the task into a message queue and then uses the same queue for multiple consumers

This not only solves the decoupling between the producer and the consumer, but also realizes the sharing of the consumer and the task, and slows down the pressure of the server.

V. Summary

The above main learning message queue concepts, principles, scenarios. Decoupling cases and peak shaving cases, as well as a simple way to understand RABBITMQ.

Vi. issues

What is the biggest difference between Redis and messaging server selection?

My understanding is that Redis is a processing request, Redis is a single thread, and the message server IO is implemented differently, one is synchronous one is asynchronous, and Redis uses synchronous blocking, and the messaging server uses asynchronous non-blocking.

Advantages and disadvantages of common queue implementations

Queue media:

Mysql: High reliability, easy implementation, slow speed
Redis: Fast, low efficiency with a single big message packet
Message system: Professional, reliable, high learning costs (such as: RABBTIMQ)

Trigger mechanism for message processing:

Dead loop mode read: Easy to implement, failure can not be timely recovery;
Timed tasks: Pressure sharing, with a maximum amount of processing. (Biggest flaw: Locating task time interval and processing the data need to grasp accurately, not the last task has not finished processing, the next think has been started)

Daemon: Similar to PHP-FPM and php-cgi, requires shell knowledge