Can detailed the next, listen to the queue all day, the head hurts, Baidu can not find a good explanation
Reply content:
First, the queue is a data structure that can be implemented with linked lists and arrays, and the queue is characterized by first putting the queue into first-out queues.
However, if you see a topic labeled Redis, guess what the main question to ask is that the message queue (MQ) is now widely used. The message here is not just a simple textual message, it can also be a serialized object. Now the more popular open source Message Queuing system has Beanstalkd,rabbitmq,redis (can be used as a queue system), its core role is to first put the message data through the system interface in order into the queue (staged in memory), when necessary and then in the order in which they are put out for subsequent processing.
Take the example of a mail delivery system I've done in the previous period:
The system provides service with HTTP protocol, the main function is to send mail, the API address is/api/msg.send. The user can send a message by passing the three parameters of the subject,body,recipient as long as the address is requested by post.
Scenario 1: Do not use the database, do not use queues
When the interface is requested, the controller directly calls the SendMail or external SMTP server to send the message, the entire sending process HTTP request is waiting for the message to be sent to completion, return the sending result (just call the sender is the result of success, not the message really send results, Really send results need to analyze mail logs)
This scheme is the simplest and most straightforward, usually sending a small amount of mail is possible, but the disadvantage is also obvious:
1. Because the entire message sending process is synchronous during the interface invocation, each request waits for the message sender to complete processing, which inevitably leads to an increase in the waiting time for each invocation of the interface.
2. When the System interface concurrent request is high, the system availability is not only limited by the processing power of webserver, but also is completely limited by the processing ability of the mail sending side software, which can cause the whole system to fail to provide service.
3. If the Mail sender software fails (such as SMTP connection timeout), causing a request to send the message failed, then the message content is completely lost, the system does not have any persistence, can not achieve automatic resend.
Scenario 2: Working with databases, not using queues
When the interface is requested, the message information to be sent is stored in the database with the following table structure:
ID | subject | body | recipient | sent_at | failed_at | failed_times
Then run a scheduled task on the server, read the records of sent_at=0 && Failed_times < 10 every second, then send the message to the sender, and then set the Sent_at to the current time after the success, and the failed_ after the failure. At and accumulate Failed_times. Scenario 2 has already used the idea of a similar queue.
Elevation relative to Scenario 1:
1. Remove the synchronous sending of messages, the interface request response will be much faster
2. e-mail will not be lost after sending failed.
3. The system can also accept the mail sending request after the message sending end is completely invalid, and can continue sending the message after the sending side resumes.
But there are drawbacks:
Each request will be written once the database, when the large concurrency or large data volume (one request contains 1 million recipients), the database load too high impact stability, but also seriously increase the interface response time (write 1 million records at once is not a joke)
Scenario 3: Database + queue
When the interface is requested, the message information to be sent is stored in a JSON format into the queue system, with a single message form such as:
{
" subject": "No Medicine Today",
" Body": "Feel Yourself",
" Recipient": "Mengmeng@da.com "
}
Now the queue is basically a memory queue, data access is very fast, write 1 million data in a flash is no longer difficult.
A resident Process task (Worker) is then run on the server to listen for new messages in the queue in real time (Job, which refers to message information). When a new message comes in, the message is fetched from the queue, the message sender completes processing, the message is destroyed and the message content is inserted into the database (as in Scenario 2), if the send fails, the message is put back into the queue, and a 60-second delay token is added, which means that the processing is removed after 60 seconds.
This improves the throughput and responsiveness of the entire system, and also allows the system to support the ability to run distributed. Each worker process can be treated as a processing node, and if the worker is dispersed on a different server, the distributed processing of the whole system is realized, which is one of the important features of the queue.
In my actual project still do a lot of improvement based on scenario 3, for mass also use mailing list and mail template design, the whole system like Mailgun and Sendcloud design, and so the whole system stabilized, I will consider the code open to GitHub.
This example is just a common usage scenario for queues, and in general you can consider using Message Queuing in scenarios where you need to mitigate the stress of database writes, as well as some scenarios that require distributed processing and are good for queues.
Most languages now have a mature Message Queuing processing component that makes it easy to use a variety of queue systems, such as my usual
Laravel natively supported the Beanstalkd,amazon Sqs,ironmq,redis.
Hold the bricks to lead the jade, please correct me, thank you. Have you heard in line? First come to the station front (team first), followed in the back (team tail), out of the team when the first out of the team, the team at the end, this is the queue! As for implementation, the list and array are all OK!
