Large Web site Architecture series: Distributed Message Queuing (i) (RPM)

Large Web site Architecture series: Distributed Message Queuing (i)

The following is an outline of Message Queuing, which focuses on Message Queuing overview, Message Queuing scenarios, and message middleware examples (e-commerce, log system).

This sharing outline

1. Message Queuing Overview
2. Message Queuing scenarios
3. Message Middleware Sample
4. JMS Messaging Service (see article Two: Large Web site Architecture series: Distributed Message Queuing (ii))
5. Common Message Queue (see Second: Large Web site Architecture series: Distributed Message Queuing (ii))
6. Reference (recommended) information (see Second: Large Web site Architecture series: Distributed Message Queuing (ii))
7. This sharing summary (see article II: Large-scale Web site Architecture series: Distributed Message Queuing (ii))

I. Overview of Message Queuing

Message Queue Middleware is an important component in distributed system, which mainly solves the problems of application coupling, asynchronous message and traffic cutting front. Achieve high performance, high availability, scalability and eventual consistency architecture. is an indispensable middleware for large-scale distributed systems.

At present in the production environment, using more message queue has ACTIVEMQ,RABBITMQ,ZEROMQ,KAFKA,METAMQ,ROCKETMQ and so on.

Second, Message Queuing application scenario

The following is a description of the usage scenarios that Message Queuing commonly uses in real-world applications. Asynchronous processing, application decoupling, traffic cutting front and message communication four scenarios.

2.1 Asynchronous processing

Scene Description: After the user registration, you need to send a registration email and registration text. The traditional approach has two types of 1. Serial mode; 2. Parallel mode.

(1) Serial mode: After the registration information is written to the database successfully, send the registered mail, and then send the registration SMS. When all of the above three tasks are complete, return to the client. (Architecture kkq:466097527, welcome to join)

(2) Parallel mode: After the registration information is written to the database successfully, send the registration message at the same time, send the registration text message. When the above three tasks are completed, they are returned to the client. The difference with the serial is that the parallel way can increase the processing time.

Assuming that three business nodes each use 50 of a second, regardless of other overhead such as the network, the serial mode time is 150 milliseconds, the parallel time may be 100 milliseconds.

Since the number of requests processed by the CPU within the unit time is certain, assume that the throughput is 100 times in CPU1 seconds. The number of requests that the CPU can process in a serial mode of 1 seconds is 7 times (1000/150). The volume of requests processed in parallel is 10 times (1000/100).

Summary: As described in the above case, the traditional way of system performance (concurrency, throughput, response time) there will be bottlenecks. How to solve this problem?

The introduction of Message Queuing will not be necessary for the business logic to be processed asynchronously. After the transformation, the following structure:

According to the above convention, the response time of the user is equivalent to the time that the registration information is written to the database, which is 50 milliseconds. Registering messages, sending text messages to the message queue, returns directly, so the message queue is written quickly and can be ignored, so the user's response time may be 50 milliseconds. As a result, the throughput of the system increases to QPS per second after the architecture changes. 3 times times higher than serial, twice times higher than parallel.

2.2 Application Decoupling

Scene Description: After the user orders, the order system needs to notify the inventory system. The traditional approach is that the order system invokes the interface of the inventory system. such as: (Architecture kkq:466097527, welcome to join)

Disadvantages of the traditional model:

1) If the inventory system is inaccessible, the order reduction inventory will fail, resulting in an order failure;

2) Coupling of order system and inventory system;

How to solve the above problems? Introduce scenarios after Message Queuing is applied, such as:

• Order system: After the user orders, the order system completes the persistent processing, writes the message to the message queue, returns the user orders the order to succeed.
• Inventory System: Subscribe to the order of the message, the use of pull/push way, to obtain the order information, inventory system according to the order information, inventory operations.
• If: The inventory system will not be used properly when order is being given. Also does not affect the normal order, because after placing orders, the order system to write to the message queue will no longer care about other subsequent operations. Realize the application decoupling of order system and inventory system.

2.3 Flow Cutting Front

Traffic clipping is also a common scenario in Message Queuing, and is generally used extensively in the second or group robbery activities.

Application scenario: The second kill activity, generally because the traffic is too large, resulting in traffic explosion, the application hangs off. In order to solve this problem, it is generally necessary to join the message queue in the application front.

1. The number of people who can control the activity;
2. It can alleviate the high flow crushing application in short time;

1. After the user requests, the server receives the message queue first. If the message queue length exceeds the maximum number, then discard the user request or jump to the error page directly;
2. The second kill business is based on the request information in the message queue, and then the subsequent processing.

2.4 Log processing

Log processing refers to the use of Message Queuing in log processing, such as the application of Kafka, to solve a large number of log transmission problems. The architecture is simplified as follows: (Schema kkq:466097527, welcome to join)

• Log acquisition Client, responsible for log data collection, timed write by write Kafka queue;
• Kafka Message Queue, responsible for the receipt, storage and forwarding of log data;
• Log processing application: Subscribe to and consume the log data in the Kafka queue;

The following is the Sina Kafka log processing application case:

Turn from (http://cloud.51cto.com/art/201507/484338.htm)

(1) Kafka: Receive Message Queuing for user logs.

(2) Logstash: Do log parsing, unified into JSON output to elasticsearch.

(3) Elasticsearch: The core technology of real-time log Analysis service, a schemaless, real-time data storage service, through index organization data, both powerful search and statistical functions.

(4) Kibana: Data visualization component based on Elasticsearch, strong data visualization capability is an important reason for many companies to choose Elk Stack.

2.5 Message Communication

Message communication means that Message Queuing generally has an efficient communication mechanism built into it, so it can also be used in pure message communication. such as the implementation of peer-to-peer Message Queuing, or chat rooms.

Point-to-point communication:

Client A and Client B use the same queue for message communication.

Chat Room Newsletter:

Client A, client B, and client n subscribe to the same topic for message publishing and receiving. Implement similar chat room effects.

The above is actually two message patterns for Message Queuing, point-to-point or publish-subscribe mode. The model is, for reference.

III. Message Middleware Example 3.1 e-commerce system

Message Queuing uses highly available, durable message middleware. such as active Mq,rabbit Mq,rocket MQ. (1) After the application has completed the backbone logic, write to the message queue. The confirmation mode of the message can be turned on if the message is sent successfully. (Message Queuing Returns the success status of the message when the application returns, which guarantees the integrity of the message)

(2) Extended process (SMS, delivery Processing) subscribe to queue messages. Use push or pull to get the message and handle it.

(3) When the message is decoupled, the data consistency problem can be solved by using the final consistency method. For example, the master data is written to the database, and the extended application is based on the message queue and the database is used to follow the message queue processing.

3.2 Log Collection system

Divided into the Zookeeper registry, the log collection client, Kafka cluster and Storm cluster (Otherapp) are composed of four parts.

• Zookeeper Registration Center, put forward load balancing and address lookup service;
• The Log collection client, which collects the log of the application system and pushes the data to the Kafka queue;
• Kafka cluster: Receiving, routing, storage, forwarding and other message processing;

Storm cluster: With Otherapp at the same level, using pull to consume the data in the queue;

Large Web site Architecture series: Distributed Message Queuing (i) (RPM)