Message Queuing MQ Technology Introduction _ Message Queuing

Overview of Message queues

Message Queue Middleware is an important component in distributed system, which mainly solves the problems of application coupling, asynchronous message, traffic sharpening and so on. Achieve high-performance, highly available, scalable, and final conformance architectures. is an indispensable middleware for large distributed systems.

At present in the production environment, the use of more message queues are ACTIVEMQ, RABBITMQ, ZeroMQ, Kafka, METAMQ, ROCKETMQ and so on. Second, Message Queuing application scenario

The following is a detailed description of the usage scenarios commonly used by Message Queuing in practical applications. The scene is divided into asynchronous processing, application decoupling, traffic sharpening and message communication four scenes. 2.1 Asynchronous processing

Scenario Description: After the user registers, needs to send the registration mail and sends the registration information, the traditional practice has two kinds: serial way, parallel way serial way

After the registration information is written to the database, the registered message is sent and the registration message is sent, and after all the tasks have been completed, the information is returned to the client
Serial Mode parallel mode

After the registration information is written to the database, the operation of sending the registered message and sending the registered message is also performed. After all tasks have been completed, the information is returned to the client. Parallel mode can improve execution efficiency and reduce execution time compared with serial mode.
Parallel mode

The above comparison can be found, assuming that all three operations require 50ms of execution time, excluding network factors, the final execution completed, the serial mode requires 150ms, and the parallel mode requires 100ms.

Because the CPU in the unit time processing of the number of requests is consistent, assuming: CPU throughput per 1 seconds is 100 this, then the serial mode of 1 seconds to execute the request amount of 1000/150, less than 7 times, the parallel mode in 1 seconds to execute the amount of 1000/100, 10 times.

It can be seen from the above that the traditional serial and parallel methods are limited by the performance of the system, then how to solve the problem.
We need to introduce Message Queuing, which will not have to be the necessary business logic to process asynchronously, and thus the process of transformation

Introducing Message Queuing to process messages asynchronously

According to the above process, the user's response time is basically the time to write the user data to the database, send the registered message, send the registration message after writing message queue, can return the execution result, writes the message queue time quickly, almost can ignore, also can raise the system throughput to 20QPS, It is nearly 3 times times faster than a serial, and twice times more than a parallel way. 2.2 Application Decoupling

Scene Description: After the user orders, the order system needs to inform the inventory system.

The traditional approach is: The order system calls the inventory system interface. As shown in the following illustration:

Traditional way: Call Inventory interface

The traditional approach has the following drawbacks:
-1. If the inventory system access fails, the order reduction inventory fails, causing the order creation to fail
-2. Excess coupling between order system and inventory system

How to solve the above shortcomings. Message Queuing needs to be introduced, and the schema following the introduction of Message Queuing is shown in the following illustration:

The introduction of Message Queuing, the application of decoupling order system: After the user orders, the order system for data persistence processing, and then write the message queue, return to the order to create a successful inventory system: Use pull/push the way to obtain the next order information, inventory system according to orders information, inventory operation.

If the inventory system can not be used normally when the order is given. Also does not affect the normal order, because after ordering, the orders system writes to Message Queuing and no longer cares about its subsequent operations. Thus, the application of order system and inventory system is decoupled. 2.3 Flow Cutting Front

Flow sharpening is also a common scene in message queues, generally used in the second kill or a group of looting activities.

Application scenario: Second kill activity, usually because the flow is too large, resulting in increased traffic, application hang out. To solve this problem, it is generally necessary to join Message Queuing at the front end of the application. Can control the number of participants in the activity, can alleviate the short time high flow to the application of great pressure;

Flow cutting front processing mode system diagram is as follows:

Flow sharpening Way system diagram The server writes the message queue first after receiving a user request. At this point, if the number of messages in the message queue exceeds the maximum number, the user is directly denied the request or returned to the error page; seconds kill the business reads the request information in the message queue according to the rule of seconds to follow up the processing. 2.4 Log processing

Log processing refers to the use of Message Queuing in log processing, such as Kafka applications, to solve a large number of log transmission problems. The architecture is simplified as follows:

Schema log collection client that Message Queuing applies to log processing: responsible for log data collection, timed write to Kafka queue; Kafka Message Queuing: Responsible for receiving, storing and forwarding log data; log processing applications: Subscribe to and consume log data in Kafka queues;

This architecture in the actual development of the application, you can refer to the case: Sina technology sharing: How we carry out the 3.2 billion real-time log analysis and processing

Technical architecture Design for services Kafka: Message Queuing to receive user logs. Logstash: Do log parsing, unified into JSON output to elasticsearch. Elasticsearch: The core technology of real-time log Analysis Services, a schemaless, real-time data storage services, through the index organization of data, both powerful search and statistical functions. Kibana: Based on the Elasticsearch data visualization component, the powerful data visualization ability is the important reason that many companies choose Elk Stack. 2.5 Message Communications

Message communication means that Message Queuing generally has an efficient communication mechanism built into it, so it can also be used in pure message communication. such as the implementation of point-to-point Message Queuing, chat room and so on. Point to Point Communication
Design of point-to-point communication architecture

In the point-to-point communication architecture design, client A and client B share a message queue, which can realize the message communication function. Chat Room Newsletter
Chat Room Communication Architecture design

Client A, client B, until the client n subscribe to the same message queue, to publish and receive the message, you can realize the chat communication scheme architecture design. Third, message middleware Example 3.1 electrical business system
Schematic diagram of Electronic Business system architecture

Message Queuing uses a highly available, sustainable message middleware. For example, active Mq,rabbit mq,rocket MQ. When the application completes the skeleton logic, it writes to the message queue. The success of sending messages can open the acknowledgement mode of the message. (Message Queuing returns the message after a successful status, the application returns, thus safeguarding the integrity of the message) extends the process (text messaging, distribution processing) subscription queue messages. Use push or pull to get messages and handle them. Message will apply decoupling at the same time, bring the data consistency problem, can adopt the final consistency method to solve. For example, the master data is written to the database, and the extended application is implemented based on Message Queuing and the database method is used to implement the post-processing. 3.2 Log Collection system
Diagram of the Log collection system architecture

Divided into Zookeeper Registration center, log Collection client, Kafka cluster and Storm Cluster (Otherapp) four components. Zookeeper Registration Center, put forward load balancing and address lookup service; Log collection client is used to collect log of application system and push data to Kafka queue; Kafka cluster: Receiving, routing, storing, forwarding and other message processing; Storm cluster: At the same level as the Otherapp, the data in the queue is consumed in a pull way; JMS Messaging Service

Message Queuing will have to mention JMS. The JMS (Java message Service,java messaging) API is a standard/specification for messaging services that allows application components to create, send, receive, and read messages based on the Java EE platform. It makes distributed communication less coupled, and messaging services more reliable and asynchronous.
In the EJB architecture, a message bean can be seamlessly integrated with the JM messaging service. In the Java EE Architecture model, there is a message provider mode, which is used to realize the direct decoupling of message and application. 4.1 Message Models

In the JMS standard, there are two types of message models-Peer-to-peer (Point-point), Publish/subscribe (pub/sub). 4.1.1 Peer-to-peer Mode
Peer-to-peer mode

Peer-to-peer mode includes three roles: Message Queuing (queue), sender (Sender), receiver (Receiver). Each message is sent to a specific queue, and the receiver gets the message from the queue. Queues keep messages until they are consumed or timed out.

Peer-to-peer features each message has only one consumer (Consumer) (i.e. once consumed, messages are no longer in Message Queuing there is no time dependency between the sender and the receiver, meaning that when the sender sends a message, no matter whether the receiver is running or not, it does not affect the message being sent to the queue Recipient must reply to queue successfully after receiving message successfully

If every message you want to send is successfully processed, then Peer-to-peer mode is required.
4.1.2 Pub/sub Mode

Pub/sub mode

Contains three roles: Subject (Topic), publisher (publisher), Subscriber (subscriber). Multiple publishers send messages to topic, which the system passes to multiple subscribers.

The characteristics of pub/sub each message can have a time dependency between multiple consumer publishers and subscribers. For a subscriber to a topic (Topic), it must create a subscriber before it can consume the publisher's message. In order to consume messages, subscribers must remain in the running state.

To mitigate such strict time dependencies, JMS allows subscribers to create a durable subscription. In this way, even if the Subscriber is not activated (running), it can also receive a message from the publisher.
You can use the PUB/SUB model if you want to send a message that is not being processed, or is handled by only one message, or can be handled by multiple consumers. 4.2 Message consumption

In JMS, the generation and consumption of messages are asynchronous. For consumption, JMS message users can consume messages in two ways. Synchronous
A subscriber or receiver receives a message through the Receive method, and the receive method blocks until the message is received (or before the timeout);
Subscribers or receivers can be registered as a message listener. When the message arrives, the system automatically invokes the listener's OnMessage method.

The Jndi:java naming and directory interface is a standard Java naming system interface. You can find and access services on the network. By specifying a resource name that corresponds to a record in the database or naming service, it returns the information necessary to establish a resource connection.
Jndi plays a role in JMS to find and access the sending target or message source. 4.3JMS programming Model 1. ConnectionFactory

The factory that created the connection object, for two different JMS message models, has Queueconnectionfactory and topicconnectionfactory two species respectively. You can find ConnectionFactory objects through Jndi. 2. Destination

Destination means the message producer's message sends the target or the message source of the consumer. For a message producer, its destination is a queue or a topic (Topic), and for a message consumer, its destination is also a queue or topic (i.e., a message source).
So, destination is actually two types of objects: Queue, topic can find destination through Jndi. 3. Connection

Connection represents the link established between the client and the JMS system (the wrapper over the TCP/IP socket). Connection can produce one or more sessions. Like ConnectionFactory, there are two types of connection: Queueconnection and Topicconnection. 4. Session

A session is an interface for manipulating messages. You can create producers, consumers, messages, and so on by session. The session provides the functionality of the transaction. When you need to send/receive multiple messages using session, you can place these send/Receive actions in a transaction. Likewise, queuesession and topicsession are also divided. 5. Producers of information

The message producer is created by session and is used to send messages to destination. Similarly, message producers are divided into two types: Queuesender and Topicpublisher. You can invoke the message producer's method (send or Publish method) to send a message. 6. Message Consumers

Message consumers are created by the session to receive messages sent to destination. Two types: QueueReceiver and TopicSubscriber. Can be created individually through the session's Createreceiver (Queue) or Createsubscriber (Topic). Of course, you can also create a persistent subscriber by the Creatdurablesubscriber method of the session. 7. MessageListener

The message listener. If a message listener is registered, the listener's OnMessage method is automatically invoked once the message arrives. An MDB (Message-driven Bean) in an EJB is a messagelistener.

In-depth learning of JMS is a great help in mastering the Java Architecture and EJB architecture, and message middleware is also a necessary component of a large distributed system. This sharing mainly makes the overall introduction, the concrete depth needs everybody to study, the practice, the summary, grasps. V. Common message queues

Generally commercial containers, such as Weblogic,jboss, support the JMS Standard and are easy to develop. But for free, such as tomcat,jetty, you need to use a third party message middleware. This section describes commonly used message middleware (Active mq,rabbit Mq,zero Mq,kafka) and their characteristics. 5.1 ActiveMQ

ActiveMQ is the most popular and powerful open source message bus produced by Apache. ActiveMQ is a JMS provider implementation that fully supports the JMS1.1 and Java EE 1.4 specifications, although the JMS specification has been around for a long time, but JMS still plays a special role in today's Java EE applications.
The ACTIVEMQ features are as follows: multiple languages and protocol authoring clients.
Language: java,c,c++,c#,ruby,perl,python,php.
Application protocol: Openwire,stomp REST,WS NOTIFICATION,XMPP,AMQP fully supports the support of the JMS1.1 and Java EE 1.4 specification (persistence, XA messages, transactions) for spring.
ACTIVEMQ can easily be embedded inside a system that uses spring, and it also supports Spring2.0 features through tests of common Java server (such as Geronimo,jboss 4,glassfish,weblogic), where the JCA The 1.5 resource adaptors configuration allows ACTIVEMQ to automatically deploy to any compatible Java 1.4 Commercial server to support multiple delivery protocols: In-vm,tcp,ssl,nio,udp,jgroups,jxta Support for high-speed message persistence through JDBC and journal design guarantees high-performance cluster, client-server, point-to-point support AJAX support and Axis integration can easily invoke the embedded JMS provider for testing 5.2 RABBITMQ

RABBITMQ is a popular open source Message Queuing system developed in Erlang language. RABBITMQ is the standard implementation of AMQP (Advanced Message Queuing protocol). Support for multiple clients, such as Python, Ruby,. NET, Java, JMS, C, PHP, ActionScript, XMPP, stomp, etc., support Ajax, persistence. Used for storing and forwarding messages in distributed systems, and performing well in terms of ease of use, scalability, and high availability.

RABBITMQ structure diagram

There are several important concepts in the above diagram: Broker: Simply the Message Queuing server entity. Exchange: A message switch that specifies what rules the message is routed to and to which queue. Queue: Message Queuing carrier, where each message is put into one or more queues. Binding: Binding, which is the purpose of binding exchange and queue to routing rules. Routing key: Routing key, Exchange is delivering messages based on this keyword. Vhost: Virtual host, a broker can open multiple vhost, used as separate permissions for different users. Producer: The message producer is the program that delivers the message. Consumer: The message consumer is the program that accepts the message. Channel: Message channel, in each connection of the client, can establish multiple channel, each channel represents a session task.

The use of Message Queuing is as follows: The client connects to the Message Queuing server and opens a channel. The client declares an exchange and sets the related properties. The client declares a queue and sets the related properties. The client uses routing key to establish a binding relationship between Exchange and queue. Clients post messages to exchange.

When Exchange receives a message, it routes messages to one or more queues based on the key of the message and the binding that has been set. 5.3 ZeroMQ

The fastest message queue in history, it is actually similar to the socket of a series of interfaces, he and socket is the difference is: the ordinary socket is end-to-end (1:1 of the relationship), and ZMQ is the relationship can be n:m, people on the BSD sockets are more understanding of the point-to-point connection, Point-to-Point connections require an explicit connection, destroy connections, select Protocols (TCP/UDP), and handle errors, and ZMQ masks these details, making your network programming simpler. ZMQ is used for communication between node and node, node can be a host or a process.
Citing the official statement: "ZMQ (hereinafter ZEROMQ abbreviation ZMQ) is a simple and easy to use the transport layer, like the frame of a socket library, he makes the socket programming simpler, concise and more performance." is a message processing queue library that can flex across multiple threads, cores, and host boxes. ZMQ's clear goal is to "become part of the standard network protocol stack and then enter the Linux kernel." They are not yet seen to be successful. However, it is undoubtedly a very promising one, and a layer of encapsulation on the "traditional" BSD sockets that people need more. ZMQ makes writing high-performance Web applications extremely simple and fun. ”

Features: High performance, non-persistent, Cross-platform: Support for Linux, Windows, OS x, etc. Multiple languages support, C, C + +, Java,. NET, Python, and more than 30 kinds of development languages. Can be deployed individually or integrated into the application, and can be used as a socket communication library.

Compared with RABBITMQ, ZMQ is not like a traditional Message Queuing server, in fact, it is not a server, more like a low-level network communication library, in the socket API to do a layer of encapsulation, network communication, process communication and thread communication abstract as a unified API interface. Supports three basic and extended models of "request-reply", "Publisher-subscriber", "Parallel Pipeline".

ZEROMQ High Performance Design essentials: Lock-free queue model
For data interchange channel pipe between interactions between threads (client and session), a lock-free queue algorithm is used to register asynchronous events at both ends of the pipe, which automatically triggers read and write events when reading or writing messages to pipe. The algorithm of batch processing
For the traditional message processing, each message in the sending and receiving time, need to call the system, so for a large number of messages, the system overhead is relatively large, zeromq for the bulk of the message, the adaptability of the optimization, can be batch to receive and send messages. Multi-core thread binding without CPU switching
Different from the traditional multithreaded concurrency mode, semaphore or critical area, ZEROMQ take full advantage of multi-core, each nuclear binding run a worker thread, avoid the CPU switching overhead between multithreading. 5.4 Kafka

Kafka is a highly-throughput distributed subscription messaging system that handles all the action flow data in a consumer-scale web site. This action (web browsing, search and other user action) is a key factor in many social functions on modern networks. This data is usually resolved by processing logs and log aggregations due to throughput requirements. This is a viable solution for log data and off-line analysis systems like Hadoop, but require real-time processing. The goal of Kafka is to unify online and offline message processing through the parallel loading mechanism of Hadoop, as well as to provide real-time consumption through the cluster machine.
Kafka is a highly-throughput distributed subscription messaging system with the following features: the use of O (1) disk data structures to provide a persistent message, this structure for even terabytes of message storage can maintain a long period of stability performance. (file appended to write data, expired data deleted periodically) high throughput: Even a very common hardware Kafka can support millions of of messages per second. Support for partitioning messages through Kafka servers and consumer clusters. Supports Hadoop parallel data loading.

Kafka related Concept Broker

The Kafka cluster contains one or more servers, which are called broker Topic

Each message published to the Kafka Cluster has a category, which is called topic. (Physically different topic messages are stored separately, and logically a topic message is saved on one or more broker but the user only needs to specify the topic of the message to produce or consume the data without having to worry about where the data is stored) Partition

Parition is a physical concept in which each topic contains one or more partition. Producer

Responsible for posting messages to Kafka Broker Consumer

Message consumer, a client that reads messages to Kafka broker. Consumer Group

Each consumer belongs to a specific consumer group (you can specify group name for each consumer, or the default group if you do not specify group name).

Kafka are commonly used in large data log processing or for real-time (small latency), reliability (small amount of data loss) requiring slightly lower scenarios.

Vi. reference materials
The following is a reference to share the information and recommend the information. Resources (resources available): JMS

Comparison of point-to-point and publish subscription models
Basic concepts of JMS (i)--JMS RABBITMQ

Baidu Encyclopedia-RABBITMQ
Conan: Looking at the IT architecture in the age of large data (2) Message Queuing rabbitmq-Basic Concepts detailed introduction to Zero MQ

Several high-performance features of high performance communication library-ZEROMQ
Baidu Library-ZEROMQ Use environment and structure detailed analysis Kafka

Baidu Encyclopedia-kafka
Apache Kafka: Next Generation distributed messaging system
Kafka: A Distributed messaging system reference:

Large Web site Architecture series: Distributed Message Queuing (i)
Large Web site Architecture series: Message Queuing (ii)

Author: Step product
Link: http://www.jianshu.com/p/689ce4205021
Source: Jianshu