About the use of Message Queuing----ACTIVEMQ,RABBITMQ,ZEROMQ,KAFKA,METAMQ,ROCKETMQ

I. Overview of Message Queuing
Message Queuing middleware is an important component in distributed system, which mainly solves the problems of application decoupling, asynchronous message, traffic cutting and so on, realizing high performance, high availability, scalable and final consistency architecture. More message queues are currently available with ACTIVEMQ,RABBITMQ,ZEROMQ,KAFKA,METAMQ,ROCKETMQ

Second, Message Queuing application scenario
The following is a description of the usage scenarios that Message Queuing commonly uses in real-world applications. Asynchronous processing, application decoupling, traffic cutting front and message communication four scenarios.

2.1 Asynchronous processing
Scene Description: After the user registration, you need to send a registration email and registration text. The traditional approach has two types of 1. Serial mode; 2. Parallel mode
A, serial mode: After the registration information is written to the database successfully, send the registration message, and then send the registration text message. When all of the above three tasks are complete, return to the client.

b, parallel mode: After the registration information is written to the database successfully, send the registration message at the same time, send the registration text message. When the above three tasks are completed, they are returned to the client. The difference with the serial is that the parallel way can increase the processing time

Assuming that three business nodes each use 50 of a second, regardless of other overhead such as the network, the serial mode time is 150 milliseconds, the parallel time may be 100 milliseconds.
Since the number of requests processed by the CPU within the unit time is certain, assume that the throughput is 100 times in CPU1 seconds. The number of requests that the CPU can process in a serial mode of 1 seconds is 7 times (1000/150). The number of requests processed in parallel is 10 times (1000/100)
Summary: As described in the above case, the traditional way of system performance (concurrency, throughput, response time) there will be bottlenecks. How to solve this problem.

The introduction of Message Queuing will not be necessary for the business logic to be processed asynchronously. After the transformation, the following structure:

According to the above convention, the response time of the user is equivalent to the time that the registration information is written to the database, which is 50 milliseconds. Registering messages, sending text messages to the message queue, returns directly, so the message queue is written quickly and can be ignored, so the user's response time may be 50 milliseconds. As a result, the throughput of the system increases to QPS per second after the architecture changes. 3 times times higher than serial, twice times higher than parallel.

2.2 Application Decoupling
Scene Description: After the user orders, the order system needs to notify the inventory system. The traditional approach is that the order system invokes the interface of the inventory system. The following figure:

Disadvantages of the traditional model: if the inventory system is inaccessible, the order reduction inventory will fail, resulting in an order failure, and the order system coupled with the inventory system

How to solve the above problems. Introduce the scenario after applying Message Queuing, as shown in the following diagram:

Order system: After the user orders, the order system completes the persistent processing, writes the message to the message queue, returns the user orders the order to succeed
Inventory System: Subscribe to the order of the message, the use of pull/push method, to obtain the order information, inventory system according to the order information, inventory operations
If: The inventory system will not be used properly when order is being given. Also does not affect the normal order, because after placing orders, the order system to write to the message queue will no longer care about other subsequent operations. Realization of the application decoupling of order system and inventory system

2.3 Flow Cutting Front
Traffic clipping is also a common scenario in Message Queuing, and is generally used extensively in the second or group robbery activities.
Application scenario: The second kill activity, generally because the traffic is too large, resulting in traffic explosion, the application hangs off. In order to solve this problem, it is generally necessary to join the message queue in the application front.
A, the number of people who can control the activity
B, can alleviate the short-term high-flow crushing application

After the user requests, the server receives the message queue first. If the message queue length exceeds the maximum number, discard the user request or jump to the error page directly.
Seconds to kill the business based on the request information in the message queue, and then do the subsequent processing

2.4 Log processing
Log processing refers to the use of Message Queuing in log processing, such as the application of Kafka, to solve a large number of log transmission problems. The architecture is simplified as follows

Log acquisition Client, responsible for log data collection, timed write to write Kafka queue
Kafka Message Queuing, responsible for the receipt, storage and forwarding of log data
Log processing app: Subscribe to and consume log data from Kafka queues

2.5 Message Communication
Message communication means that Message Queuing generally has an efficient communication mechanism built into it, so it can also be used in pure message communication. such as the implementation of peer-to-peer Message Queuing, or chat rooms, etc.
Point-to-point communication:

Client A and Client B use the same queue for message communication.

Chat Room Newsletter:

Client A, client B, and client n subscribe to the same topic for message publishing and receiving. Implement similar chat room effects.

The above is actually two message patterns for Message Queuing, point-to-point or publish-subscribe mode. The model is a schematic diagram for reference.

Third, message middleware example
3.1 E-commerce system

Message Queuing uses highly available, durable message middleware. such as active Mq,rabbit Mq,rocket MQ.
(1) After the application has completed the backbone logic, write to the message queue. The confirmation mode of the message can be turned on if the message is sent successfully. (Message Queuing Returns the success status of the message when the application returns, which guarantees the integrity of the message)
(2) Extended process (SMS, delivery Processing) subscribe to queue messages. Use push or pull to get the message and handle it.
(3) When the message is decoupled, the data consistency problem can be solved by using the final consistency method. For example, the master data is written to the database, and the extended application is based on the message queue and the database is used to follow the message queue processing.

3.2 Log Collection system

Divided into the Zookeeper registry, the log collection client, Kafka cluster and Storm cluster (Otherapp) are composed of four parts.
Zookeeper registry, proposing load balancing and address lookup services
Log collection Client for capturing application system logs and pushing data to the Kafka queue
Kafka cluster: Receiving, routing, storing, forwarding and other message processing
Storm cluster: Consume data from a queue in the same level as the Otherapp, using pull

Iv. JMS Messaging Service
Message Queuing has to mention JMS. The JMS (JAVA message Service,java message Service) API is a standard/specification for messaging services that allows application components to create, send, receive, and read messages based on the Java EE platform. It makes the distributed communication less coupled, the message service more reliable and asynchronous.
In the EJB architecture, a message bean can be seamlessly integrated with the JM message service. In the Java EE architecture pattern, there is a message service pattern that enables direct decoupling of the message from the application.

4.1 Message Model
In the JMS standard, there are two message model-to-peer (point-to-point), Publish/subscribe (pub/sub).

4.1.1 Peer Mode

The peer mode contains three roles: Message Queuing (queue), Sender (sender), receiver (receiver). Each message is sent to a specific queue, and the recipient obtains the message from the queue. The queue retains messages until they are consumed or timed out.

The characteristics of peer-to
Only one consumer per message (Consumer) (that is, once consumed, the message is no longer in the message queue)
There is no dependency on time between the sender and the receiver, that is, when the sender sends the message, regardless of whether the recipient is running, it does not affect the message being sent to the queue
The recipient needs to answer the queue successfully after receiving the message successfully
If you want to send every message will be successfully processed, then you need to peer mode.

4.1.2 Pub/sub Mode

Contains three role topics (Topic), publisher (publisher), Subscribers (subscriber), multiple publishers send messages to Topic, which the system delivers to multiple subscribers.

Features of Pub/sub
Each message can have multiple consumers
There is a time dependency between the Publisher and the Subscriber. Subscribers to a topic (TOPIC) must create a subscriber before they can consume a publisher's message
In order to consume messages, subscribers must remain in a running state
To mitigate such strict time dependencies, JMS allows subscribers to create a durable subscription. This way, even if the Subscriber is not activated (running), it can receive the message from the publisher.
The PUB/SUB model can be used if the message that you want to send can be processed without any processing, or only handled by a single message person, or can be processed by multiple consumers.

4.2 Message consumption
In JMS, both the generation and consumption of messages are asynchronous. For consumption, JMS messages can consume messages in two ways.
(1) Synchronization
A subscriber or receiver receives a message through the Receive method, and the receive method blocks until the message is received (or before it times out);

(2) asynchronous
Subscribers or receivers can register as a message listener. When the message arrives, the system automatically calls the listener's OnMessage method.

The Jndi:java naming and directory interface is a standard Java naming system interface. You can find and access services on the network. By specifying a resource name that corresponds to a record in the database or naming service, and also returns the information necessary to establish the resource connection.
Jndi plays a role in JMS in finding and accessing the sending target or the source of the message.

v. Common message queues

Generally commercial containers, such as Weblogic,jboss, support the JMS Standard and are easy to develop. But for free, such as tomcat,jetty, you need to use third-party message middleware. This section describes the common message middleware (Active mq,rabbit Mq,zero Mq,kafka) and their features.

5.1 ActiveMQ
ActiveMQ is the most popular, powerful, open source messaging bus that Apache has produced. ActiveMQ is a JMS provider implementation that fully supports the JMS1.1 and the Java EE 1.4 specification, although the JMS specification has been around for a long time, but JMS still plays a special role in the middle of today's Java EE applications.

The ACTIVEMQ features are as follows:
⒈ multiple languages and protocols for writing clients. Language: java,c,c++,c#,ruby,perl,python,php. Application protocol: Openwire,stomp REST,WS NOTIFICATION,XMPP,AMQP
⒉ fully supports JMS1.1 and the Java EE 1.4 specification (persistence, XA messages, transactions)
⒊ support for spring, ACTIVEMQ can easily be embedded in a system that uses spring, and also supports Spring2.0 features
⒋ passed a test of a common Java EE server (such as Geronimo,jboss 4,glassfish,weblogic), with the configuration of the JCA 1.5 resource adaptors, Allows ACTIVEMQ to automatically deploy to any compatible Java 1.4 Business Server
⒌ supports multiple transfer protocols: In-vm,tcp,ssl,nio,udp,jgroups,jxta
⒍ support for high-speed message persistence through JDBC and journal
⒎ is designed to ensure high-performance clustering, client-server, point-to-point
⒏ supports Ajax
⒐ support for integration with axis
⒑ can easily invoke the embedded JMS provider for testing

5.2 Kafka
Kafka is a high-throughput distributed publish-subscribe messaging system that handles all the action flow data in a consumer-scale website. This kind of action (web browsing, search and other user actions) is a key factor in many social functions on modern networks. This data is usually resolved by processing logs and log aggregations due to throughput requirements. This is a viable solution for the same log data and offline analysis system as Hadoop, but requires real-time processing constraints. The purpose of Kafka is to unify online and offline message processing through Hadoop's parallel loading mechanism, and also to provide real-time consumption through the cluster machine.

The

Kafka is a high-throughput distributed publish-subscribe messaging system with the following features:
provides persistence of messages through the O (1) disk data structure, which can maintain long-term stability even with terabytes of message storage. (The file is appended to the data, the expired data is deleted periodically)
High throughput: Even very common hardware Kafka can support messages of millions of per second
supports partitioning messages through Kafka servers and consumer clusters
support for Hadoop parallel data loading
Kafka related Concepts The
Broker
Kafka cluster contains one or more servers, called Broker[5]
Topic
Each message published to a Kafka cluster has a category called Topic. (Physically different topic messages are stored separately, logically a topic message is saved on one or more brokers but the user only needs to specify the topic of the message to produce or consume data without worrying about where the data is stored)
Partition
Parition is a physical concept, and each topic contains one or more partition.
Producer
is responsible for publishing messages to the Kafka broker
Consumer
Message Consumer, to the client that reads the message to Kafka broker.
Consumer Group
Each Consumer belongs to a specific Consumer group (the group name can be specified for each Consumer, and the default group if the group name is not specified). The
is typically used in large data log processing or in the case of real-time (small latency), and reliability (a small amount of data loss) requires a slightly lower scenario.