Web site schema: Message Queuing

Overview of Message queues

Message Queue Middleware is an important component in distributed system, which mainly solves the problems of application coupling, asynchronous message, traffic sharpening and so on. Achieve high-performance, highly available, scalable, and final conformance architectures. is an indispensable middleware for large distributed systems.

At present in the production environment, the use of more message queues have ACTIVEMQ,RABBITMQ,ZEROMQ,KAFKA,METAMQ,ROCKETMQ and so on. second, Message Queuing application scenario

The following describes the usage scenarios commonly used by Message Queuing in practical applications. Asynchronous processing, application decoupling, flow sharpening and message communication four scenes. 2.1 Asynchronous Processing

Scene Description: After the user registers, needs to send the registration mail and the registration message. The traditional approach has two kinds of 1 serial ways; 2. Parallel mode.

(1) Serial mode: The registration information will be written to the database after the successful, send the registered mail, and then send the registered SMS. Once all three of these tasks are complete, return to the client.

(2) Parallel mode: After the registration information written to the database successfully, send the registered mail at the same time, send the registration SMS. When the above three tasks are completed, return to the client. The difference with the serial is that the parallel way can improve the processing time.

Assuming that three business nodes each use 50 seconds, regardless of other costs such as the network, the serial mode time is 150 milliseconds, and the parallel time may be 100 milliseconds.

Because the number of requests processed by the CPU within the unit time is certain, it is assumed that throughput is 100 times in CPU1 seconds. The number of requests that the CPU can handle in a serial mode of 1 seconds is 7 times (1000/150). The number of requests processed in parallel mode is 10 times (1000/100).

Summary: As described above, the traditional way system performance (concurrency, throughput, response time) will have bottlenecks. How to solve this problem.

The introduction of Message Queuing will not be necessary for the business logic to be processed asynchronously. The reconstructed structure is as follows:

According to the above conventions, the user's response time is equivalent to the time the registration information was written to the database, which is 50 milliseconds. Registering messages, sending text messages to message queues, and returning directly, so the message queue is written quickly and can be largely ignored, so the user's response time may be 50 milliseconds. As a result, the system throughput increases to QPS per second after the schema changes. 3 times times faster than serial, and twice times higher than parallel. 2.2 Application Decoupling

Scene Description: After the user orders, the order system needs to inform the inventory system. The traditional approach is that the order system calls the inventory system's interface. The following figure:

Disadvantages of traditional models:

1 if the inventory system can not be accessed, the order to reduce inventory will fail, resulting in an order failure;

2 The order system is coupled with the inventory system;

How to solve the above problems. Introduce the scenario after applying Message Queuing, as shown in the following illustration:

Order system: After the user orders, the order system completes the persistence processing, writes the message to the message queue, returns the user order to be successful.

Inventory System: Subscribe to a single message, using pull/push the way to obtain the next single information, inventory system according to the order of information, inventory operations.

If: In the order of the inventory system can not be used normally. Also does not affect the normal order, because after ordering, the system writes to Message Queuing and no longer cares about other subsequent operations. Realize the application of order system and inventory system decoupling. 2.3 Flow Cutting front

Flow sharpening is also a common scene in message queues, generally used in the second kill or a group of looting activities.

Application scenario: Second kill activity, usually because the flow is too large, resulting in increased traffic, application hang out. To solve this problem, it is generally necessary to join Message Queuing at the front end of the application.

The number of persons who can control the activity;

Can alleviate the short time high flow pressure collapse application;

The user's request, after the server receives, writes the message queue first. If the message queue length exceeds the maximum number, then directly discard the user request or jump to the error page;

Second kill business follow up on the request information in the message queue. 2.4 Log processing

Log processing refers to the use of Message Queuing in log processing, such as Kafka applications, to solve a large number of log transmission problems. The architecture is simplified as follows:

Log Collection client, responsible for log data collection, timed write Kafka queue;

Kafka Message Queue, responsible for receiving, storing and forwarding of log data;

Log processing application: Subscribe and consume the log data in the Kafka queue;

The following is the Sina Kafka log processing application case:

Transfer from (http://cloud.51cto.com/art/201507/484338.htm)

(1) Kafka: A message queue that receives user logs.

(2) logstash: Do log parsing, unified into JSON output to elasticsearch.

(3) elasticsearch: Real-time log Analysis Service core technology, a schemaless, real-time data storage services, through the index organization of data, both powerful search and statistical functions.

(4) Kibana: Based on the Elasticsearch data visualization component, the powerful data visualization ability is the important reason that many companies choose Elk Stack. 2.5 Message Communications

Message communication means that Message Queuing generally has an efficient communication mechanism built into it, so it can also be used in pure message communication. such as the implementation of point-to-point Message Queuing, or chat room and so on.

Point to point communication:

Client A and Client B use the same queue for message communication.

Chat Room Newsletter:

Client A, client B, client n subscribe to the same topic for message publishing and reception. Achieve a similar chat room effect.

The above is actually two message modes for Message Queuing, point to Point, or publish subscription mode. The model is a schematic diagram for reference. III. Message-oriented middleware example 3.1 Electric Business system

Message Queuing uses a highly available, persistent message middleware. For example, active Mq,rabbit mq,rocket MQ. (1) When the application completes the backbone logic processing, it writes to the message queue. The success of sending messages can open the acknowledgement mode of the message. (Message Queuing returns the message after a successful state, the application returns, thus safeguarding the integrity of the message)

(2) Extended process (text messaging, distribution processing) subscription queue message. Use push or pull to get messages and handle them.

(3) The message will apply decoupling at the same time, bring the data consistency problem, can use the final consistency method to solve. For example, the master data is written to the database, and the extended application is implemented based on Message Queuing and the database method is used to implement the post-processing. 3.2 Log Collection system

Divided into Zookeeper Registration center, log Collection client, Kafka cluster and Storm Cluster (Otherapp) four components.

Zookeeper Registration Center, put forward load balancing and address lookup service;

The Log collection client is used to collect the log of the application system and push the data to the Kafka queue;

IV. JMS Messaging Services

Message Queuing will have to mention JMS. The JMS (JAVA message Service,java messaging) API is a standard/specification for messaging services that allows application components to create, send, receive, and read messages based on the Java EE platform. It makes distributed communication less coupled, and messaging services more reliable and asynchronous.

In the EJB architecture, a message bean can be seamlessly integrated with the JM messaging service. In the Java EE Architecture model, there is a message provider mode, which is used to realize the direct decoupling of message and application. 4.1 Message Models

In the JMS standard, there are two types of message models-Peer-to-peer (Point-point), Publish/subscribe (pub/sub). 4.1.1 Peer-to-peer Mode

Peer-to-peer mode includes three roles: Message Queuing (queue), sender (Sender), receiver (Receiver). Each message is sent to a specific queue, and the receiver gets the message from the queue. Queues keep messages until they are consumed or timed out.

The characteristics of Peer-to-peer

Each message has only one consumer (Consumer) (that is, once consumed, the message is no longer in the message queue)

There is no time dependency between the sender and the receiver, meaning that when the sender sends the message, no matter whether the receiver is running or not, it does not affect the message being sent to the queue

Recipient must reply to queue successfully after receiving message successfully

If every message you want to send is successfully processed, then Peer-to-peer mode is required. 4.1.2 Pub/sub Mode

Contains three role topics (Topic), publisher (publisher), Subscriber (subscriber). Multiple publishers send messages to topic, which the system passes to multiple subscribers.

Characteristics of Pub/sub

Each message can have multiple consumers

There is a time dependency between the Publisher and the Subscriber. For a subscriber to a topic (Topic), it must create a subscriber before it can consume the publisher's message.

In order to consume messages, subscribers must remain in the running state.

To mitigate such strict time dependencies, JMS allows subscribers to create a durable subscription. In this way, even if the Subscriber is not activated (running), it can also receive a message from the publisher.

You can use the PUB/SUB model if you want to send a message that is not being processed, or is handled by only one message, or can be handled by multiple consumers. 4.2 Message Consumption

In JMS, the generation and consumption of messages are asynchronous. For consumption, JMS message users can consume messages in two ways.

(1) Synchronous

A subscriber or receiver receives a message through the Receive method, and the receive method blocks until the message is received (or before the timeout);

(2) asynchronous

Subscribers or receivers can be registered as a message listener. When the message arrives, the system automatically invokes the listener's OnMessage method.

The Jndi:java naming and directory interface is a standard Java naming system interface. You can find and access services on the network. By specifying a resource name that corresponds to a record in the database or naming service, it returns the information necessary to establish a resource connection.

Jndi plays a role in JMS to find and access the sending target or message source. 4.3JMS programming Model

(1) ConnectionFactory

The factory that created the connection object, for two different JMS message models, has Queueconnectionfactory and topicconnectionfactory two species respectively. You can find ConnectionFactory objects through Jndi.

(2) Destination

Destination means the message producer's message sends the target or the message source of the consumer. For a message producer, its destination is a queue or a topic (Topic), and for a message consumer, its destination is also a queue or topic (i.e., a message source).

So, destination is actually two types of objects: Queue, topic can find destination through Jndi.

(3) Connection

Connection represents the link established between the client and the JMS system (the wrapper over the TCP/IP socket). Connection can produce one or more sessions. Like ConnectionFactory, there are two types of connection: Queueconnection and Topicconnection.

(4) session

A session is an interface for manipulating messages. You can create producers, consumers, messages, and so on by session. The session provides the functionality of the transaction. When you need to send/receive multiple messages using session, you can place these send/Receive actions in a transaction. Likewise, queuesession and topicsession are also divided.

(5) The producer of the message

The message producer is created by session and is used to send messages to destination. Similarly, message producers are divided into two types: Queuesender and Topicpublisher. You can invoke the message producer's method (send or Publish method) to send a message.

(6) Message Consumers

Message consumers are created by the session to receive messages sent to destination. Two types: QueueReceiver and TopicSubscriber. Can be created individually through the session's Createreceiver (Queue) or Createsubscriber (Topic). Of course, you can also create a persistent subscriber by the Creatdurablesubscriber method of the session.

(7) MessageListener

The message listener. If a message listener is registered, the listener's OnMessage method is automatically invoked once the message arrives. An MDB (Message-driven Bean) in an EJB is a messagelistener.

In-depth learning of JMS is a great help in mastering the Java architecture, EJB architecture, and message middleware as a necessary component of a large distributed system. This sharing mainly makes the overall introduction, the concrete depth needs everybody to study, the practice, the summary, grasps. v. Common message queues

Generally commercial containers, such as Weblogic,jboss, support the JMS Standard and are easy to develop. But for free, such as tomcat,jetty, you need to use a third party message middleware. This section describes commonly used message middleware (Active mq,rabbit Mq,zero Mq,kafka) and their characteristics. 5.1 ActiveMQ

ActiveMQ is the most popular and powerful open source message bus produced by Apache. ActiveMQ is a JMS provider implementation that fully supports the JMS1.1 and Java EE 1.4 specifications, although the JMS specification has been around for a long time, but JMS still plays a special role in today's Java EE applications.

The ACTIVEMQ features are as follows:

⒈ multiple languages and protocols to write clients. Language: java,c,c++,c#,ruby,perl,python,php. Application protocol: Openwire,stomp REST,WS NOTIFICATION,XMPP,AMQP

⒉ fully supports JMS1.1 and Java EE 1.4 Specifications (persistence, XA messages, transactions)

⒊ support for spring, ACTIVEMQ can easily be embedded inside a system that uses spring, and it also supports Spring2.0 features

⒋ passed the test of a common Java server, such as Geronimo,jboss 4,glassfish,weblogic, through the JCA 1.5 resource adaptors configuration, Allows ACTIVEMQ to be automatically deployed to any compatible Java 1.4 Business Server

⒌ supports multiple delivery protocols: In-vm,tcp,ssl,nio,udp,jgroups,jxta

⒍ support for high-speed message persistence via JDBC and journal

⒎ from the design to ensure a high-performance cluster, client-server, point to point

⒏ Support Ajax

⒐ support for integration with axis

⒑ can easily invoke the embedded JMS provider for testing 5.2 rabbitmq

RABBITMQ is a popular open source Message Queuing system developed in Erlang language. RABBITMQ is the standard implementation of AMQP (Advanced Message Queuing protocol). Support for multiple clients, such as Python, Ruby,. NET, Java, JMS, C, PHP, ActionScript, XMPP, stomp, etc., support Ajax, persistence. Used for storing and forwarding messages in distributed systems, and performing well in terms of ease of use, scalability, and high availability.

The structure diagram is as follows:

Several important concepts:

Broker: In simple terms, Message Queuing server entities.

Exchange: A message switch that specifies what rules the message is routed to and to which queue.

Queue: Message Queuing carrier, where each message is put into one or more queues.

Binding: Binding, which is the purpose of binding exchange and queue to routing rules.

Routing key: Routing key, Exchange is delivering messages based on this keyword.

Vhost: Virtual host, a broker can open multiple vhost, used as separate permissions for different users.

Producer: The message producer is the program that delivers the message.

Consumer: The message consumer is the program that accepts the message.

Channel: Message channel, in each connection of the client, can establish multiple channel, each channel represents a session task.

The use process for Message Queuing is as follows:

(1) The client connects to the Message Queuing server and opens a channel.

(2) The client declares an exchange and sets the related properties.

(3) The client declares a queue and sets the related properties.

(4) The client uses routing key to establish a good binding between Exchange and queue.

(5) Clients post messages to exchange.

When Exchange receives a message, it routes messages to one or more queues based on the key of the message and the binding that has been set. 5.3 ZeroMQ

The fastest message queue in history, it is actually similar to the socket of a series of interfaces, he and socket is the difference is: the ordinary socket is end-to-end (1:1 of the relationship), and ZMQ is the relationship can be n:m, people on the BSD sockets are more understanding of the point-to-point connection, Point-to-Point connections require an explicit connection, destroy connections, select Protocols (TCP/UDP), and handle errors, and ZMQ masks these details, making your network programming simpler. ZMQ is used for communication between node and node, node can be a host or a process.

Citing the official statement: "ZMQ (hereinafter ZEROMQ abbreviation ZMQ) is a simple and easy to use the transport layer, like the frame of a socket library, he makes the socket programming simpler, concise and more performance." is a message processing queue library that can flex across multiple threads, cores, and host boxes. ZMQ's clear goal is to "become part of the standard network protocol stack and then enter the Linux kernel." They are not yet seen to be successful. However, it is undoubtedly a very promising one, and a layer of encapsulation on the "traditional" BSD sockets that people need more. ZMQ makes writing high-performance Web applications extremely simple and fun. ”

Features are:

High performance, non-persistent;

Cross-platform: Support for Linux, Windows, OS X, and more.

Multiple languages support, C, C + +, Java,. NET, Python, and more than 30 kinds of development languages.

Can be deployed individually or integrated into the application;

Can be used as a socket communication library.

Compared with RABBITMQ, ZMQ is not like a traditional Message Queuing server, in fact, it is not a server, more like a low-level network communication library, in the socket API to do a layer of encapsulation, network communication, process communication and thread communication abstract as a unified API interface. Supports three basic and extended models of "request-reply", "Publisher-subscriber", "Parallel Pipeline".

ZEROMQ High Performance Design essentials:

1, lock-free queue model

For data interchange channel pipe between interactions between threads (client and session), a lock-free queue algorithm is used to register asynchronous events at both ends of the pipe, which automatically triggers read and write events when reading or writing messages to pipe.

2, the batch processing algorithm

For the traditional message processing, each message in the sending and receiving time, need to call the system, so for a large number of messages, the system overhead is relatively large, zeromq for the bulk of the message, the adaptability of the optimization, can be batch to receive and send messages.

3, multiple cores under the thread binding, no CPU switching

Different from the traditional multithreaded concurrency mode, semaphore or critical area, ZEROMQ take full advantage of multi-core, each nuclear binding run a worker thread, avoid the CPU switching overhead between multithreading. 5.4 Kafka

Kafka is a highly-throughput distributed subscription messaging system that handles all the action flow data in a consumer-scale web site. This action (web browsing, search and other user action) is a key factor in many social functions on modern networks. This data is usually resolved by processing logs and log aggregations due to throughput requirements. This is a viable solution for log data and off-line analysis systems like Hadoop, but require real-time processing. The goal of Kafka is to unify online and offline message processing through the parallel loading mechanism of Hadoop, as well as to provide real-time consumption through the cluster machine.

Kafka is a highly-throughput distributed subscription message system with the following characteristics:

The use of O (1) disk data structures to provide a persistent message, this structure for even the number of terabytes of message storage can maintain a long period of stable performance. (file appended to write data, expired data is periodically deleted)

High throughput: Even the most common hardware Kafka can support millions of of messages per second.

Support for partitioning messages through Kafka servers and consumer clusters.

Supports Hadoop parallel data loading.

Kafka Related Concepts

Broker

The Kafka cluster contains one or more servers, which are called broker[5]

Topic

Each message published to the Kafka Cluster has a category, which is called topic. (Physically different topic messages are stored separately, and logically a topic message is saved on one or more broker but the user only needs to specify the topic of the message to produce or consume the data without having to care where the data is stored)

Partition

Parition is a physical concept in which each topic contains one or more partition.

Producer

Responsible for posting messages to Kafka broker

Consumer

Message consumer, a client that reads messages to Kafka broker.

Consumer Group

Each consumer belongs to a specific consumer group (you can specify group name for each consumer, or the default group if you do not specify group name).

General application in large data log processing or for real-time (small latency), reliability (small amount of data loss) requires slightly lower scene use.

Http://mp.weixin.qq.com/s?__biz=MzI4OTU3ODk3NQ==&mid=2247483797&idx=1&sn= 108bc92611366079f689310373de3c7e&chksm= ec2c4b25db5bc233de4a51461567a961a5b6e865ca3e75079cd85ecfeba2ce9d3778a1907b5b&mpshare=1&scene=23& SRCID=0113XVBST2CKOUH2657Y85SO#RD