Large Web site Architecture series: Message Queuing (ii)

This article is a large Web site Architecture series: Message Queuing (ii), which mainly shares the JMS messaging service, common message middleware (Active mq,rabbit Mq,zero Mq,kafka). "The second part of the content of the network resources to organize and summarize, for everyone to learn summary use, the last an article source"

This sharing outline

1. Message Queuing Overview (see First: Large Web site Architecture series: Distributed Message Queuing (i))
2. Message Queuing scenario (see First: Large Web site Architecture series: Distributed Message Queuing (i))
3. Message Middleware Example (see First: Large Web site Architecture series: Distributed Message Queuing (i))
4. JMS Messaging Service
5. Common Message Queuing
6. Reference (recommended) information
7. Summary of this share

Iv. JMS Messaging Service

Message Queuing has to mention JMS. The JMS (JAVA message Service,java message Service) API is a standard/specification for messaging services that allows application components to create, send, receive, and read messages based on the Java EE platform. It makes the distributed communication less coupled, the message service more reliable and asynchronous.

In the EJB architecture, a message bean can be seamlessly integrated with the JM message service. In the Java EE architecture pattern, there is a message service pattern that enables direct decoupling of the message from the application.

4.1 Message Model

In the JMS standard, there are two message model-to-peer (point-to-point), Publish/subscribe (pub/sub).

4.1.1 Peer Mode

The peer mode contains three roles: Message Queuing (queue), Sender (sender), receiver (receiver). Each message is sent to a specific queue, and the recipient obtains the message from the queue. The queue retains messages until they are consumed or timed out.

The characteristics of peer-to

• Only one consumer per message (Consumer) (that is, once consumed, the message is no longer in the message queue)
• There is no dependency on time between the sender and the receiver, that is, when the sender sends the message, regardless of whether the recipient is running, it does not affect the message being sent to the queue
• The recipient needs to answer the queue successfully after receiving the message successfully

If you want to send every message will be successfully processed, then you need to peer mode. (Architecture kkq:466097527, welcome to join)

4.1.2 Pub/sub Mode

Contains three role topics (Topic), publisher (publisher), Subscriber (subscriber). Multiple publishers send messages to topic, which the system passes to multiple subscribers.

Features of Pub/sub

• Each message can have multiple consumers
• There is a time dependency between the Publisher and the Subscriber. Subscribers to a topic (TOPIC) must create a subscriber before they can consume the publisher's message.
• In order to consume messages, subscribers must remain in a running state.

To mitigate such strict time dependencies, JMS allows subscribers to create a durable subscription. This way, even if the Subscriber is not activated (running), it can receive the message from the publisher.

The PUB/SUB model can be used if the message that you want to send can be processed without any processing, or only handled by a single message person, or can be processed by multiple consumers.

4.2 Message consumption

In JMS, both the generation and consumption of messages are asynchronous. For consumption, JMS messages can consume messages in two ways.

(1) Synchronization

A subscriber or receiver receives a message through the Receive method, and the receive method blocks until the message is received (or before it times out);

(2) asynchronous

Subscribers or receivers can register as a message listener. When the message arrives, the system automatically calls the listener's OnMessage method.

The Jndi:java naming and directory interface is a standard Java naming system interface. You can find and access services on the network. By specifying a resource name that corresponds to a record in the database or naming service, and also returns the information necessary to establish the resource connection.

Jndi plays a role in JMS in finding and accessing the sending target or the source of the message. (Architecture kkq:466097527, welcome to join)

4.3JMS programming Model

(1) ConnectionFactory

The factory that created the connection object has two queueconnectionfactory and topicconnectionfactory for two different JMS message models. You can find the ConnectionFactory object through Jndi.

(2) Destination

Destination means that the message producer's message is sent to the target or to the source of the message consumer. For a message producer, its destination is a queue or a topic (Topic), and for the message consumer, its destination is also a queue or topic (that is, the source of the message).

So, destination is actually two types of objects: Queue, topic can find destination through Jndi.

(3) Connection

Connection represents the link established between the client and the JMS system (the wrapper to the TCP/IP socket). Connection can produce one or more sessions. Like ConnectionFactory, there are two types of connection: Queueconnection and Topicconnection.

(4) Session

The session is an interface for manipulating messages. You can create producers, consumers, messages, etc. through the session. The session provides the functionality of the transaction. When you need to send/receive multiple messages using the session, you can put these send/receive actions into a transaction. Similarly, queuesession and topicsession are also divided.

(5) Producers of messages

The message producer is created by the session and is used to send messages to destination. Similarly, there are two types of message producers: Queuesender and Topicpublisher. You can call the message producer's method (send or Publish method) to send the message.

(6) Message Consumers

The message consumer is created by the session to receive messages sent to destination. Two types: QueueReceiver and TopicSubscriber. Can be created by Createreceiver (Queue) or Createsubscriber (Topic) of the session, respectively. Of course, you can also create a persistent subscriber by the Creatdurablesubscriber method of the session.

(7) MessageListener

Message listeners. If a message listener is registered, the listener's OnMessage method is automatically invoked once the message arrives. An MDB (Message-driven Bean) in an EJB is a messagelistener.

Deep learning JMS is very helpful for mastering the Java architecture, and the message middleware is also a necessary component of a large distributed system. This sharing mainly to do a global introduction, specific deep needs everyone to learn, practice, summary, understanding.

V. Common message queues

Generally commercial containers, such as Weblogic,jboss, support the JMS Standard and are easy to develop. But for free, such as tomcat,jetty, you need to use third-party message middleware. This section describes the common message middleware (Active mq,rabbit Mq,zero Mq,kafka) and their features.

5.1 ActiveMQ

ActiveMQ is the most popular, powerful, open source messaging bus that Apache has produced. ActiveMQ is a JMS provider implementation that fully supports the JMS1.1 and the Java EE 1.4 specification, although the JMS specification has been around for a long time, but JMS still plays a special role in the middle of today's Java EE applications.

The ACTIVEMQ features are as follows:

⒈ multiple languages and protocols for writing clients. Language: java,c,c++,c#,ruby,perl,python,php. Application protocol: Openwire,stomp REST,WS NOTIFICATION,XMPP,AMQP

⒉ fully supports JMS1.1 and the Java EE 1.4 specification (persistence, XA messages, transactions)

⒊ support for spring, ACTIVEMQ can easily be embedded in a system that uses spring, and also supports Spring2.0 features

⒋ passed a test of a common Java EE server (such as Geronimo,jboss 4,glassfish,weblogic), with the configuration of the JCA 1.5 resource adaptors, Allows ACTIVEMQ to automatically deploy to any compatible Java 1.4 Business Server

⒌ supports multiple transfer protocols: In-vm,tcp,ssl,nio,udp,jgroups,jxta

⒍ support for high-speed message persistence through JDBC and journal

⒎ is designed to ensure high-performance clustering, client-server, point-to-point

⒏ supports Ajax

⒐ support for integration with axis

⒑ can easily invoke the embedded JMS provider for testing

5.2 RabbitMQ

RABBITMQ is a popular open source Message Queuing system, developed in Erlang language. RABBITMQ is the standard implementation of the AMQP (Advanced Message Queuing protocol). Support for multiple clients such as Python, Ruby,. NET, Java, JMS, C, PHP, ActionScript, XMPP, stomp, etc., support Ajax, persistent. It is used to store and forward messages in distributed system, which is very good in ease of use, extensibility, high availability and so on.

The structure diagram is as follows: (Schema kkq:466097527, welcome to join)

Several important concepts:

Broker: The Message Queuing server entity is simply the case.

Exchange: A message switch that specifies what rules the message is routed to and to which queue.

Queue: A message queue carrier in which each message is put into one or more queues.

Binding: Bind, which is the role of binding exchange and queue according to routing rules.

Routing key: The routing keyword, exchange messages are delivered based on this keyword.

Vhost: Virtual host, a broker can open multiple vhost, as a separate user permissions.

Producer: The message producer is the program that delivers the message.

Consumer: The message consumer is the program that receives the message.

Channel: The message channels, in each connection of the client, multiple channels can be established, each channel represents a session task.

The process of using Message Queuing is as follows:

(1) The client connects to the Message Queuing server and opens a channel.

(2) The client declares an exchange and sets the related properties.

(3) The client declares a queue and sets the related properties.

(4) The client uses routing key to establish a good binding relationship between Exchange and queue.

(5) Clients post messages to exchange.

When Exchange receives a message, it routes messages to one or more queues based on the key of the message and the binding that has been set.

5.3 ZeroMQ

Known as the history of the fastest message queue, it is actually similar to the socket of a series of interfaces, and his socket is the difference between: the ordinary socket is end-to-end (1:1 relationship), and ZMQ can be n:m relationship, people on the BSD socket understanding more is the point-to-point connection, A point-to-point connection requires an explicit connection, a destroy connection, a selection protocol (TCP/UDP), and a processing error, and ZMQ masks these details to make your network programming easier. ZMQ is used for the communication between node and node, which can be either a host or a process.

Citing the official saying: "ZMQ (hereinafter referred to as ZEROMQ ZMQ) is a simple and easy to use transport layer, like a framework of a socket library, he makes the socket programming simpler, more concise and higher performance. is a message processing queue library that can elastically scale across multiple threads, cores, and host boxes. ZMQ's clear goal is to "become part of the standard network protocol stack and then into the Linux kernel". They are not yet seen to be successful. However, it is undoubtedly a very promising and a layer of encapsulation on the "traditional" BSD sockets that people need most. ZMQ makes writing high-performance Web applications extremely simple and fun. ”

Features are:

• High performance, non-persistent;
• Cross-platform: Supports Linux, Windows, OS X, and more.
• Multi-lingual support, C, C + +, Java,. NET, Python and more than 30 different development languages.
• Can be deployed separately or integrated into the application;
• Can be used as a socket communication library.

Compared with RABBITMQ, ZMQ is not a traditional Message Queuing server, in fact, it is not a server at all, more like a low-level network communication library, on the socket API to do a layer of encapsulation, the network communication, process communication and thread communication is abstracted into a unified API interface. Support "Request-reply", "Publisher-subscriber", "Parallel Pipeline" three basic models and extension models.

ZEROMQ High Performance Design essentials:

1. Lock-Free queue model

For the data exchange channel between cross-thread interaction (user-side and session) pipe, using the lock-free queue algorithm CAS; asynchronous events are registered at both ends of the pipe, and read and write events are automatically triggered when the message is read or written to the pipe.

2. Batch processing algorithm

For the traditional message processing, each message in the sending and receiving time, need to call the system, so for a large number of messages, the system overhead is relatively large, zeromq for the bulk of the message, the adaptive optimization, can be bulk receive and send messages.

3, multi-core thread binding, without CPU switching

Unlike traditional multithreaded concurrency patterns, semaphores or critical sections, ZEROMQ leverages the benefits of multicore, each of which runs a worker thread and avoids CPU switching overhead between threads.

5.4 Kafka

Kafka is a high-throughput distributed publish-subscribe messaging system that handles all the action flow data in a consumer-scale website. This kind of action (web browsing, search and other user actions) is a key factor in many social functions on modern networks. This data is usually resolved by processing logs and log aggregations due to throughput requirements. This is a viable solution for the same log data and offline analysis system as Hadoop, but requires real-time processing constraints. The purpose of Kafka is to unify online and offline message processing through Hadoop's parallel loading mechanism, and also to provide real-time consumption through the cluster machine.

Kafka is a high-throughput distributed publish-subscribe messaging system with the following features:

• Provides persistence of messages through the disk data structure of O (1), a structure that maintains long-lasting performance even with terabytes of message storage. (The file is appended to the data, the expired data is deleted periodically)
• High throughput: Even very common hardware Kafka can support millions of messages per second.
• Support for partitioning messages through Kafka servers and consumer clusters.
• Supports Hadoop parallel data loading.

Kafka Related Concepts

• Broker

A Kafka cluster contains one or more servers, which are called broker[5]

• Topic

Each message published to the Kafka Cluster has a category, which is called topic. (Physically different topic messages are stored separately, logically a topic message is saved on one or more brokers but the user only needs to specify the topic of the message to produce or consume data without worrying about where the data is stored)

• Partition

Parition is a physical concept, and each topic contains one or more partition.

• Producer

Responsible for publishing messages to Kafka broker

• Consumer

The message consumer, the client that reads the message to Kafka broker.

• Consumer Group

Each consumer belongs to a specific consumer group (the group name can be specified for each consumer, and the default group if the group name is not specified).

General applications are used in large data log processing or for scenarios where there is a slight delay in real-time (small latency), and reliability (few drops of data) is required.

Vi. references

The following is a reference to the sharing of information and recommend the information you reference.

reference materials (available in reference):

( 1 ) Jms

Http://blog.sina.com.cn/s/blog_3fba24680100r777.html

http://blog.csdn.net/jiuqiyuliang/article/details/46701559 (JMS (i)--JMS basic concepts)

( 2 ) RabbitMQ

Http://baike.baidu.com/link?url=s2cU-QgOsXan7j0AM5qxxlmruz6WEeBQXX-Bbk0O3F5jt9Qts2uYQARxQxl7CBT2SO2NF2VkzX_XZLqU-CTaPa

http://blog.csdn.net/sun305355024sun/article/details/41913105

( 3 ) Zero MQ

Http://www.searchtb.com/2012/08/zeromq-primer.html

http://blog.csdn.net/yangbutao/article/details/8498790

Http://wenku.baidu.com/link?url=yYoiZ_ Pypcuuxesgqvmmley08bcptzvwf3imho2w1i-ti66yxxpplljbgxboddwggbnoehhiudslfhtz7rgzykrtmqq02dv5sv9jff4lznk

( 4 ) Kafka

Http://baike.baidu.com/link?url=qQXyqvPQ1MVrw9WkOGSGEfSX1NHy4unsgc4ezzJwU94SrPuVnrKf2tbm4SllVaN3ArGGxV_N5hw8JTT2-lw4QK

http://www.infoq.com/cn/articles/apache-kafka/

Http://www.mincoder.com/article/3942.shtml

Shared e-Data (in a group file)

(1 ) Active MQ

(2 ) Kafka

(3 ) Notify

Vii. Summary of this sharing

This week's share includes a summary of Message Queuing, common Message Queuing scenarios (asynchronous processing, application decoupling, traffic clipping, log processing and messaging), the JMS Java Messaging Service, and several popular Message Queuing introductions. Finally, two architectures using message middleware are demonstrated.

Because of the time, some of the explanation is not meticulous, we can ask the/google Niang, hope this sharing to everyone helpful.

This is the last time before the Spring Festival, we will continue to share the year, next year will continue the "large Web site Architecture series", and will increase the "Step by Step Learning Architecture series." The specific time and share content will be announced by QQ group notice to everyone. Thank you for your attention.

Large Web site Architecture series: Message Queuing (ii)