Introduction to message-oriented middleware Apache Qpid

Introduction to message-oriented middleware Apache Qpid

Overview: Qpid is an object-oriented message-oriented middleware developed by Apache. It is an AMQP implementation that can communicate with other systems that comply with the AMQP protocol. Qpid provides client libraries for mainstream programming languages such as C ++, Python, Java, and C #, which are easy to install and use. Compared with other AMQP implementations, The Qpid community is very active and is expected to become a standard AMQP middleware product. In addition to complying with the basic requirements of AMQP, Qpid provides many additional HA features and is very suitable for message communication in cluster environments.

Introduction: Qpid Application Scenario

Communication is a basic requirement. Just as people need to communicate with each other, large software systems often need internal or external communication.

The most basic communication mode for inter-system communication is socket, but the socket layer is relatively low. It is very difficult to use. If you need some advanced features, it requires a lot of programming burden.

With the original Correspondence of socket, enterprise-level application software often has a variety of communication requirements from simple to complex, represented by different communication models. Common examples include:

• Point-to-point: A sends messages to B.
• Broadcast: messages sent by A to all others
• Multicast: messages sent by A to multiple but not all others.
• Requester/response: similar to the webpage access communication method, the client sends a request and waits, and the server replies with the request.
• Pub-sub: People who publish magazines do not know who is reading this magazine, and those who subscribe to it do not care who is publishing this magazine. The publisher only publishes the information, and the subscriber only receives the information as needed.
• Store-and-forward: the storage and forwarding model is similar to mail delivery. The sender of a letter writes the message to someone, but the recipient does not have to wait at home when the letter is sent, he does not know that a message is sent to him. However, this message will not be lost and will be placed in the recipient's mailbox. This model allows asynchronous information exchange.
• Other communication models...

In addition to various communication models, communication between systems has other issues to consider. For example, enterprise-level applications often have a huge amount of data to be exchanged and require high reliability. For example, if a distributed financial processing software is used by thousands of users every moment, it is necessary to generate a massive amount of unimaginable messages. Each message may be related to key information such as someone's bank account, the loss will cause huge losses. Writing such a communication middleware is not easy. Even if it is written, if it needs to interact with other software systems, it requires a lot of format conversion and interface migration.

To solve these problems, many software products and protocols have been developed. People have made a lot of progress from early RPC to complex message-oriented middleware (MOM) to JMS, but these technologies still have their own problems.

RPC, Corba, and other technologies are synchronous, that is, the caller must wait for the reply from the other party, which means that the caller must understand the receiver and is a tightly coupled model. Tight coupling means that it is not flexible, and the only thing that remains unchanged in the software industry is changes. When the demand and environment change, the cost of tightly coupled application modification is very high.

Therefore, many message-oriented middleware Products have emerged, breaking the tight coupling relationship between message producers and consumers. However, middleware Products are defined and implemented by various vendors. When Integrating enterprise-level applications, it is found that different technologies and middleware Products are often used for various applications, it is very difficult for these products to communicate with each other.

JMS is a kind of standardization effort, but its disadvantage is that JMS is a J2EE standard. If it is not a product implemented using Java technology, it is still troublesome to use JMS.

Therefore, even today, people still want to have a message-oriented middleware product that is powerful, platform/language independent, and standardized.

If this is what you always think, Qpid is an open-source software worth understanding. It implements reliable and complex communication middleware, supports a variety of communication models, high efficiency, platform language independence, and implements the industry's communication standard AMQP.

AMQP and Qpid

AMQP is the Advanced Message Queuing Protocol, that is, the Advanced Message Queue Protocol. Unlike the technology listed above, AMQP is a standard message-oriented middleware protocol. Her ideal is to allow applications in different languages and systems to communicate with each other and provide a simple and unified model and programming interface. In this way, people can use various languages and platforms to implement their own applications. When they need to communicate with other systems, they only need to acknowledge the AMQP protocol.

For example...

People from all over the world use different languages for geographical and historical reasons and it is very difficult to communicate with each other. AMQP is similar to an automatic translation machine. When I say something to it in Chinese, if a person in the English world wants to listen, he can hear the English version of the exact translation.

In addition, this translator provides many other benefits, such as the 12-hour time difference between China and the United States. If I want to talk to an American now, he must get up in the middle of the night, or I have to wait until he gets up, but through this machine, I can just finish talking about what the Americans will hear when they get up. I am very relieved that this sentence will never be lost, nor will it be the same;

I don't really care how many people listen. If more people want to listen, they can hear it at any time.

If I only want some people to hear it, I can also encrypt the authentication;

If some people don't want to listen, some people want to listen, then this translation machine can also know who wants to listen, rather than sending my voice to people who don't want to hear it.

I have never enjoyed this kind of communication and experience as a human, but AMQP has already provided such services for many computer software.

AMQP comes from JPMorgon, which was initially the messaging middleware used by the wealthy investment bank. The initiator John o'hara was very frustrated. He said, "from 1996 to 2003, I have been waiting for such a standard, but I have never waited. I can't wait ", moreover, "investment banks have the most urgent need for such standards, and banks have never lacked technical experts", so he developed one by himself. I think if you want to accomplish something, you need such a heroic spirit.

Due to his efforts, AMQP quickly moved from the financial sector to the entire computer industry. The participants included many IT giants. Although AMQP is still a draft today, it remains to be seen.

The basic architecture of AMQP is as follows:
Figure 1. AMQP System Architecture

In the AMQP model, the Message producer sends the Message to Exchange. Exchange is responsible for switching/routing and correctly forwards the Message to the corresponding Queue. The Consumer of the message reads the message from the Queue.

This process is asynchronous. Producer and Consumer are not directly connected or even do not know the existence of each other.

How does Exchange perform routing?

This relies on the Routing Key. Each message has a routing Key, and each Queue can use a Binding to notify Exchange of the Routing Key that you are interested in, in this way, Exchange can correctly forward messages to the corresponding Queue. The following table lists the definitions of these key concepts.
Table 1. Concepts of AMQP

Concept	Description
Producer	A program that writes messages toExchange. To do this, the program creates a message, fills the message with content, gives the messageRouting Key, And sends the message toExchange.
Routing Key	A string thatExchangeCan use to determine to whichQueuesThe message shoshould be delivered.
Exchange	Accepts messages fromProducersAnd routes themQueuesIf the message meets the criteria expressed in a binding.
Binding	Defines the relationship betweenExchangeAndQueue, Specifying which messages shocould be routed to a givenQueue
Queue	Holds messages and delivers them toConsumersThat subscribe toQueue.
Consumer	A program that reads messages fromQueue.ConsumerCan create, subscribe to, share, use, or destroyQueueAnd theirBindings(As long as it has have permission to do so ).

To support various common communication models, AMQP defines different Exchange types, as shown in the following table:
Table 2. Exchange types defined by AMQP

Exchange type	Routing Behavior
Fan-Out	Messages are routed to every Queue bound to the Exchange, ignoring the Routing Key
Direct	A message is routed only ifQueue's Binding KeyIs the same as the message'sRouting Key
Topic	Similar to a Direct Exchange, but it supports multipart keys that contain multiple words separated by ". "delimiter; for instance, a message Producer can create messages with Routing Keys like usa. news, usa. weather, europe. news, and europe. weather.

AMQP is still a draft, and the latest version is 0.10.

QPID is an open-source project of Apache Foundation and an AMQP implementation. It provides C ++ and Java brokers and supports clients in multiple languages. It also includes a configuration tool set.

In addition to fully implementing the basic functions of AMQP, Qpid also provides some additional features:

• Corosync is used to ensure the Fault-tolerant feature in the cluster environment.
• Supports XML-type Exchange. When the Message format is XML, you can use Xquery to filter messages.
• Supports plugin, allowing you to easily add new features, such as the new exchange type.
• Security authentication is provided. Any producer/consumer that needs to communicate with the broker must provide identity authentication to prevent malicious irrelevant programs from entering the message system. The security authentication of QPID uses the SSL protocol.

Use Qpid for programming

Currently, the latest version of Qpid is 0.10. Since version 0.6, The Qpid programming interface has changed a lot. Although the previous programming interface continues to support it, it is outdated. Therefore, this article will go over the 0.5 API and introduce the Qpid Messaging API directly.

First, we need to build an experimental environment.

Install

I was overwhelmed when I first saw the Qpid download homepage. In the past, when I needed to try out an open-source software, I often saw a tar package on its download page. At most, it was divided into several tar packages based on the target operating system, download the corresponding one. However, the Qpid download page is somewhat confusing to me. In fact, there are many tar packages classified by programming languages, so I don't know which one to download...

Now it seems a bit clear that Qpid is a message-oriented middleware, which is divided into two parts: broker and client library.

Let's take a look at the Client library. Different programming languages, such as Python and Ruby, require their separate client packages. This is easy to understand.

However, broker and Qpid have two implementations: C ++ and Java.

It is easier to select the Client. You can select the language you are familiar. However, choosing a Java Broker or a C ++ broker will make it easy for beginners to choose KFC or McDonald's.

Java and C ++ brokers have their own advantages and disadvantages. Most of the features are the same, but they are also different. For example, Java supports more AMQP versions, while C ++ supports RDMA. The QPID community will gradually eliminate these differences, but just as Coca-Cola and Pepsi share the same world, these two brokers will eventually have their own fans. Gordon Sim, a senior developer in the Qpid community, answers a beginner's post which should be helpful: http://fossplanet.com/f13/re-why-use-c-broker-versus-java-broker-71322/

I do not know much about java and prefer C ++ and python. Therefore, I plan to use C ++ broker and C ++ and Python as the client.

Download the Qpid broker installation package, decompress, compile, and install it:

tar xzvf ./configure make make install

 

After the above steps, we already have the C ++ broker and C ++ client library. If you want to write an application in Python, you also need to download the tar package of the pyhonn client.

Fortunately, Python does not need to be compiled, so the so-called installation is just to set some environment variables.

If you download the full package, I am afraid you will have to pay a weekly discount. This package contains everything (brokers and clients in various languages). Although there are also C ++ brokers, it is actually different from the C ++ package downloaded separately. To the cpp directory, you cannot see the configure executable file. You must first run bootstrap to initialize autotools.

After Bootstrap, the conditions required by autoconfig are ready, followed by several general installation commands, which are as follows:

./bootstrap ./configure make make install

 

We hope your installation will go smoothly.

Start broker

The simplest startup method is

qpidd --port=60302 --no-data-dir --auth=no

 

Three main options are specified for the preceding startup method.

The -- port parameter specifies the port number of the qpidd listener. The default value is 5672.

-- No-data-dir indicates that you do not need to specify a directory for data persistence. You can also specify a directory to store persistent messages.

-- Auth = no indicates that qpidd does not perform security verification on all connection requests.

Other startup parameters are not described in detail here. You can read help information on your own.

 

Manage qpid

By default, some Exchange (switches) are automatically created after the Broker is started, corresponding to the standard exchange types defined by AMQP. They are called

• Amp. topic
• Amp. direct
• Amp. fanout

Applications can create a queue and bind it to these default exchange for sending and receiving information. However, in real application environments, people often need more exchange, queue, and binding to meet various needs. Or in some complex networks, you also need to configure the broker Federation, that is, a broker network that routes each other.

All of these require various broker configurations, such as adding new exchange, queue, and broker routing. These require various management tools provided by Qpid. In addition to configuration management, the Qpid management tool also provides monitoring functions. There are three common tools:

• Qpid-config
• Qpid-route
• Qpid-tool

Qpid-config is used to add/delete Queue, add/delete Exchange, and other configurations on the broker. Qpid-route is used to configure broker Federation; qpid-tool is used to monitor Qpid Broker in real time. Listing their help information is meaningless to readers. I will demonstrate some of their usage in the following examples.

For more details, please continue to read the highlights on the next page:

1 2 3 4 Next Page