Open Source Message middleware DOTNETMQ

Since this open source project is very helpful for my middleware rookie, I translate the official documentation as follows:

Introduction

In this article, I'll introduce a new and independent Open Source Message Queue system that is entirely built I N C # and. NET Framework 3.5. DOTNETMQ is a message broker this has several features including guaranteed delivering, routing, load Balancing, Server graphs ... so on. I'll start by explaining messaging concepts and the need for message brokers. Then I'll examine what dotnetmq are and how to use it.

In this article, I will introduce a new open source Message Queuing system that is developed entirely based on the C # and. Net Framework 3.5 Framework. DOTNETMQ is a message broker with multiple features including delivery assurance, routing, load Balancing, server diagrams, and more. I'll start with parsing the message concept and the necessity of the message agent, and then I'll examine what DOTNETMQ is and how to use it.

What is Messaging?

Messaging is a- asynchronous communication of applications running on same or different machines With reliable delivery. Programs communicate by sending packets of the data called messages to all other [1].

Messaging is an asynchronous communication program that runs on the same or different machines with reliable transmissions. Programs communicate by sending packets called messages to other programs.

A message is a string, a byte array, an object ... etc. Typically, a sender (producer) program creates a message and pushes it to a message queue and a receiver (con Sumer) program gets the message from the queue and processes it. The sender and receiver programs don’t have a to being running at the same time, since messaging are an asynchronous process. This is called loosely coupled communication.

A message can be a string, a byte array, an object, and so on. Typically, a message sender (producer) program creates a message and inserts it into the message queue, and the message receiver (consumer) program takes the message out of the message queue and uses the message. The send and receive programs must run concurrently, so that the message tradition is an asynchronous process. This is known as loosely coupled communication.

On the other hand, a Web Service method call (Remote method invocation) is a type of tightly coupled and sync Hronous Communication (both applications has to be running and available during the whole communication; if the Web Service is offline or a error occurs during the method call and the client application gets an exception).

On the other hand, a call to a Web service method is a tightly coupled and synchronous communication (the sending side, the receiving end program and the Web service must run simultaneously throughout the communication process and be available if the Web service is offline or an error occurs during the function call) , the client program gets an exception).

In the figure above, and the applications communicate over a message queue in a loosely coupled manner. If the receiver consumes messages slower than the sender produces it, the message count on the queue would increase. Also, the receiver may be offline and the sender is sending messages. In this situation, the receiver gets the messages from the queue when it becomes online (when it starts and joins the Queu e).

message Queues is typically provided by Message Brokers. A message broker (message broker) is a standalone application (service), that other applications connect to and Send/receive Messages. A Message Broker is responsible to store messages until a receiver receives them. A message Broker can route messages (message routing) across machines to deliver a message to the destination application and Can try delivering the message until the receiver correctly handles it. A message Broker is sometimes called a message oriented middleware (MOM message oriented middleware ) or simply message Q Ueue (MQ).

What is DOTNETMQ?

DOTNETMQ is an open source Message Broker This has several features:

• persistent or non-persistent messaging. Persistent or non-persistent message delivery.
• Guaranteed Delivery of persistent messages even in a system crash. Guaranteed delivery of persistent information even when the system crashes.
• Automatic and manual routing of messages in a custom machine graph. Automatic or manual message routing on a custom machine diagram.
• Supports Multiple Databases (MS sql Server, MySQL, SQLite, and memory-based storage for now). Support multiple databases (MS SQL Server, MYSQL, SQLite, and other currently memory-based databases).
• Supports don’t Store, direct send style messaging. Supports storage sending or sending two kinds of message delivery directly.
• Supports request/reply style messaging. Support for message delivery in Request/reply mode.
• Easy-to-use-client library to communicate with the DOTNETMQ Message Broker. Easy to use the client library to communicate with the DOTNETMQ message agent.
• Built-in framework to easily construct RMI services upon message queues.
• Supports delivering messages to ASP. NET Web Services.
• gui-based Management and monitoring tool.
• Easy-to-install, manage, and use.
• Written entirely in C # (using. NET Framework 3.5).

I preferred to the name Dotnetmq as MDS (Message Delivery System) when first creating it. Because It is designed isn't just to being a message queue, but also as a system that delivers messages directly to application S and an environment The provides a framework to build application services. I called it DOTNETMQ since it is entirely developed using. NET and the DOTNETMQ name are more memorable. So, it's original name (and internal project name) is MDS and the applications has many classes with the prefix mds .

Why a New Message Broker? The need for a Message Broker

First, I'll demonstrate a simple situation where a message broker is needed.

In my experiences in business life, I ' ve observed really bad and uncommon asynchronous enterprise application integration Solutions. Usually there is an application that runs in a server and performs some tasks and produces data, and then sends the result Data to another application on another server. The second application performs other tasks on the data or evaluates the result (the servers is on the same network or CO nnected over the Internet). Also, the message data must be persistent. Even if the remote application is isn't working or the network is not available, the message must being delivered on the fi RST Chance.

Let's look at the design of the figure below.

Application-1  and  Application-2  are executable Applications ( or Windows services) and  Sender service  is a Windows Service. Application-1 performs some task, produces data, and calls a  Remote Web Service  method On&nbs P Server-b  to transmit data. This Web Service inserts data into a  database table . Application-2 periodically checks the table for new incoming data rows and processes them (and deletes them from the tab Le or marks them as processed to not process the same data again).

If an error occurs during the Web service call or while processing data in the Web service, data must not being lost and must be sent later. However, Application-1 has the other tasks to does, so it can not be try to send data again and again. It simply inserts data into a database table. Another Windows service (or a thread in Application-1, if the application is runs) checks this table period Ically and tries to send data to the WEB Service until data is successfully sent.

This scenario is really reliable (messages was guaranteed to being delivered) but isn't an efficient of the Tween, applications. This solution has some very critical problems:

• It takes a long time to develop (to code).
• Individual coding for all message types (or remote method calls). For a new Web Service method call, you must change all the services, applications, and database tables.
• Almost same software and structures must is developed (or copied and modified) for every similar service.
• Testing and maintenance of too many services/applications/databases after coding.
• Some applications and services periodically check the database even if there is no new message (if the database was not wel l Indexed and optimized, this may consume serious system resources).

　　Message Brokers do all this job and takes all the responsibility to deliver messages to the remote application in The most efficient. The same application integration using DOTNETMQ is shown in the figure below.

DOTNETMQ is a standalone Windows service this runs on both server-a and server-b. Thus, you just need-to-write code to communicate with DOTNETMQ. Using the DOTNETMQ Client Library, it is very easy and fast to connect and send/receive messages To/from the DOTNETMQ serv Ice. Application-1 Prepares the message, sets the destination, and passes the message to the DOTNETMQ Broker. DOTNETMQ brokers would deliver the message toApplication-2 in the most efficient and fastest the.

What's about Existing Message Brokers

It is clear to see that there are a need for Message Brokers to integrate applications. I searched the web, and read books to find a free (and Open Source, if available) Message Broker . NET. Let's talk about what I found:

• Apache ActiveMQ(http://activemq.apache.org): It is Open Source and implementsJMS(Java Message Service is a standard API for messaging in the Java World). It has also a. NET client library. I read a complete book ' ActiveMQ in Action ' to learn more and I developed some simple applications. Even though I read the book, I do not see a easy and reliable-to-construct an ActiveMQServer GraphThat worked together and routed messages. I also did not see a by-set the destination server for a message. It routes messages automatically but I can not control the routing efficiently. I understood that it's commonly used withApache Camel(http://camel.apache.org) To achieve common application integration patterns. Apache Camel is also another world to discover, and even worse, it's just for Java. Finally, I think that it isn't simple enough to use and especially to configure, monitor, and manage it. So I gave up working on ActiveMQ.
• MSMQ   (http://msdn.microsoft.com/en-us/library/ms711472 (vs.85). aspx): This is a solution from  Microsoft and it is the most suitable framework for use with. NET applications. It's easy-to-use and learn, and it had tools to monitor queues and messages. It's especially very suitable for asynchronous communication of applications that's running on the same machine or can Directly connect to the same machine. But I could don't find a built-in solution to construct a graph of the MSMQ servers that route messages. Since routing is my first start point, I eliminated the this Broker.
• RabbitMQ (http://www.rabbitmq.com): It is developed using the Erlang programming platform (that's developed by Ericss on). You need to install Erlang first. I spent a lot of time to install, configure, and write a sample application. It had a. NET client but I got many errors when trying to develop and run a simple application. It was very hard-to-install and to-make the RabbitMQ Brokers work together on the different servers. After a-few days, I-gave up because I thought it must is not the hard-to-learn and to start developing applications.
• OPENAMQ (http://www.openamq.org), ZeroMQ (http://www.zeromq.org): I examined thesebrokers overall but I found That I can not easily does what I want to using. NET.
• Others: I also found a few other projects but they has important features missing like routing, persistent Messa Ging, request/reply messaging ... etc.

You see that there are no Message Broker that's developed entirely in. NET in the list above.

From a user perspective, I just want-pass "message data, destination server, and application name" To my l OcalBroker. I am not interested in the rest. It would route a message over the network as many times it requires and delivers the message to my destination application On the destination server. My Messaging system must provide this simplicity for me. This is my first start point and I evaluated a Message Brokers according to the point. The figure below shows what I want.

Application-1 passes a message to message Broker in the local server (server-a):

• Destination server: server-d
• Destination application: Application-2
• Message data: Application Specific data

server-a has no direct connection to server-d. The message Brokers forward the message over the servers (the message is transmitted through SERVER-A, Server-b, Se Rver-c, and server-d sequentially) and the message finally reaches the message Broker in server-d to deliver the MES Sage to Application-2. Note that there was another instance of Application-2 running on SERVER-E, but it does not receive this message, since The destination server of the message is server-d.

DOTNETMQ provides this functionality and simplicity. It finds the best (shortest) path from the source server to the destination server on the graph and forwards the Message.

Open Source Message middleware DOTNETMQ