How do I get a message? Using JMS technology as a solution to data replication

Background

When thinking about messaging solutions, you might think of a system that integrates two different applications through a remote message invocation mechanism. In general, this coupling is often used for distributed entities that do not communicate frequently and for data transfer volumes. A more classic example is a homogeneous interface to a heterogeneous backend and portal that assigns the backend processing of a user request and then formats those requests for the end user representation.

Public threads in messaging methods have always assumed that although messaging solutions provide robust, highly available communication between systems, they are basically inefficient and are used only as a last resort when it is not possible to avoid communicating with external systems. This view of message passing occurs when a remote method call (RMC) is present until a more modern messaging solution like CORBA and DCOM is present, and the message delivery that is usually applied is limited to solving several types of problems.

Goal

Over the past decade, there has been a deeper understanding of the needs of distributed systems. Emerging Technologies (like Java and. NET) already contains code distributions as part of their basic programming model. By doing so, these technologies have incorporated high availability and fault tolerance into messaging, while encouraging vendors offering solutions to deliver systems that consider performance on a wider range of issues.

Our company has recently been asked to implement a solution for file distribution and replication, which requires a custom system that integrates secure FTP, database replication, and other one-time solutions. Instead of blindly following the path of custom development, we studied the possibility of applying the latest messaging solution to the problem. We found that JMS not only provides the necessary infrastructure for information delivery, but it also handles all of the infrastructure issues that our customers require for quality of service, security, reliability, and performance. This article describes the challenges our team faces and how JMS (in MQSeries form) allows us to meet and exceed the requirements of our customers.

Problem

Our customers face a major distributed data challenge, with many call centers across the country, and in call centers across the country, operators want to record interactions with customers. These records must be indexed and archived quickly and reliably in remote data centers. Indexed and archived stored procedures do not affect the operator's system record and the ability to store information that the operator is interacting with the customer. The customer already has a system that contains the combined code, VPN, and other technologies. However, the existing solution is far from the goal of performance and reliability, and it is a poor technology that is difficult to understand and costly to maintain.

When developing alternative customer systems, we considered JMS and a variety of non-JMS solutions, especially those based on FTP and secure Replication (SCP). However, non-JMS solutions have two major drawbacks:

They are at a disadvantage in terms of security flaws.

The infrastructure they provide applies only to the actual data transfer, but requires custom development to address issues such as reliability, fault tolerance, security, platform independence, and performance optimization.

Our team finally concluded that the development work needed to add these additional features was daunting, so we decided to choose a JMS solution that could get rid of these problems.

Solution

We have developed a JMS based system that:

Provide a reliable archive of recorded multimedia files

Supports scalability to enable multiple data centers to receive files

Support for archiving of other data types

The file we are discussing here is larger than the data that was passed in the project that involved messaging solutions (50k-500k). Our first task is to ensure that the data size does not affect the JMS solution. By testing the performance of the system to deliver a variety of message payloads of all sizes, we evaluated many JMS solutions, including IBM MQSeries. The results show that a message with a size of 1 trillion will not have a significant impact on the overall system performance after proper configuration. Because common sense holds that messaging solutions apply only to regular, small payloads, our results are a major discovery. We continue to analyze the architecture of the system (this architecture is outlined in Figure 1), which provides the security, high availability, and reliability that customers need.

Figure 1. Advanced System Architecture

The existing infrastructure has a system on each client, and when the operator interacts with the user, it creates a multimedia file as a response. In addition, these files need to be archived. Our system starts a process (running on each machine) and looks up these files in a known directory. When new files are detected, the process packs them into a JMS payload and sends them to a JMS server in one of the data centers for delivery. Once the JMS server confirms receipt, the files in the sender are dropped. The JMS server transmits the data to an available handler in the datacenter for archiving.