Architecture Design: Inter-system Communication (34)-The deified ESB (UP)

1. Overview

Starting with this article, we will spend one to two chapters on the basic concepts of ESB (Enterprise Service Bus) technology to clarify the terminology of multiple and ESB technologies for readers. We will also explain to the reader in which case the ESB technology should be used. Next, in order to deepen the reader's intuitive understanding of ESB technology, we will use Apache Camel to build a service implementation of ESB technology, although this example does not bring all the functional modules of the current mainstream ESB service implementation to Paul in, At the very least, the reader can see how the core services of ESB technology work in its entirety.

2. Why need esb2-1, ESB and Soa2-1-1, SOA

SOA (service-oriented Architecture) Chinese full name "Service Oriented Architecture". In today's technical environment (2015/2016), SOA is not a new concept. But in the early 2000 when SOA became popular (the concept of SOA was originally presented by Gartner Group in 1996), this architectural model is very popular. This section attempts to introduce some of the core elements of the SOA concept to your readers using the most level of language.

First, SOA is an architectural pattern of ideas, mainly around how multiple "services" are integrated to achieve some purpose of discussion . So what does the service defined in SOA mean? in the business system is published for the user to use, can complete a complete business process function, is the service.

• Services focus on the complete business

  From the above definition of "service" can be seen, the definition of the service object is the business system of the full business functions. For example, the "Confirm order" function in the e-commerce system is a service, and the "Monthly fee settlement" function of the billing system is one of the services, but in the CRM system, the "User login" action that is required to complete the "work Order generation" function is not the definition of the service, because the user logs in to complete the Ticket generation "function, which is not logged in for" sign in ".

• Although the granularity of service is relatively extensive, but controllable, the goal is to reuse

  Next, the "User login" feature also satisfies the definition of "service" in some cases: when the User Center system provides a unified "login" service for all other business systems. service granularity is completely based on business process definition and analysis , so the granularity of service is relatively extensive. Just like the example in the last paragraph of the "Fee settlement" service: the possibility of completing the "Fee settlement" function, within the billing system needs to complete the user identity confirmation, the last month fee query, package list query, and so on the cost breakdown generation process. But none of these processes will be used by any other business system alone , which means that it is meaningless for other business systems to publish these processes separately.

  In the event of a subsequent business change/Business re-compilation, the business designer discovers that a business system needs to use the billing system's "cost query" function separately , and the billing system will need to publish this function as an independent service to the third-party business system. At this point, this function satisfies the definition of "service".

• The purpose of integration is to form a new service

  Integration of business services within or between enterprises is called Atomic services , and the purpose of integration is to reuse these atomic services to form a new service. This ensures that the technical team/business team can use the existing services at the lowest possible cost, with the highest efficiency , but also with higher requirements for the software that implements the SOA architecture idea.

• SOA needs to ensure shielding details

  Using SOA architecture ideas to build integrated relationships across multiple business systems requires that each business system be shielded in detail. These details include technical details and business details. From a technical point of view, regardless of which development language the business system uses, which external transport protocol, which message format can be integrated using SOA, and can be implemented in the SOA architecture of the implementation of different transport protocol transformation and different message format conversion; From the business detail level, SOA needs to mask the details of the functional steps of the business system. This means that a third-party system needs to know that invoking a service can achieve its business purpose, and that the business system that provides the service does not need to be concerned about how the business process is implemented.

• SOA keeps business systems loose

  The SOA architecture model provides a good idea for loosely integrating multiple business systems: by masking the technical details and business details of each business system, compatible with different transport protocols and different message formats for each business system, you can keep the various business systems that integrate SOA into a low-coupling state. This is because all protocols and message formats are open, and business integration does not require additional conversion work for individual business systems, and does not even require any additional work for SOA-based business integration, or the presence of the other system.

If your enterprise or customer's business system has not yet reached a high level of complexity, it is not recommended that you immediately use the SOA architecture model for business integration. Because the various implementations of the SOA architecture model now have some complexity.

2-1-2, ESB

The full name of the ESB (Enterprise Service Bus): Business Services Bus is a way of realizing the idea of SOA architecture. Since it is a way of thinking, there is a concrete consideration of this realization, and the actual environment in which the problem needs to be solved:

The information construction of enterprise generally experiences a long time development, less is 5, 6 years is 10几 year. So we see the most likely scenario for a large enterprise's information system: There are multiple business systems, and even the functional responsibilities of each business system overlap. These systems employ different times of programming languages, programming frameworks, communication protocols, message formats, and storage scenarios.

For example, the billing system may be written in C + +, the external invocation of the function of CORBA, the long-time CRM system using Delphi to write, the same use CORBA publishing call function, and the last two years the enterprise has just a CRM system using the C # language for an upgrade, However, due to the design of the data storage layer, and did not cut all the old system data to the new system, so the current two sets of CRM systems are in use, the latest development of the financial linkage system, in the Java language development, and no longer use the application window, instead of using the browser for page display and user action. This financial linkage system most external services use the HTTP protocol to publish, there are some services using thrift RPC External announcement ...

Thus, due to a variety of visible and invisible reasons, the construction of enterprise information system history and reality is often complex. If these systems require service integration, but do not have a mature, stable, and easy-to-use middleware to reconcile, it is virtually impossible to achieve the above invocation requirements (even if the implementation is quite difficult to maintain and extend).

The ESB bus technology needs at least helping these business systems to do the following, in order to meet the design essentials of the SOA architecture idea and achieve the desired goals:

• Multi-call protocol support and conversion

The ESB technology allows for compatibility conversions regardless of which invocation protocol is used by the business system to externally advertise the service. For example, the service of a business system only accepts calls in the form of Web service soap, and the services of the B business system can be invoked using thrift RPC (it is not necessary to adapt the Protocol of a business system specifically to call a business system). The middle tier based on the ESB service helps implement the conversion of the two protocols.

• Multi-message format support and conversion

Regardless of which message description format is carried by the calling protocol, it is possible to convert to each other through the ESB middle tier. The ESB middle tier should support translating the information description in JSON format into an XML format that the target business system can recognize, or convert the XML description format to plain text format, or implement the conversion of two different XML formats, and so on ...

• Service Monitoring Management (registration, security, version, priority)

Now that the ESB is integrating the atomic services, there are more questions to consider. First, the services provided by the business system may change over a period of time, such as periodic upgrades, and runaway business systems may even present a completely non-warning, irregular service change, such as abrupt data cuts that lead to service interface changes. There should be a set of features in the ESB implementation software that can guarantee that the integration service will still work in this case. Second, not all services provided by the business system can be integrated in the ESB, and not all services can be orchestrated by any routing rule. The ESB should have a complete set of capabilities to guarantee the security and permissions of the service integration.

As an integrated business system, how the ESB integrates the atomic services it provides, the former does not need to be concerned about,

• Service Integration and orchestration

In order to integrate multiple services through ESB technology to form a new service, ESB technology must be able to orchestrate services. The role of service orchestration is to clarify the sequencing of atomic service execution, to determine the conditions for atomic service execution, and to ensure that the integrated service can work as required by business designers. Example of the new service "Ticket distribution" is the process of working on the ESB bus in accordance with the set execution conditions of the atomic service provided by multiple business systems:

In fact, the ESB technology and the service governance techniques mentioned earlier in this topic are at the same level of architecture: they are the realization of the idea of SOA. But the application scenario and the focus of the two are completely different.

2-2. ESB is the evolution of EAI

ESB Enterprise Service Bus technology occurs after the SOA architecture, before which the most used technical ideas for integrating multiple systems are EAI (Enterprise application integration): Integration of enterprises applications. EAI technology does not have a unified standard, but a unified approach to different enterprise integration business systems.

Characteristics of 2-2-1 and EAI

As can be seen, the main function of EAI is to complete the conversion of the message format. Since the main use of EAI is in the 890 's, it is also very limited if you look back from now on for the transport Protocol supported by EAI (though it is certainly based on the 7 layer/5 Layer network protocol). However, prior to the emergence of SOA architecture, EAI technology does provide a feasible way for enterprises to integrate business systems.

It is important to note that EAI technology is not an implementation of the SOA architecture idea. It comes before the idea of SOA architecture, and most importantly, it lacks the basic elements of SOA-business services, granular but controllable: because there is no process orchestration in EAI, these atomic services are not organically combined to form new services or to re-comb business processes in EAI. So that the atomic service is required to do the appropriate granularity splitting.

2-2-2, which features have evolved

• Evolution on message conversions:

As mentioned in the previous section, because EAI technology has been in the early days, there is basically not much industry-standard transport protocols and message formats at that time, with most of the XML format and semi-structured text data. Therefore, in the company to implement EAI technology, generally do not consider too many industry standards, using the company's internal custom-made transport protocol and message format can be done. While this has some benefits, the company's internal business teams are aware of the business significance of these custom-made formats, reducing the amount of communication costs. But the problem with this is obvious, and if you need to integrate with your brother's business systems in the future, the time saved will be wasted.

There is no such consideration in mature ESB products, generally with an open transport protocol and message format. For example, using the HTTP transport protocol to carry query requests, using the FTP transport protocol to carry uploaded file information, using the XMPP message format to describe IM content, using the MQTT message format to describe the IoT device capture content, using the AMQP message format to describe MQ content.

• Evolution in the process orchestration

EAI has no rigid requirements for process orchestration, meaning that he only targets the data conversion process and does not target business services. So readers can look at this: after the idea of SOA architecture emerged, the great technical dick-Silk team immediately discovered the constructive role of service-oriented concepts in EAI software, and neglected to add a variety of service-oriented elements to various already-running EAI software. The key is to join service-oriented management functions such as service orchestration. In fact, the above situation is the most real situation in reality.

• Evolution in service Management

From EAI to ESB is actually an optimization of the business management approach, but as mentioned above, not all enterprises are suitable for using ESB technology based on SOA architecture ideas. At present, the integration of the internal business systems of most enterprises is more appropriate to the definition of EAI technology, because these enterprises business systems have not yet reached a high degree of complexity (most of the situation is that they only have a necessary financial system).

Therefore,EAI is not an obsolete product, nor is an ESB a great invention of "trans-era". The latter is the former continuous improvement of the product, the relationship between the two as "basic version" and "Upgrade version" more appropriate . It is not true that some web articles belittle the EAI's ability to elevate the ESB, and that there are enterprises or platforms that are not so complex as to have to use an ESB, without using ESB technology. so only the right architecture is the ideal architecture. If it is a soft article of an ESB vendor, the purpose is to benevolent see.

2-3. ESB and Cyclic dependence

ESB technology guarantees the low coupling of each business service, and indirectly avoids the problem of service circular dependency that the technical team intentionally or unintentionally manufactures in the integration of each business system.

2-3-1, what is cyclic dependence

First of all, we need to clarify what is cyclic dependence, and cyclic dependencies on the programming level, software PRODUCT design level, the top level of architecture design may appear on the scene. From a conceptual model, as long as two or more elements produce interdependencies, they can be seen as generating cyclic dependencies:

Is the correct example of two dependencies: a element's normal work depends on the normal operation of the B element, or the normal work of a element depends on the normal work of the B, C, D elements. Here the A, B, C, D four elements can refer to the four-segment code, can also refer to a business system of four functional modules, but also refers to the top-level architecture design of the 4 independent operating business systems.

A circular dependency is a circular graph at a logical level . The two wrong instances of dependency are shown: A element's normal work depends on the B element working properly, and the B element is dependent on the normal work of a element. So which element can work first (chicken or egg first)? The diagram on the right shows the cyclic dependency of three elements, a element depends on the C element, the C element relies on the D element, and the D element is dependent on the a element. So which elements a, C, D three elements is the underlying element?

• Cyclic dependencies at the code level:

The code-level loop dependency is the most prone to coding errors for developers, but sometimes it's not all about developers, after all, business designers may have problems from the requirements understanding stage. The following example provides a circular dependency at the code level:

/** * This class relies on BUSINESSB * @author Yinwenjie */ Public  class Businessa {    PrivateBUSINESSB BB; Public Businessa(BUSINESSB BB) { This. bb = BB; }    ......}/** * This class relies on BUSINESSC * @author Yinwenjie */ Public  class businessb {    PrivateBUSINESSC BC; Public BUSINESSB(BUSINESSC BC) { This. bc = BC; }    ......}/** * This class relies on Businessa * @author Yinwenjie */ Public  class businessc {    PrivateBusinessa AC; Public BUSINESSC(Businessa AC) { This. ac = AC; }    ......//Next we try to instantiate Businessa ...     Public Static void Main(string[] args) {//How to instantiate Businessa?         //New Businessa (New BUSINESSB (New BUSINESSC (New Businessa (! The programmer has gone insane! ))))}}

In fact, in accordance with such reference structure and constructor requirements, instantiation of Businessa This thing is never done .

• Functional-level cyclic dependencies:

There may also be cyclic dependencies between the functions of a business system. Circular dependencies at the functional module level can affect the design quality of a business system, relative to the code level. The author has been involved in a software, the client has raised such a business need:

When creating a new "departure order" in a freight system, you must select a free driver and a free lorry (of course, the type of truck is to be judged). The idle driver and the idle van lack any of the same to complete the creation of the "departure list". At the same time, in order to record a corresponding "departure" of a lorry, the customer requests that only after the new departure order can be created, the truck can remove the previous "departure" binding relationship and become a "free van".

So the question is, if only after the completion of the new "order" created, the truck can be lifted and the previous "order" binding relationship, then the new "order" when the "free Truck" from where? In fact, customers do not know the technology, is that we have encountered in the demand research phase of a problem, but the key to see the need for personnel from which to start to explain to the user Guide users to analyze the requirements logic . Not necessarily in the technical language to tell the user directly, his requirements on the technical level is illogical.

• Circular dependencies at the architectural level:

When multiple business systems are integrated, they may also have cyclic dependencies. In particular, the greater the number of business systems involved in integration, the more likely this cycle-dependent situation will be:

When the number of systems has not reached a certain number (usually this threshold is 4), the cycle dependence between systems is most likely caused by business personnel/technicians unintentionally. At this point, the dependencies between systems are still at a manageable level, and the technical team/business team can quickly correct them, even if there is a cycle-dependent relationship between the systems. However, when the number of business systems participating in the integration exceeds the number of controllable thresholds, this inspection and correction work is no longer a man-made range . As shown in the integration of the 5 systems, it is easy to see the cycle dependencies between systems:

2-3-3, avoid cyclic dependence

• Dependency inversion principle to prevent cyclic dependence

The dependency inversion principle can help prevent circular dependencies at the code level and at the functional module level. The problem of circular dependencies at the top level of the architecture can also be avoided by following this principle. The principle of dependency inversion is described in detail in many books and web materials. Basically the principle is to say two points: high-level modules should not be dependent on the lower levels of the module, the implementation of the module should rely on abstraction (interface), and abstraction (interface) can mask the details of the design of the function.

• Automatic detection of cyclic dependencies

If the product you are responsible for is a legacy product. After a number of design staff turnover, the product internal design more or less there are some circular dependency problem, what should be done? The first thing you need to do is to check which modules of the product have cyclic dependencies and then think about how to modify them. There are many such tools on the market that can help you check the code level and the system module level dependencies are good, here I recommend Sonarqube. The following is the result of a project that I have experienced in which a submodule was tested by Sonarqube:

Hehe, V0.0.9 version, it seems that there are a number of classes of the complexity of the high, the module used in the design pattern has improved space. OK, this subsystem just opened a head, has now developed to V0.5.3 version number, the code line number is nearly 50,000 line. However, as one of the authors of this sub-module, I think that the package coupling degree has been well controlled, until now there is no packet cycle dependency situation.

• Pumping from the bottom leads the business people to optimize the dependency structure

In the previous section, "Functional-level circular dependence" We raised a driver-truck-to-order three elements in the business functional level of the circular dependency problem. Now let's continue the discussion. The reason why the customer wants to create the new "order" is to release the truck and the history of the "departure order" binding relationship, because the user is concerned about the loss of the software to the history of the "departure list" tracking ability , and finally unable to statistics vehicle usage or driver performance situation. But in fact, customers do not have to worry about this situation, to understand the real idea of the customer, the need to lead the customer to open a new log module, the log module is in the entire business system design of the lower level, specifically tracking a variety of historical behavior, vehicle, personnel, financial, goods, is OK:

We can solve the cyclic dependency problem by using the method of stripping cyclic dependency on the underlying capability of the element. This solution is not only applicable to the decoupling of business system functions, but also to the code level or the decoupling at the top level of the system.

• Isolate dependent elements using the middle tier/base layer

The ESB provides an ideal middle tier/base layer for the integration of individual business systems . By extracting the underlying capabilities from the middle-tier/base layer, we put the integration of all business systems into the ESB:

Note: Although the ESB undertook the business integration work that was originally done inside the system. However, depending on the dependency inversion design principle, the ESB is also dependent on the calling interface of each business system registered on the ESB bus, and the specific implementation of the functionality in the business system is transparent to the ESB.

==========================================
(next)

Architecture Design: Inter-system Communication (34)-The deified ESB (UP)