Suggestions on ESB implementation

Document directory

• Leverage ESB to make peripheral applications
• Use ESB to implement business processes
• Use ESB for Big Data Transmission
• Manage services
• Service-oriented complex dynamic routing rules
• Try to use XSLT to implement the conversion logic.

Http://www.infoq.com/cn/articles/mgy-esb-implementation-suggestion

Preface

When talking about the Enterprise Service Bus (ESB), developers with experience in implementing Service-Oriented Architecture (SOA) will not be unfamiliar. Over the years, people have been talking about it, and some people think that "Implementing SOA must require ESB", or "we need SOA as long as the ESB is set up ". These arguments have merits and are one-sided. As there is no unified and standard definition of ESB in the industry, one thousand "ESBs" in the eyes of one thousand people become reasonable. However, how can we make good use of ESB? We need to clearly understand the role of an ESB in SOA, and understand what is within the responsibility of an ESB, but what is not. Only by correctly understanding the functions of the ESB and assigning appropriate tasks can we use it in the cutting edge and exert its enormous energy.

This article is arranged as follows:

The first part briefly introduces the role of ESB in SOA.

Part 2: Introduce the misuse of three common ESB.

The third part introduces three recommendation implementations.

The fourth part is the development trend of ESB products.

Statement: all opinions in this article only represent the author's personal opinions, not those of the company where the author works.

Correctly understand the role of ESB in SOA

In all fairness, ESB is indeed a very important integration layer component in SOA, whether it is the SOA reference architecture released by opengroup, several major SOA vendors (refer to IBM's SOA solution implementation through reference architecture, Oracle and F5 SOA reference architecture, and the definition of ESB in Microsoft's BizTalk ESB Toolkit ), both place the ESB at the core of the SOA architecture.

In fact, ESB plays an important role in SOA. It solves the integration problem of SOA at the Technical layer, coupled with the integration logic between applications, making SOA more flexible, it is easier to expand and more agile. With ESB, new service consumer applications do not need to care about where the service provider is, what communication protocols are used, and what data is exchanged with them ......, It only needs to send requests to the ESB using open and standard communication protocols. On the contrary, if a service with a large reusable value is located in a legacy system, and for various reasons, the legacy system cannot be rewritten in the short term, in this case, the ESB can bridge the communication between the service and its users. Of course, the role of ESB is far more than that, and there are many discussions in the industry. This article will not go into detail. You can search for ESB patterns on Google to obtain relevant information.

However, ESB is not the "savior" and it is doomed that it cannot solve all problems arising from application system integration. The principle is very simple. The history of computer development has never seen a product/tool that can meet all application needs. The technology develops rapidly and the demand develops faster, so the technology will never keep up with the demand. In addition, the ESB or ESB product has its specific applicability. It is a concept/product at the infrastructure layer and solves common problems in integration, such as service connectivity, routing, rich messages, Service Registration/search/release, and other service management, service monitoring, and quality assurance. The ESB cannot solve many problems than it can solve. For example, it is not appropriate to orchestrate a manual process. It is extremely difficult to make it provide user interaction like Portal products .......

I have supported many customer projects. In these projects, some customers use the ESB well, while others barely use it. Some use the function very easily, and some use the ESB to do some work that they did not originally do. Here, I only share my experience in implementing the ESB over the years from my personal standpoint. The following lists the implementation methods that I often see but not recommended and some good practices that I have accumulated during the implementation of ESB for your reference. We also welcome criticism and correction.

It is not recommended to attach ESB to external applications.

• Symptom:

  The ESB architect designs a set of standard data interfaces (common XML format) on the ESB, and stipulates the use of unified protocols (such as Web Service/HTTP ). All ESB service consumers and services connected to the ESB must comply with this standard. The purpose is to simplify the development work on the ESB. This is a kind of practice, because in actual situations, leaders may stipulate that "all services must go through the ESB, even passthrough ".

• Analysis:

  Most of the domestic ESB implementers are Si/isV. For cost/manpower or other reasons, there will always be some architects who want to achieve this goal: can I design/implement an ESB intermediate platform once and for all? In the future, no matter which service can be easily connected to the ESB?

  This is a romantic and ideal goal, but it cannot be achieved. There are two reasons: first, the arbitration logic is generally very specific, and specific services have specific integration requirements. For example, a service provider provides a Web Service WS1 Based on HTTP, and we want to expose this service to two clients (clienta and client B) through ESB. Client A uses XML/HTTP to access the service, client B uses soap/JMS to access the service, as shown in figure 1. People with experience in implementing the ESB can see that the arbitration logic is very specific. We need to consider not only the differences between protocols, but also the differences in message formats. Second, if such a design/implementation exists, why doesn't the ESB manufacturer directly implement this feature? You may say that the producer does not understand the specific business, but the specific business is complex, but Si/isV understands the complex business. In fact, the ESB solves more technical tasks. Most of the work at the business layer does not fall into the ESB category. complex business logic is not implemented in the business system, is implemented in other SOA components, such as the process management engine.

  Figure 1. One of the main functions of ESB is to connect heterogeneous protocols and data.

• Causes and hazards:

  The root cause of this practice is: The Role of the ESB itself is not recognized, and the design capability above the ESB is over-valued. In fact, ESB is designed for the integration logic with different functions, and the Integration logic between services is also different. Therefore, the architecture above the ESB does not exist once and for all.

  The harm is that it has lost the most powerful connectivity functions of the ESB itself, and gradually developed components such as the ESB because of the differences between connected applications. If the protocol and data standards of all peripheral applications are required to be unified (in fact, this is impossible, especially when integrated with third-party applications), the necessity of ESB is inferior.

• Suggestion:

  Do not try to implement an architecture above the ESB once and for all. This architecture does not exist and there is no need for it to exist. However, I am not opposed to design. In fact, the ESB can be designed, or a certain degree of automation, flexibility, and versatility can be achieved through a certain design. For example, all requests of a group of services with similar functions must be audited and security reinforced, we can design the corresponding technical components/services on the ESB in advance, and then call the component/service in the middle of the ESB's arbitration flow.

Use ESB to implement business processes

• Symptom:

  Some architects see that ESB supports the service composition mode, and then think that this mode can be used to implement business processes. Therefore, the ESB product has evolved into a BPM product.

• Analysis:

  If you are concerned about the common use mode of ESB, you will find that the service composition of ESB is very similar to the service cherography in BPM, both combine and assemble fine-grained services to form a large-granularity Service or business process.

  In fact, there are essential differences between the two. 1. ESB is a component that tends to be integrated at the Technical layer. For example, after assembling the "Customer Data Query" service and the "log" service, the result is the "Customer Data Query" service, only a new additional feature is inserted in the middle of the arbitration stream, that is, the "log" service. The meaning of service Orchestration in BPM focuses on the concept of business flow. For example, in the "project approval process", the implementation of this process may involve services in several related systems. They may be "project application", followed by "first-level functional experience approval ", then there are "department manager approval" and "financial approval. These services flow to form a complete business process. Second, the Service combination on the ESB is stateless. When the ESB is running, an instance is created for each request to execute its arbitration logic. There is no time sequence relationship between the request and the request, is independent from each other. On the contrary, the Service orchestration on BPM is different. Unfinished Business Process instances will survive in the BPM runtime, and the survival period may be one day, one week, or even one month or longer; there may be some associations between requests, such as the approval process for a project with the same project id, the requests sent by level-1 functional managers, department managers, and finance personnel to the process are for the same runtime instance.

• Causes and hazards:

  Aside from other non-technical factors, an important reason for this practice is that it obfuscated two concepts: ESB service composition and BPM service orchestration.

  Hazard: although it is technically feasible to enable the ESB to implement bpm, especially the long run process, it violates the design philosophy of the ESB product, it will greatly affect the overall operational efficiency of its ESB.

• Suggestion:

  Recognize the differences between technical service combinations and service orchestration, analyze the concept scope of each product, and select the appropriate product to solve the appropriate problems.

Use ESB for Big Data Transmission

• Symptom:

  Use ESB to transmit large amounts of data between two systems, such as video files and large documents. The information, files, and messages transmitted in these scenarios share a common feature: only the connectivity function provided by ESB is used, and other functions, such as message parsing and conversion, are not used, routing.

• Analysis:

  In order to seize a favorable position in the ESB market, the ESB products provided by major manufacturers are becoming more and more powerful. The ESB product is designed to support the transfer of messages from one system to another or several systems. Therefore, some architects have built a data transmission bus using this function provided by the ESB product itself.

  However, ESB is not suitable for this function. Although it can implement this function, it is not the best practice. As an enterprise-level service Unicom and management platform, the service that needs to penetrate the ESB should be the services that may reuse the largest and most valuable in the enterprise, and the application should frequently access such services, therefore, the services that require ESB support at the same time may be very heavy. Therefore, the original intention of the ESB product is to implement a stateless, high-throughput service bus to provide transparent, standard, and open access capabilities for important business services in the industry.

• Causes and hazards:

  The reason for this practice is that the ESB's ability to transmit data is excessively magnified. If a large amount of information is transmitted in the ESB, the overall performance of the ESB may decline, compromising the QoS of other important services.

• Suggestion:

  If the data to be transmitted needs to be parsed or converted, or you need to perform certain routing or other arbitration logic requirements based on the information in the data, we recommend that you use ESB; however, if the message body is too large at the same time, you can consider splitting the messages to be parsed and the data parts that do not need to be parsed, and transmit the unparsed parts through other specialized data transmission platforms, for example, FTP and Database Backup mechanisms. The control flow of the transmission task is implemented in ESB.

Recommended implementation services should be managed

An important function of ESB is to expose the services in the enterprise/partner in an open and standard way of service, so that service consumers can conveniently find the services, to promote service reuse and management. Many ESB products do not include functions such as service registration, storage, publishing, and subscription. These functions include the UDDI library. Therefore, an ESB generally requires a Service Registration/Repository to collaborate with it. The ESB executes various arbitration logic, and the registration database provides the necessary metadata information for the ESB.

The Service Registration library can store service metadata, including service descriptions, functions, versions, Input/Output Message modes, service endpoints, SLA, and security policies, this information can be used by ESB for message format verification, dynamic routing of services, security policy execution, SLA guarantee, and so on.

The ESB can be implemented at multiple levels to implement the entire enterprise-level/national-level ESB, or the department-level/municipal-level ESB. Service consumers should be able to find the services they need in the registration database and obtain the location, service description file, request/corresponding message format, and other information required to call the service. Without a registered database, the increasing number of ESB-Connected Services and the increasing arbitration logic will inevitably lead to confusion in service management. At last, it offsets the value of ESB.

Service-oriented complex dynamic routing rules

Routing is one of the most important arbitration logics in ESB. Routing scenarios are very common. For example, in scenarios where different customers provide different QoS, You need to route the QoS to the service provider with different execution capacities based on the customer type. For example, when a response message arrives at the ESB, always send the response message back to the original service requestor.

Routes can be classified into static routes and dynamic routes. Static Routing means that the route Branch has been clearly defined during the design, and the route Branch selection of dynamic routing is dynamically determined at the runtime. Whether it is static or dynamic routing, the choice of Route branches must be accompanied by one or more decision-making bases. The decision basis can be as simple as an IF-else statement, or it can be complicated to obtain the final route branch through multi-dimensional decision table and multiple judgments.

For complex routing, we recommend that you externalize the logic of the routing rule and service it, as shown in figure 2. In this way, the arbitration logic can first access the routing rule service before sending and distributing messages to obtain clear routing results. So far, some may ask how to define "complicated" and "simple. There is no uniform rule for this problem. Based on my previous implementation experience, if one of the following conditions is met, it can be considered as a complex routing rule:

1. Determine whether the dimension is greater than or equal to 2
2. Continuous decision nodes greater than or equal to 3
3. Route branches may increase or decrease at any time

Figure 2. Service-based Dynamic Routing rules.

The benefits of servitization of complex dynamic routing rules are simple and clear. First, the service of complicated rules can reduce the variability of the arbitration logic. When a rule changes, you do not need to modify the arbitration logic. Instead, you only need to change the rule service. Further, if the rule service is implemented by a professional rule engine, the change will be simpler. Second, because the Implementation and Design of ESB products do not have industry standards, configuration development based on ESB products cannot be transplanted between different products. After the rules are serviced, the workload required to port the arbitration logic between different ESB products can also be reduced. Finally, the servitization of Rules conforms to the SOA principles. Business Rules in an enterprise should be managed in the form of services to promote the reuse of rules.

Try to use XSLT to implement the conversion logic.

Data conversion is also a very common and important function in arbitration logic. The performance of the same data in different business systems may be different. For example, when client a accesses the WS1 service in the preceding example, the arbitration logic needs to convert the XML message to the SOAP message that calls Service.

In general, most ESB products provide a variety of data conversion methods. Many vendors are promoting the "drag-and-drop" visual ing conversion method. This feature sounds really cool and looks intuitive. However, as we mentioned earlier, ESB is a component that tends to be integrated at the Technical layer. Business personnel generally do not care about how XML is converted to soap. Therefore, for developers, this "dazzling" feature is not very attractive. More importantly, they may be very accustomed to their own programming languages, such as Java, XSLT, esql, and PHP. These languages are much easier to operate. I personally don't like the need to use visualized implicit methods to achieve data conversion, especially when it comes to conversion logic such as cyclic operation and complex computing, it is more difficult to implement visual ing.

Here, I recommend using XSLT to implement data conversion, as shown in figure 3. The reasons are as follows: First, XSLT is a standard XML Conversion language. The conversion logic implemented by XSLT can be easily transplanted between different ESB products. According to my investigation, almost all mainstream commercial ESB products support the conversion mechanism of XSLT. Second, selecting XSLT also increases the architecture flexibility on the ESB. The conversion logic in the arbitration logic is the most computation-intensive logic, which consumes the most CPU and affects the overall performance. In this architecture, we can easily strip this transformation logic out of the arbitration logic, which is done by some specialized XML Conversion acceleration software/hardware (such as IBM WebSphere datapower. This design improves the flexibility of the architecture and the overall computing performance through distributed computing.

Figure 3. XSLT implements data conversion logic and architecture Extension

Outlook

In the SOA architecture, there is no doubt that ESB plays an important role. With the evolution and evolution of enterprise architecture, the ESB cannot remain unchanged. Only by keeping pace with the times can it adapt to the development and progress of technology and thoughts. With the rise of restful Web Services, soap services are no longer the most favored standard; SOA architecture is constantly breaking through the Enterprise Firewall and extending to the cloud computing world. In such a historical environment, the ESB must develop in at least two ways to remain invincible in the competition. Otherwise, it may be replaced by new products.

First,Enhanced support for restful Web Services for ESB. Compared with the soap service, restful Web services are simple, easy to learn, and easy to expand. The call for using restful web services as basic elements of SOA is very high. Although services that do not pass HTTP transmission cannot be converted into restful services, such calls for restful services at least represent a simple desire. The author believes that this trend is irreversible, and the ESB product must enhance the integrated support for restful services. For example, you can use ESB to convert the functions of MQ applications left behind in the background into a restful service for access by restful clients.

Second,Enhanced integration of ESB with SaaS cloud applications. With the overwhelming popularity of cloud computing, people have become increasingly clear about the cloud, more and more recognized for the cloud computing economy, and cloud applications have become increasingly popular. As SaaS applications become increasingly popular, enterprises cannot deploy all their data/applications on the cloud for security, performance, and investment assets, therefore, the integration between local applications and SaaS applications on the cloud will surely become the hottest integration requirement in the next few years, which leads to the author's Outlook on the second aspect of ESB.