Enterprise implementation case of ESB Architecture

ArticleDirectory

• ESB quality attributes
• Element
• Scenario
• Element
• Scenario
• Element
• Scenario

Original address: http://guoshiguan.iteye.com/

This article describes the practical application of the ESB architecture in the enterprise, including the design and case of the ESB architecture within the Department, between departments, and enterprise. It also shares the key issues to be considered during the ESB design process; describes the implementation focus of different ESB domains.

Related Vendor Content

Registration for the Baidu technology salon Phase 1: Design and Application of the Javascript Library

Sybase online seminar: Application of Complex Event Processing and real-time analysis (Thursday, April 14)

Adobe Flash Builder 4 Simplified Chinese official high-speed download

Registration for the web app developer conference (free of charge in Beijing in December April 27 )!

Overview

The existence of ESB is mainly to integrate internal applications of the enterprise, so that the applications within the enterprise can be integrated, rather than being isolated from each other. It can be said that ESB is the center of all services in the enterprise, and the interaction between other systems must be completed through ESB. To this end, it must have the following quality attributes: availability, performance, modifier, testability, and ease of use. Refer to the quality attributes section of ESB.

To explain these architectural attributes, we can explain them at three levels: Enterprise Domain, Department domain, and ESB. In addition to high availability and performance, the high scalability of ESB is also very important. In the actual implementation process, readers can cut down the entire structure. At the beginning, they may only need one department domain, horizontal scaling is supported within a department. When the bottleneck is reached, it may need to be deployed to multiple Department domains, so that multiple horizontally scalable nodes can be expanded to reduce the responsibilities of a single node.

ESB quality attributes

Availability

An ESB is a centralized point of interaction between applications in an enterprise and third-party systems. It centrally manages all services that interact with each other. It also provides functions such as service search, management, audit, monitoring, and analysis. When the ESB service fails, the normal operation of all applications in the Enterprise will be affected. Therefore, availability comes first.

Performance

As the internal integration of an enterprise advances, the service transaction volume inside the ESB should not be small, and high performance is also very important for the ESB.

Modifiable

Because the enterprise governance of SOA is a step-by-step process, it is difficult to accurately estimate the future transaction volume at the beginning of the deployment of ESB. Therefore, performance scalability requirements are also high.

In the actual production and O & M process, we often find that services may encounter problems in one way or another. Quick modification and deployment is an important issue to avoid affecting the normal use of services by service consumers.

The ESB project is iterated over and over again with the development of SOA governance, which requires a high level of modification.

Testability

Since the release of ESB is an iterative process, services will increase in depth according to the SOA concept. In the iteration process, it is important to ensure that the previous services can pass the test smoothly.

Enterprise applications only need to be oriented to the ESB. They do not need to know where or who is using the service during interaction. At this time, the ESB test is a big problem, because when a transaction starts, you may not know where it will be, but we must ensure that the transaction is correct, in this way, the correctness of the service can be maintained.

Ease of use

To achieve ease of use, you need to improve service development efficiency, that is, you can quickly develop and deploy services. Because it has no impact on production activities, it is placed at the end.

Enterprise Domain View

In most cases, if the transaction volume is small, you can use only one department domain to support services in the entire enterprise. However, if it is only an ESB Department domain, there is no way to support the subsequent increase in transaction volume year by year. Although each department domain can perform horizontal scaling on its own, there is still a degree. After this degree is exceeded, the difficulty of horizontal scaling will increase, in this case, you may need to vertically split the business. This method is of course not cheap as horizontal expansion, but it can easily support larger business volumes.

In the enterprise domain, the biggest feature is that there are multiple Department subdomains (figure 2.1), and each department subdomain is highly autonomous. They can independently process the integration of various systems in the domain. Only when the services in other domains need to be used will they request other domains. To prevent inter-department domains from becoming a Spider network, we introduce the Enterprise Domain manager to manage services in the domain and perform necessary monitoring on these Department domains.

Enterprise Domain manager has the following components:

• Enterprise Service lookup and registration component: this component is generally deployed independently and should have high availability. Ideally, all transactions in all Department domains can be found. Cross-Domain transactions all need to use this component to find the services in the corresponding domain.
• Monitoring Component: this component allows you to view the running status of various departments and domains.

Fig 2.1

Element Enterprise Domain manager enterprise service Search registration component

This is the core component of the Enterprise Domain manager. It is used to manage all services in the enterprise. This component should have the following functions.

• Service Registration: Register the service address, service description, and Service Specification.
• Service Version Management: Manage multiple versions of a service.
• Service ClientCodeGeneration: Generate a service client based on the service address and description. In our implementation, it is generally a Java version.
• Search Service route table: The main function is to find the corresponding service address and push it to the Service Router.
• Service User Registration: To call a service, you must register with the registration component. unregistered users are not allowed to call the service. In this way, you can find the service usage path through this component, so that when the service is changed, you can effectively notify the corresponding service user.

Monitoring components

This component allows you to view the operation status of each department and issue an alert when the operation of the Department domain exceeds the threshold. If necessary, operate the traffic control components in the domain. The specific functions are as follows:

• View the running status of each department domain. Such as hardware resources, transaction information, traffic control information, and configuration information.
• Alert the resource usage.
• Perform operations on configuration parameters in a department, such as traffic control configuration parameters.
• The traffic control components in the operation domain ensure important transactions and discard secondary transactions.
• Periodically collects information in various domains. After saving the information, it provides information support for reports and decision analysis.

Department domain

The Department domain is an ESB node in an enterprise. Each department domain is divided by project group or department. Each department domain has an ESB component, this ESB is used to integrate services and applications within a department. This component will be described in detail from the department domain perspective.

Scenario subdomain Interaction

All services are registered with the service Search Component of the Enterprise Manager. This component contains the description and address information of these services. Figure 2.2 describes a process example. to initiate a cross-domain service request from department Domain A, you must use the service lookup component of the Enterprise Domain manager, the service address of Department B of the service provider can be obtained through the route table of this component before requesting the corresponding service.

To improve the performance, you can also obtain all route information at startup and cache it. The service request uses the cache to locate the address of Department Domain B. Note that when the Department domain changes the service address, there are several policies in our implementation:

• Push Service search components
• If an error occurs in the service request address, request the service Search Component again.
• Regularly clear route Cache

Fig 2.2

Department Domain View

The Department domain is the basic unit of Enterprise ESB implementation. It can even exist independently within the enterprise within a certain transaction volume range. The Department domain ESB can perform horizontal scaling independently for performance scaling. In addition, this performance scaling should be relatively cheap to some extent.

From the perspective of Department domains, we do not need to care about the internal implementation of ESB. Generally, there are only four scenarios

• Synchronous request service
• Asynchronous request service
• Synchronous service provision
• Asynchronous service provision

Systems in the same domain only need to know the above four scenarios, and other work will be integrated within the ESB. For example, for a transaction with a legacy system, the ESB will be integrated with it through the adapter, and we will elaborate on this content in the internal view of the ESB.

The Department domain mainly involves multiple application systems and ESB elements (Figure 3.1). The interaction between all application systems must go through ESB, which is a star topology. Therefore, an ESB becomes a single point of failure, which affects all subsystems of the entire department domain. This is why the quality attribute of availability in the ESB system is so important.

Fig 3.1

Element ESB Components

The ESB component is the core. The functions of this element are described in detail in the internal view of the ESB. It is located in a department, and its main role is:

• Reduce dependencies between applications: The biggest benefit of ESB is that it can streamline the dependencies of the Spider network structure so that each application only depends on ESB.
• Integration of existing applications: ESB can implement protocol conversion for existing applications through some technical components, so that existing applications can be quickly integrated into the environment of enterprise integration, it will not form an isolated information islands.
• Traffic control: ensures high availability of high-priority services.

Intra-Domain Application System

An intra-domain application system is the actual provider and consumer of internal information/services of an enterprise. When it needs to provide information/services to other consumers or consume information/services of other systems, it will be related to the ESB.

External Enterprise System

Most enterprises today have relationships with their external systems. We should not directly interact with external systems in the application system, which will consume more time on security management. In many cases, these external systems are not only used by one application. In this case, not only the complexity of a single application system is increased, but also some redundancy occurs. We can use ESB to accomplish these tasks in a unified manner and simplify the process of consuming services in the application system.

BPM

BPM systems are actually part of application systems. BPM is managed independently because BPM occupies a large part in the ESB architecture. In the actual implementation process of ESB, we can use the combination of various routes and endpoints in the ESB to implement BPM functions to a certain extent, but this will actually complicate the ESB. If you can use bpm to orchestrate the transaction process, you can reduce the internal complexity of the ESB.

If there is no BPM Application System in the application system, if possible, we 'd better add a BPM component in the Enterprise Domain to implement business processes. In this case, the ESB needs to be able to interact well with the BPM application system.

Scenario

In the following scenarios, a request actually passes through two or more components. The reason is that the ESB shields the details of the ESB requestor and service provider, when the system needs to interact with the external system, only the ESB system should be known. This is why testability is important in the ESB system. In the ESB system, integration testing is very important.

In the scenario provided by the same asynchronous service, because the transaction requestor only knows the ESB, or sometimes there is no requester at all, testing in this scenario is very important, the ESB must not only meet the technical test functions, but also complete the business test with the service provider so as to ensure the correctness of the transaction.

Synchronous request

It is the simplest application scenario (Figure 3.2). In this scenario, the ESB only performs transaction forwarding. Of course, packet conversion may also be done in the middle. All of this should be transparent from a department-level perspective. service users in a department only need to rely on the service interfaces provided by the ESB, rather than the final implementation of the service. The details here are described in the internal perspective of ESB.

Fig 3.2

Asynchronous request

If the service cannot complete the operation in a short time, synchronous requests should not be used, but asynchronous requests should be used. After the service completes the operation, a method is called back to obtain the processing result. Of course, the result may not be returned.

Fig 3.3

Synchronous service provision

Fig 3.4

Asynchronous service provision

Fig 3.5

ESB internal view

Static view of the ESB system, which consists of three parts (Figure 4.1)

• Endpoint): Its responsibilities can be divided into two parts: one is to receive service requests, and the other is to call the service provider
• Router): Mainly refers to the message routing. When the endpoint receives a request, the router selects the corresponding message service provider.
• Basic Components: Supports common components in the general ESB mode.

Fig 4.1

Looking at the dynamic ESB system, you will find that we can regard the ESB as an organized Message channel (Figure 4.2), and select a corresponding message channel when the client requests the ESB. There are many message processors in this message channel, which process the message stream according to the processor's responsibilities.

Fig 4.2

Element message processing channel

The message processing channel is the core part of the ESB architecture. The message processor of the ESB core is divided into two parts: the routing processor and the endpoint processor. Of course, our basic components will also intercept and add multiple basic component processors between the two processors in a timely manner. For example, the log component adds a log processor to record the logs of the ESB operation.

Figure 4.2 describes a simple channel. There is no branch in this channel, and there is only one route processor. It is certainly not so simple in actual use. There may be multiple route processors, there may also be multiple endpoints. When the branches of the entire channel are too complex, we suggest you think of it as a business process and hand it over to professional BPM products. This not only reduces the complexity of ESB implementation, it can also be greatly improved.

In actual development, we can use the chain of responsibility pattern figure 4.3 ). A responsible chain is a channel, and a message processor is a processor (handler) in the responsible chain ). We can use the Configuration component to complete the initialization of message processors during ESB initialization.

Fig 4.3

Message object

Because the Message channel is used, the message stream in the channel must be well defined by a unified message object (Figure 4.4). This message object not only contains the message content, it should also contain the message status and context information of the channel in which the message is located.

Fig 4.4

Messageobject: Message Carrier

Context: context. All components and configurations are registered in context.

Session: records user request information.

Endpoint)

There are two types of Endpoints: one is integrated into the ESB and the other is embedded into the application system using the ESB.

Ideally, the second type is recommended because it has a larger operational scope. It also supports some features, such as supporting SOA distributed transactions and load balancing. One advantage of implementing Server Load balancer in Embedded endpoints is that it can avoid single point of failure caused by horizontally Scalable Server Load balancer and improve availability to a certain extent. Although the ideal is good, the reality does not allow us to embed endpoints in all application systems.

In the ESB system, we can see two types of Endpoints: Inbound endpoint and outbound endpoint. One of these two types of endpoints is to receive user messages, and the other is to send messages to the service provider.

A typical endpoint can be divided into three functional blocks: transport, transformer, and filter:

Transport

Transport receives and sends data and provides the communication protocol adapter. Figure 4.5 is a typical structure, which consists of three parts:

• Connector: it is mainly responsible for handling communication protocol connections. This connection can be a TCP connection or a virtual connection within the VM.
• Messagerecerver: it is mainly responsible for listening to connections through ctor, Obtaining user request messages, and forming data into an ESB internal message object, the message object is sent to the ESB pipeline for processing.
• Messagedispatcher: it is mainly responsible for sending messages to the service provider and receiving responses from the message service provider. It will be formed into a message processing node and put into the pipeline.

Fig 4.5

Transformer 

Its main responsibility is to convert heterogeneous message formats so that they can be recognized inside the ESB. It is a message processor, which can be placed in the message processing pipeline at the beginning of activation.

To reduce the number of message conversions, generally, an ESB defines a common message format. If the incoming message does not conform to the common message format, it is converted to a common format. At present, we like to use soap as a common message format.

If some systems in the enterprise use XML as the message carrier, we recommend that you use XSLT to convert XML messages to the SOAP format. XSLT is a relatively convenient packet Conversion Technology/tool, and XSLT itself is also XML, it is also convenient to modify it, which is helpful for improving the ESB's Modifiability. With the popularization of XSLT, a large number of auxiliary development tools are available to improve the development efficiency of XSLT.

Filter 

It is mainly responsible for message filtering. The common application scenario is as follows: Determine whether to send a message to the next message processor for processing based on certain data information. The best practice of this determination process is to use the rule engine in the basic components to determine, which will achieve good repair and modification.

Routing

In architecture, the route should be the same as transformer, but only a message processing node. It must implement the interface of the message processor. In this way, it can become part of the message channel. Its biggest responsibility is to let messages find the right consumers.

Routing can work together. For example, I can first use the route splitter to break down messages, then use content routing to determine the trend of each part of messages, and use filter to filter messages. In this way, the routing work is completed through layer-by-layer nesting, but this seriously increases the complexity of routing. Although we support this approach, we do not recommend this, professional tasks should be handled by a dedicated system. These operations should be handled by the process arrangement, so we should hand them over to bpm. Our principle is that if the number of routes exceeds three layers of nesting, we recommend that you put the routes in BPM.

Basic Components

The basic components are components that ensure that the entire ESB can better complete the ESB task. Each component must be well designed. Here, we only need to describe some components as necessary, some basic components may be added in actual implementation.

• Log component: Records the logs generated by an ESB transaction. These logs are used to track business transactions.
• Traffic Control Component: Ensure that the ESB does not suddenly go down because the transaction volume suddenly exceeds its capacity.
• Rule Engine: It can be used when a filter or content-based route needs to be executed to improve the maintainability.
• Thread Pool: It is very important to ESB. It is one of the most basic components.
• Object pool: In the implementation of ESB, some objects are not thread-safe, but instantiation of such objects consumes a lot of resources. In this case, we will improve the performance of the Object pool. It is also one of the most basic components.
• Message Queue: A Message Queue can be an internal queue or an external queue. It is also a very basic component.
• Configuration component: Because ESB is actually a system that can be scaled horizontally and must be controlled by enterprise domains, the configuration components must be highly modifiable, it is best to use JMX to manage configurations.
• Monitoring components: This component is actually only a client of the monitoring platform. It is mainly used to collect some information in the ESB for the monitoring platform.
• SedaComponents(Staged event-driven architecture) the core idea is to divide the request processing process into several stages. Different Stages use different numbers of threads for processing and use the event-driven asynchronous communication mode between stages.

Simple scenario Synchronization

A large part of the implementation of ESB is simple synchronization. The flowchart in the simple process is as follows (Figure 4.6 ). This is the simplest process. Of course, the ESB framework will automatically add some processing among these message processors, such as basic message processing components such as logging.

Fig 4.6

Simple asynchronous Processing

The public component Seda is required for simple asynchronous processing. Seda puts the message object in the message queue and then directly returns it to the requester. Then the thread pool of Seda will process the message objects in the queue. The subsequent processing is the same as synchronous processing. This simple asynchronous processing is only suitable for the case where no message is returned.

The typical structure of Seda 4.7 is shown in. It requires an event queue. When a request message is asynchronous, the message will be placed in the event queue in the pipeline after the packet conversion is processed. The thread pool in the pipeline listens to this queue, and obtains the messages in the queue and sends them to the next route component. That is to say, we just put a Seda component in the transport and transformer questions on the basis of simple synchronization, so that we can implement Asynchronization.

Fig 4.7

Complex Routing

Complex routing is multi-layer nesting of routes, because each route is completed by a messageprocessor. This provides the necessary conditions for nesting. complex routing can be considered as a routing tree.

Embedded endpoint

Sometimes we embed the endpoint into the application system, which has the following benefits.

• The routing function of the endpoint can reduce the single point of failure in Server Load balancer.
• To reduce the development difficulty of the application system, we have encapsulated the embedded endpoint, and the application system can use it very easily.
• It can have special features such as transactions.

Exception Handling

Exception Handling is a necessary component of each system, and is no exception in ESB. In the ESB system, we have a dedicated Exception Handling component. We cache all exceptions in the pipeline, and then throw the exceptions to the exception handling component. The exception handling component selects the corresponding exception handling channel based on the context and exception type, handle exceptions accordingly.

Transaction guarantee Channel

Transaction guarantee request is to ensure that the consumer of the message actually consumes the message. Maybe the consumer may encounter an exception when processing the message, but in this case, we will think that the message has been consumed. The message assurance channel is used only when the service/message consumer loses connection. It uses a certain policy to re-send messages.

Custom exception Channel

A custom exception channel is used to handle user-defined exceptions. A custom exception may be a business exception.

Summary

ESB is an important node of the SOA architecture. I think it has many similarities with the traditional EAI in terms of technology. There are not many new ideas, and there are already a variety of mature products on the market, or open-source or charged, the technology is quite mature and there won't be too many technical problems. At the same time, we may also need to consider issues at the business layer. For example, when implementing an ESB, the stakeholders of a project are much greater than that of a common project. How can we deal with the relationships between them, such as service governance, it is very important. In the ESB project, I am more engaged in technical architecture and core code development, so I still cannot summarize a set of methodologies to deal with such problems. This is my most pity.

About the author

GUO Shi: Gao Weida Xiamen sub-software engineer, with many years of experience in enterprise system architecture and implementation, has a strong interest in establishing high-concurrency and high-availability Internet sites, and is now focusing on the SOA architecture. Blog http://guoshiguan.iteye.com/welcome to step on.