Build a bridge between BI and SOA

Brief introduction

We all know that business intelligence (Business Intelligence,bi) can bring many advantages to the organization. By collating, aggregating, and analyzing data, BI can help us discern what is happening within the organization and what is going to happen. Bi lets us identify the direction in which the organization is going or should be moving forward. The process of bi typically starts with extracting, transforming, and loading data (Extract, Transform, Load,etl). In general, ETL is a process in the Data Warehouse (Warehouse), which includes:

Extracts data from external and internal data sources.

Transform data to meet business needs.

Load the converted data into the Data Warehouse (or data mart, Mart)

Basically, we want to achieve BI's perfect state, "only" need one thing: data. Bi needs to hide data in the organization system.

Go into SOA

Over the past few years, we've seen advances in service-oriented architecture (service-oriented Architecture,soa) at the forefront of it architecture. As exaggerated propaganda subsided and organizations gradually completed their transition to SOA, suddenly people discovered that the data needed by BI was dispersed across services and hidden behind various interface contracts.

Take a look at the SOA component shown in Figure 1 (from my paper, what is SOA?). , we can see that in addition to the most important components--services--SOA has several other components related to the service interface:

A contract that is realized by a service

Endpoint, where the service is established

Messages that are passed back and forth between the service and its users

Policies, policies followed by services

Users, users interacting with the service

These components, along with the tenets of SOA's "shared data definition (Scheme) rather than data" and "services should be autonomous", tell us how SOA attaches importance to its interfaces. It is through this emphasis on tightly defined interfaces that the SOA provides technical and business advantages in terms of loose coupling, flexibility, and agility.

Figure 1:soa components and their relationships

On the one hand, BI focuses on the in-depth understanding of the data, on the other hand, SOA focus on isolating internal data to the interface, the target conflict is real. As Pat Helland in external Data vs. internal data, the internal data of the service should never be exposed to the outside of the service, which is what BI needs. It is for this reason, according to a recent survey conducted by Ventana Research (by Dan Everett at Dr. On Dobb's journal, only 1/3 of respondents said that their in-house IT staff had the knowledge and ability to implement BI services, a result that was not surprising to us.

We seem to face two options: either get data directly without ignoring certain SOA principles (such as "shared data definition rather than data"), or blindly expect bi to get enough data to follow the immediate interface contract. (the third option is equivalent to the first option, which I'll discuss later, that is, to create a contract that is specific to the BI requirement.) ）

Get data directly, or ...

The first option is to get the data needed for BI directly, and the process is based on the ETL that has been proven effective in the past.

SOA poses a challenge to ETL, and you have to centralize data from a number of decentralized locations (services). However, aggregating data from multiple resources is not a new issue for BI or ETL. There are already many data sources in large enterprises: ERP, CRM, data silos scattered across departments, and so on. Silos It may be easier to have soa,etl, considering that SOA promises to weave enterprise data into a high cohesion organizational structure rather than spaghetti-style point-to-point integration. (Please note that the word "commitment", in fact, to achieve high cohesion of the service structure is not an easy task.) But this is a digression, this question to use another article or even a book to explain clearly. ）

As I've said before, ETL is already mature and proven to be the foundation of a successful BI solution. However, using ETL basically means erasing most of the benefits that have made us pursue SOA. In the prehistoric era of SOA, one of the major problems we faced (the problem still exists in many organizations) is the spaghetti integration of enterprise systems. Consider the scenario shown in Figure 2. For historical reasons, each department has established its own system. As the new business needs emerge, the result is a stack of separate, fragmented systems. Then, when the data is shared between the systems, a new point-to-point interface is added to address the need for system integration. As people use the system, they find themselves needing data from another system, and the result is a point-to-point integration. Figure 2 shows four types of point-to-point integration: ETL (extract, transform, and load), this is a relationship between the database, online integration and file-based integration, both are applications of the relationship between the direct connection to the database, which is applied to the database relationship. The relationships enumerated here are not complete; there are more types of relationships, such as Replication (replication), message based relationships, etc., which are not represented in Figure 2.

The end result is spaghetti-style systems. Changes to one system can spill over into other systems, and the results are unpredictable. SOA emphasizes common interfaces and autonomy in order to solve these problems.

Figure 2: Typical spaghetti Enterprise system integration