(GO) Integrated architecture: Compare WEB APIs with service-oriented architecture and enterprise application integration

Summary: In general, SOA and Web APIs seem to solve the same problem: exposing business functionality in a real-time, reusable way. This tutorial will analyze how these initiatives differ and how they can be incorporated into an evolving integration architecture. The article will also discuss how API management differs from the integration architectures (such as SOA and EAI) that appear before it. Brief introduction

Almost all enterprises have multiple applications as a recording system for their critical data, and they also have the business functions on which they are founded. As a result, it is common for organizations to keep revealing valuable assets in these operating systems to a wider audience, both inside and outside their businesses. However, it takes time. In this tutorial, we'll look at the key stages of this assessment to help you assess the location of your business on this journey, and analyze what you might want to do to move your integration architecture toward or beyond the API's exposure.

First, let's briefly describe the history of the business function exposure, and then analyze the differences between the following two newest concepts in more detail: service-Oriented Architecture (SOA) and Web APIs.

Over the past few decades, the evolution of the entire industry in the integrated architecture has been obvious, and the level of business functionality has increased, as shown in 1.

Figure 1. Progressive disclosure of business functions

Our goal is to understand the difference between SOA and modern business Web APIs. To understand this difference effectively, we need to understand clearly what SOA brings.

Let's take a quick look at what has changed in the first 3 phases (until SOA), and then analyze what changes the Web API has added.

Point-to-point connections using the low-level API

Since enterprises have applications, they need to move and share data between applications. Allowing users to enter information repeatedly in different systems ("Swivel chair" integration) is not conducive to sustainable development in most cases. This leads to the need to establish a direct (point-to-point) low-level connection between isolated applications. It is often impossible to get a real-time response, so data is usually sent asynchronously through a file or a one-way message. Each end of each interface requires a new serialization and parsing code, shown in 2.

Figure 2. Point-to-point integration

The different connections between applications indicate that many different protocols are often required to implement different interfaces, which complicates the integration task.

To do this, we introduced the Application Programming Interface (API), which includes the transmission, the Protocol, and the data format for real-time interaction, making it a reality to get a response directly from the called System. Of course, this is the origin of the acronym API, and the API now has a new purpose called the Web API. This tutorial will clearly differentiate between the two APIs, called the original type "low-level API," and the new type is called the "Web API." In everyday language, Web APIs are often referred to simply as "APIs".

The maturity level of integration is typically measured using the service Integration Maturity Model (SIMM). The SIMM level for point-to-point integration is typically 2. SIMM is a maturity model, but we should be careful to understand the meaning of "maturity". As you move to the next level, the techniques used at each level in the model are not obsolete, they are only used more selectively. For example, even in companies that have implemented SIMM-level 4 services, it is possible to have sufficient reasonable demand for occasional point-to-point or hub-and-spoke integrations.

These low-level APIs differ significantly on different platforms, which requires that complex application-specific connection code be written to the applications on each end of the integration.

Enterprise Application integration

In the 90 's, integration tools and runtimes became more common. They know how to perform the connection and provide a central hub to perform all integration.

This implements a more similar "hub-and-spoke" architecture, and significantly reduces the amount of dedicated integration code written, as shown in 3. This typically has SIMM level 3, which is called Enterprise application Integration (EAI).

Figure 3. Hub-and-spoke architecture

These new tools and techniques mean that you can reuse connections within the scope of an integrated hub, and you just need to determine how to connect to an application once. This work is always done using the same tools and runtimes, rather than using integrated code across multiple languages and multiple platforms.

Because of the fundamentally different interaction styles between applications, they usually do not have real-time connectivity. More commonly, an inbound adapter obtains data from the system and stores it in file-or message-based storage, and then an integrated stream processes the data and passes it to the target system. When data needs to be used only for reference purposes, this inevitably results in large amounts of data being copied between systems. The real-time interface (real-time interfaces) that is connected to the original system reduces this repetition.

Gradually, the real-time interfaces connected to the operating system became more prevalent, reducing the need to replicate data across systems. However, for a new system to use one of these real-time interfaces, some work still needs to be done to connect it to a hub. Admittedly, many of the attempts made in the hub-and-spoke architecture have only slightly reduced the point-to-point problem, introducing point-to-point coding to a runtime and a tool. Unless the integration is carefully designed for reuse, creating a new interface still requires a lot of new code.

We need a more standardized way to expose functionality from a hub so that it can be reused without additional work.

Service-Oriented Architecture

At the beginning of the 2000 's, with the adoption of transport, protocol, and data format standards, such as soap/http (often referred to as "WEB services"), it became possible to expose services in a standardized manner. This means that the requestor (who understands these modern standards) can use these services with minimal effort. The direct reuse of these exposed business functions has now become possible. A well-controlled public service suite should have SIMM level 4.

Any opportunity for reuse will bring new benefits and new challenges. Using Soap/http to simply expose business functions is not enough to ensure the integrity of the service. It poses many challenges, from the unmanageable dependencies between systems to security exposures.

SOA is much more complex than protocol and data format standardization from the point of view of service disclosure. To be effective in exposing services, there is also a need to standardize the following:

• Virtualization : A user must call a virtual service endpoint that hides the complexity of its eventual implementation and location. Conversions to standardized protocols and transports are part of virtualization, but services also need to provide standard configurable aspects, such as routing and data transformations, while continuing to provide the same virtual services to users to minimize the impact of changes.
• Visibility : If you expose core business functions, you need to manage and monitor them. Achieving effective monitoring on a large scale requires a standardized approach to all services.
• security : To make your service easy to use and easier to manage, you need standardized access control, identity management, and other key security aspects. Security is complex, and you need your service to be easy to use. You need to reduce the changes to the security model exposed to the user.
• Traffic Management : How do you ensure that high-priority users always have access to the services they need and receive acceptable response times? What if you need to temporarily sacrifice one user to keep another user? How do you manage planned or unplanned outages? You need to perform some form of configurable operational control on the service exposure point, so you can tweak it without having to go through the code cycle.

These aspects are described in more detail at the beginning of this tutorial: solution Design in WebSphere Process Server and WebSphere ESB.

To achieve all of the above standardization, you need to formally separate the service exposure functionality in the schema, as shown in the service exposure gateway in 4. It may not be a separate runtime component in the final physical architecture, but at least it needs to be explicitly depicted in the design. Virtualization, visibility, security, and traffic management needs must be met in a first-class manner.

Figure 4. Service Disclosure

You will notice that we intentionally did not use the common SOA-related terminology in the chart shown in Figure 4, Enterprise Service Bus (ESB). This is because there is a big disagreement about the exact boundaries of the ESB. After all, an ESB is an architectural pattern, not a component description. Some say it's just a service public gateway, while others think it contains an integrated hub. Some people think it also contains adapters, and there are many variations between these views. There is a lot of literature describing the details of the ESB pattern, but we finally found that it would be better to clearly describe each specific component and their responsibilities.

In addition to the runtime components of SOA, there are governance aspects. If you have a large number of services, how do you decide which features to expose and their priorities? How will people discover the features they expose? How do you control the changes in the data model that you use? Some form of directory of candidates and current services must be preserved to govern the service lifecycle.

All of these worries can eventually be attributed to public service not trivial. If you simply expose functionality as a generic WEB service, there will be a lot of failure opportunities in manageability and security. In short, reuse has a cost, and it comes with the question of how to find SOA. Any project with tight deadlines and budget constraints does not want to be the first project to build a service-at least not appropriate.

In addition, in fact, the standards that people need to implement SOA concepts are constantly being developed during the implementation of SOA initiatives, so they are not mature enough. They are also changing as businesses try to implement them. It's easy to see why some SOA is hard to get the momentum of development. In many companies, SOA is confined to a specific area of the business, or virtually only a few core services are in effect.

Json/http Interface Introduction

Browser-based applications have become more complex, and mechanisms have been introduced to write more feature-rich, responsive web pages. These mechanisms take advantage of the browser's increasingly sophisticated client-side scripting capabilities, and their ability to use technologies such as AJAX to perform background HTTP requests to retrieve data, and the user experience is not interrupted by page preload.

Web pages typically request Web-specific data from a Web server that is associated with a page. Soap/http requests that are common in SOA are difficult to handle in JavaScript, and the sending of requests often makes Internet connections with low bandwidth overburdened. Performing finer-grained data requests is fast becoming popular and, if possible, can be changed to use JavaScript native JSON data format, shown in the red area of 5.

Figure 5. Fine-grained exposure of rich browser applications

Because they are not limited by the SOAP standard, these interfaces can be considered as a way to simplify how to perform "actions" or "actions" on the data being represented. In an interesting retrospective of the early roots of the Web, the initial intent behind HTTP was revealed. Many aspects of the HTTP standard are designed around the architecture principles of the Representational state Transfer (REST) submitted by Roy Fielding in 2000. A more simplistic interaction style based on entities is derived. This interaction style is recommended to use common HTTP insights (POST, GET, PUT, delete) in a way that is similar to common database interaction Insights (Create, request, update, delete). The global network identifies these insights in a first-class way to provide hidden benefits, such as caching. It also uses URL paths to navigate the relationships between data entities.

In a more streamlined example, you can imagine the process of adding a product to an order, which can be done by issuing an HTTP "POST" to a URL that resembles the following URL:

Https://www.mycompany.com/orders/123456/item

The JSON format data in the body of the HTTP request resembles the following form:

{"title": "Romeo and Juliet",   "note" "" Special Edition ",   " quantity:1,   

Where the URL describes a particular data entity (often referred to as a "resource"), the use of the HTTP verb "POST" means that it is a "create" operation for a new order item. To host the same information in SOAP, the code is more similar to the following form:

Http://www.example.org/ordermanagement http/1.1<?xml version= "1.0"? ><soap:envelope xmlns:soap= "/http/ ... "soap:encodingstyle=" http://... "> ...  SOAP Headers  ... <soap:body xmlns:m= "Http://www.example.org/ordermanagement" >    <m:AddOrderItem>      <m:order Orderid= "123456"        <m:item>          <m:title>romeo and juliet</m:title>          <m:note> Special edition</m:note>          <m:quantity>1</m:quantity>          <m:price>9.99</m:price >        </m:item>      </m:order>    </m:AddOrderItem>  

These json/http-based interfaces provide some useful simplification compared to the more complex SOAP standards that have emerged earlier. However, SOAP has a larger set of standards that can accomplish many things that these interfaces do not do. They are used by different audiences, and not all of these standards are necessary in this area.

At least initially, there are some limitations on the reusability of these new interfaces. Because of the same-origin policy implemented by browsers, web-based applications are difficult (but not impossible) to make HTTP requests to interfaces provided by other companies. This means that the most common initial use of these json/http-based interfaces is between the company's Web page and its own corporate data. However, some technologies (such as proxies, JSONP), and standards (such as CORS) mitigate these limitations, enabling them to be reused more broadly, making the Web API a reality.

What is a Web API?

There is no formal definition for the exact meaning of the web API, just like the Web service before it, but we try to be clear about what it means.

Generally speaking, "Web API" refers to features and data that are exposed in the following ways:

• Exposed via HTTP (S)
• Using the HTTP protocol in restful style
• Using JSON as the data format
• Can be used via the Internet

For anyone who creates a new Web API today, this could be your starting point. However, at some level, this definition is too simple:

• Not all Web APIs use JSON: most APIs use JSON as the data format, but some APIs provide XML as an alternate format, or even the only use of XML. In theory, any request that HTTP can respond to is valid. If you include MIME types (which, for example, may mean using PDF files), this broader use is less common.
• Not all Web APIs are public: we will see in a later section that the API is not only exposed and used on the public Internet. However, it is reasonable to assume that many of the consensus on style, usage, and supported products and agreements are driven by the use of the Internet.
• Not all Web APIs use the REST-style features of HTTP directly : There are many soap/http interfaces for the Internet, and it is hard to deny that these interfaces are, in some form, Web APIs. They may be less "restful" and more difficult to use. However, many soap/http interfaces subsequently introduced the equivalent functionality of the json/http "REST style".
• Few web APIs are completely restful: using Json/http in the Web API means that it is definitely more restful than before. Therefore, they are often referred to as "REST" interfaces. However, in practice, most interfaces only conform to some of the rest recommendations described in the original material for that topic. A common complaint against self-proclaimed, restful APIs is that they rarely provide the links recommended by the Hateos method.
• It is highly recommended to use HTTPS: HTTPS is clearly preferred, and many people consider it a mandatory requirement for WEB APIs. The payload typically contains private data, and the credentials used to access the Web API are usually confidential.

So there is a new, lighter protocol and interaction style, but this protocol and interaction style alone cannot protect the evolution of real-time data integration.

What are the triggering factors that make Web APIs mature?

There have been major changes in the industry when smartphones, which have an easy-to-access "app" store, have been in the mainstream for about 2007 years. Mobile Apps ("apps") have become popular and large groups of developers can be involved in the development. In addition to some notable exceptions, applications rarely run alone. They need to interact with the world around them. Developers can write much more powerful applications if they have an easy way to consolidate access to data and functionality provided by other companies.

This is not just a requirement from mobile app developers, but developers of richer sites also need more extensive and easier access to data. However, mobile often leads to a large number of new Web API users. They have no barriers to security restrictions that pose some challenges to the use of APIs in browser-based applications. So, you can assume that the introduction of WEB APIs has two important implications: requirements and capabilities.

• new ways to make money: the emerging-profit financing model, driven by the prevalence of, but not limited to, the new generation of mobile service users who represent applications such as phones, tablets, watches, and so on, requiring access to real-time data and functionality.
• Proven Ability: the standards, techniques, and methodologies for exposing business functions have matured over a decade of efforts to open up reusable services. For example, the openness of protocols and data formats is maturing in ongoing experimentation. The API's public gateway can now be used as a one-level component, and it has a broadly recognized range of responsibilities.

What is the difference between a Web API and a previous API?

It is in the new demand and its associated financing model that big data plays an important role. The audience for the exposed business functions is outside the enterprise. As shown in 6, SOA services are more common in the Enterprise, generally based on a project vulnerability and their respective known requirements. Web APIs are often exposed externally to often unknown and potentially large user groups, often for highly innovative and unpredictable purposes.

Figure 6. New factors affecting public service outside the enterprise

If a Web API can provide data that is useful to an application, it may also be useful for other applications. Put these services (or APIs) on the network, and the Web API suddenly gets potential users across the Internet, even to many segments of the customer base that were previously inaccessible. This is inseparable from the new financing model. We have the opportunity to profit from these open business functions. The Web API becomes a new "product" provided by the organization.

Return on investment may have many different forms:

• Direct Revenue: For example, a Web API for purchasing goods.
• Indirect Income: For example, earn commissions by providing aggregation services to other service providers.
• cost savings: For example, implementing self-service to reduce an expensive customer service application.
• Marketing : For example, put product information in front of a larger customer base.

To be sure, new markets can be obtained by wrapping the innovation of the Application Designer into the enterprise.

How do you cater for this significant change in the needs of your integrated architecture?

What is the difference between this public component and the Web API?

Based on the early definition of the Web API, you can see that there are 3 key aspects of exposing external Web APIs that are fundamentally changing when compared to Enterprise services:

1. Beyond the Enterprise boundaries: Web API users are not part of the enterprise, so they are not directly affected and controlled by you.
2. Countless users: the potential users of your API are much more than the Enterprise service users.
3. Competitive Markets : If your Web API doesn't meet the expectations of your users, they have alternative options that other companies offer. Within an enterprise that has SOA, they may have only one choice.

These key differences can lead to significant revisions in the way you architect and design your Web API. In this tutorial, we focus primarily on the differences in architecture. On the schema, you obviously still need some form of public gateway, but there are some new elements to the requirements of that gateway.

Recalling the previous tutorial, reuse is always a cost. As you can see from Figure 7, your exposure capabilities need to be extended beyond the core SOA requirements.

Figure 7. New capabilities for exposing APIs outside of the enterprise

Let's look at the two main aspects of these new challenges:

• Partner Management: You now have a large team of mobile application designers who might want to experiment with your Web API. If it is not designed to be very easy to use and attractive, your Web API "product" will soon be overtaken by competitors. How do you build new relationships with these new partners? How do you keep track of who is accessing these features? You may have external stakeholders that rely on your Web API as a fundamental part of their business. How do you establish and monitor what level of service they need? Partner management must be a one-level feature provided by the Web API exposure component. Because of the large number of potential partners, partner management must be self-service, and it needs to identify partners and monitor and control their usage based on agreed authorization plans.
• Security: It is clear that exposing Web APIs through public media (such as the Internet) means that new levels of security issues need to be considered, from a variety of payload-borne attacks, such as XML threats, to denial-of-service attacks on throughput or connections. You must also reliably validate your partner's applications to effectively control their service levels.

This increased complexity has led to the advent of what is now known as API management . Web API Management is an architectural intent, not a single component, although they are clearly products designed for that purpose. It enables organizations to expose and manage Web APIs in a simple and secure manner. It combines a more robust and secure gateway with the capabilities associated with partner management.

Figure 8 shows the main components required to implement API management, as well as the typical roles involved.

Figure 8. Example of a typical model for API management

All of these roles are loosely presented in one form or another in SOA, but their implementations are often less formal. The fact that Web APIs are public-facing has forced them to become mature. A typical set of roles is:

• API Product Manager: This role builds a Web API for sales, prepares and manages its use of the plan, and evaluates the success of the Web API with historical analysis.
• API Developer: This role provides the essence behind the Web API surface by configuring connectivity and data mapping with back-end systems that provide real data and functionality.
• Application Developer: This role leverages the Web API on the product through the API portal to sign agreements to use them through a plan defined by the API Product manager.
• operations: This role monitors and manages the Web APIs on a daily basis to ensure that they meet the service levels defined by the plan.

What is the "essence" of the architecture behind the API surface?

The Web API gateway is just a surface or exposure point for the Web API. It does not provide any actual functionality or data provided by the Web API. This allows us to understand the entire evolution of the integrated architecture within the enterprise-from isolated applications to point-to-point communications and hub-and-spoke middleware, potentially implementing SOA.

Admittedly, when deciding where the underlying implementation of the Web API should come from, it is highly dependent on how the enterprise evolves. Figure 9 shows an example of the most common options:

1. Re-exposing an existing enterprise service
2. Exposes a new integration that is tuned through a hub
3. Exposing an interface that has been provided by the provider system

Figure 9. Examples of options for Web API implementations

It is easy to think that re-exposing existing services (option A in Figure 9) is the most common. After all, SOA has existed for more than 10 of years, and it is certain that most companies already have a set of services. Exposing these services will be the most efficient way to profit from previous investments in integration. While there are certain organizations that are doing this, it may not be as common as you might expect for the following reasons:

• Integration Maturity: As mentioned earlier, services are usually exposed only in specific parts of the business where they have value. For other parts of the business, other integration modes, such as hub or even point-to-point patterns, may still be sufficient.
• granularity: Because Enterprise services are typically created based on known business requirements, they typically have a coarse granularity, bring a relatively large data graph, and may contain many sub-objects. These operations are too burdensome for fast-responding applications on mobile devices, especially considering the often-changing bandwidth of the device.
• Security: When an enterprise service performs an update operation, the operation is often performed in an enterprise-specific manner, for example, assuming the caller's trustworthiness, the security of the channel, or the security token mechanism for internal use only.
• Relevance: data and features that are noteworthy or available for sale within a company are usually irrelevant or unsuitable for external disclosure. What we look for in the Web API is usually a brand new market opportunity, a novel concept that may expose completely different data.

Therefore, option B in Figure 9 may be common, and the Web API is implemented using integration. Perhaps they reuse components and adapters within the integration hub itself, but do not reuse existing services.

Option C is rarely seen because the desired Web API is almost certainly different from the existing application. Web API gateways often have limited data conversion capabilities. However, once the integration logic goes far beyond the basic data mapping and simple aggregation, the Web API gateway is no longer suitable for this logic in architecture.

Cloud-based API management

In today's era, people are eager to reduce the footprint of their internal infrastructure, and businesses are looking for features that are easier to get from cloud-based providers. API Management is a good example. WEB API gateways and API management functions have clear responsibilities, so they are easily separated from the internal architecture, as shown in 10. Because the goal is to expose them to external stakeholders, it is reasonable to host Web APIs from a cloud-based provider.

Figure 10. Cloud-based API management service provider

Although relevant to a specific enterprise, this cloud-based API management is particularly well-suited for companies that are "born in the Web". For example, a start-up company chooses not to establish an internal infrastructure to host all features in a cloud-based environment. Cloud-based API management enables them to provide a single, unified public point for users to find their Web APIs, regardless of where they are hosted in the cloud.

With this approach, you need to consider the following factors:

• access control: How do you securely provide cloud-based API management Services the ability to access your internal middleware or real-world operating systems? Obviously, some form of secure connector is needed, but how much control do you want to give to external stakeholders?
• Delay: Users of the Web API now need to jump two times over the Internet to reach your site: The first time you jump to a WEB API provider and then jump to your organization. The Web API appears to be more "colloquial", which means that you usually need to perform more calls to achieve the same purpose. As a result, more end users are waiting for the time associated with communication delays. Depending on the service's global reach and Web API interactions are typically busy, this additional latency can be high. The API management solution often provides caching options to reduce this delay for requests that are appropriate for that pattern.

The next stage in the Api-soa in the enterprise?

The new "REST"-style json/http interface is lighter, suitable and supported by modern programming languages and frameworks. API Management Gateway products are becoming more and more mature. Why take advantage of all these benefits within the enterprise? Many people now see the internal API management layer as an alternative, and in some cases, a more efficient way to expose data and functionality within an enterprise-the extension of the service exposure gateway shown in Figure 4.

Figure 11 shows the minimum architectural configuration required to implement a single management point for both internal and external APIs. This added advantage is that internal users can directly use externally exposed APIs without having to route to the Internet. In fact, businesses often still prefer to have separate internal and external API gateways for more reliable separation of concerns.

Figure 11. APIs for internal and external users

So for some, APIs have become the next stage in their company's journey toward a service-oriented architecture and the opportunity for them to open up externally.

A new business-to-business form

Another case worthy of consideration is the continued sophistication of WEB API technologies and practices that are used to provide inter-business communication. The business-to-business interface is no longer novel from any perspective. For decades, companies have used a variety of standards to exchange data, such as those related to electronic data Interchange (EDI). Because of specialization in specific industry requirements, the standards and tools used to implement them must be very deep and complex. Many businesses need a more lightweight alternative to exchanging data, but they still need to meet some core requirements, such as security features and partner self-management. Even if the company does not plan to provide APIs to the general public, the security of API management and the utilization of partner management can be valuable for implementing a simple private interface with a specific business partner.

Beyond the Web API

Trying to guess how the integration architecture will evolve as a result of technological advances in this area in recent years can be difficult and may be foolish. However, this seems to involve further understanding of the recent situation of mobile users. An important factor is that, for larger communities, the desire to "always be online" continues to change to "intermittent connections".

This means the revival of message-based event-driven interactions. We have seen more interactive models based on event subscriptions. For example, all major mobile vendors have a mechanism to push events to devices. Sensors can now emit events in more widely accepted protocols, such as a less specialized coupling between sensors and event subscribers. The Internet of Things is introducing sensors in everything from wearable technology to connected cars to a different type of application. These may be less concerned with the specific data obtained from the operating system, but rather pay close attention to the topics of interest and intelligently interpret the events received.

Conclusion

Looking back at Figure 1, you can see how the integration architecture enables progressively exposing business functions to more public, as well as the growing sophistication of the functionality used to achieve this openness. API Management is the latest technology in this area, and it is important to recognize that evolving patterns, technologies, and concepts (such as hub-and-spoke architectures and SOA) are still useful and appropriate in the right circumstances.

Thanks

This tutorial is the result of regular and ongoing coordination with Brian Petrini on the integration theme.

(GO) Integrated architecture: Compare WEB APIs with service-oriented architecture and enterprise application integration