Teiid data Federation-based integration solution

Teiid data Federation-based integration solution

Virtual Database-VDB combines one or more physical data sources to provide a simpler data integration solution. It provides containers containing components, these components are used to integrate data from different data sources and access them through consistent APIs. These physical data sources include JDBC data sources, CSV text files, data tables, and even Web services. This technology is called "Data federation"-Data federation. Data Federation technology provides an abstract data interface for data. The virtual view composed of multiple data sources makes it unnecessary for the data consumer (Application) to know the physical location, data structure, and storage method of the data.

Compared with other data integration technologies, data Federation has obvious advantages in the convenience and timeliness of data retrieval. Nowadays, many enterprises are implementing SOA. The flexibility and agility of SOA requires organizations to provide data with less latency. In the past, data mart and data warehouse creation using Batch Tasks or ETL were not very real-time, and were replaced by data Federation technology in many application scenarios.

However, federated technologies access data through a federation view, which is real-time. As the enterprise's Data volume increases, the performance problem is a problem faced by all Data Integration tools. However, due to the fundamental design defects, Data Federation has made great progress in this respect, however, there is still no comparison with other data integration technologies. This shortcoming restricts its applicability in the field of data integration. Although data Federation technology is used in many fields, it is rarely used in key core business systems.

Figure 1-1 Data Federation

Currently, the data Federation project has the following features: simple and limited data sources, small data result sets, read-only data access, and simple data quality requirements. Typical federated Data products include AquaLogic Data Services Platform of BEA, Federation Server of IBM, Enterprise Information Integration of Software AG, DataFederation of Sybase, and JBoss MetaMatrix of Red Hat, its open-source implementation is Teiid.

What is Teiid? Teiid is a data virtualization system consisting of tools, components, and services. It is used to create and execute two-way data services. It provides runtime queries for Virtual databases (VDB, enables applications to use data from multiple heterogeneous storage. Abstract and federal technologies are used to achieve real-time data access and integration of distributed data sources without the need to copy or move data from the record system. With Teiid, you can use the designer or dynamic VDB to create a VDB and then access data in a federated manner. 1-2.

Figure 1-2 Teiid Runtime

In computer science, a model in Teiid is a collection of information structures. A common relational model locates a table consisting of columns and data records. Another common model is the XML model, which defines hierarchical data sets. The Teiid virtual database also has the concept of a model. The model is defined as the relationship between the entity and the entity. The entity represents the information integrated and exposed to the application and must be completely defined.

Use the Source Model and View Model in Teiid ). The source model defines the structured and data features of the information contained in the data source. Teiid maps the information in the source model to the information in different data sources, just like a single data source.

In addition to the source model, Teiid also provides the ability to define the view model. Views are changes between models. Through the view model, an abstract layer can be defined on the physical layer. Therefore, the information presented to users can be closer to the commercial needs, rather than the original physical storage. These business views can be presented in multiple forms: relational, XML, or Web Services.

In short, the source model represents the structure and features of the physical data source, while the view model represents the structure and features of the abstract data structure to be presented to the application.

Figure 1-2 internal virtual database

The model must be in a valid state so that VDB can use it for data access. Verification of a single model means that the model must be in the self-consistent and complete state, that is, the model has no missing or reference entities that do not exist. Verification of multiple models will check that the dependencies of all internal models exist and can be decomposed.

Teiid Components

Teiid includes the following components:

• Query Engine (Teiid Query Engine): the Core of Teiid is a high-performance Query Engine that processes relational, XML, XQuery, and (storage) process queries from federated data sources, supports homogenous schema, hetrogenous schema, transaction, and user-defined function ).
• Virtual database
• Connector Framework: includes a converter and a resource adapter. The converter provides an abstraction layer between the query engine and the physical data source. It can convert the query commands sent by Teiid to the specific commands of the data source and execute these commands through the resource adapter.
• Teiid Console: provides a Web interface for users to monitor the operation of Teiid.
• Admin Shell: A Shell Interface written in Groovy that supports automated management and task Testing Using command lines or scripts.

The relationships between components are as follows:

Figure 1-3 Teiid Components

In addition, the Teiid release package includes some tools, mainly including:

• Designer: Teiid designer is an Eclipse-based GUI tool used to define Virtual databases, including views, stored procedures, and even dynamic XML documents.
• Connector development tools-used to develop user-defined converters and resource adapters.

There are two methods to create VDB in Teiid. You can create a dynamic VDB. This type of VDB is defined by a simple XML file, which defines the data source it wants to integrate and provides access through JDBC. You can use all defined data sources to write queries for the VDB, just like using a single data source. However, dynamic VDB does not provide a view abstraction layer.

You can also use the Teiid designer to create a VDB. It can be used not only to define data source models, but also to import metadata and statistics, and to define associations and XML views on these data sources. This allows you to abstract the underlying physical data structure and expose the required information structure to applications.

VDB must always be in a complete state, which means that all information is contained in VDB and there cannot be external dependencies. After a VDB is defined, it must be deployed to Teiid runtime. If no errors occur during the deployment and the underlying data source is correctly configured, The VDB can be accessed by the customer application. Once the VDB is deployed, it can be accessed through JDBC-SQL, SOAP (Web Services), SOAP-SQL, or XQuery.

Figure 1-4 deploy VDB to runtime

So far, we have introduced some important concepts of Teiid. Next we will learn how to create and release a virtual database. So let's begin!

Download and install Teiid to run Teiid, you must first install the following components in the system:

• Java 6 (OpenJDK or Oracle JDK)
• Maven 3: This is a small tool required to run the instance.
• JBoss AS, the latest version is 7.1.1. Unzip it directly.

After everything is ready, you can download and install the Teiid runtime. The latest version is 8.2.0 (teiid-8.2.0.Final-jboss-dist.zip). You only need to extract all the content to the JBoss AS installation directory. Because Teiid and JBoss AS have the same directory structure, Teiid will be merged into the corresponding directory. We recommend that you download the Admin Console and Admin Shell toolkit. The installation method is similar to runtime.

Run the following command to start the Standalone mode of JBoss:

<Jboss-install>/bin/standalone. sh-c = standalone-teiid.xml

To enter the Admin Console, you must add administrative users, run the add-user.bat, and enter:

What type of user do you wish to add?
A) Management User (mgmt-users.properties)
B) Application User (application-users.properties)
():

 

Enter the details of the new user to add.
Realm (ManagementRealm ):
Username: raylinn
Password:
Re-enter Password:
About to add user 'raylinn' for realm' managementrealm'
Is this correct yes/no? Yes
Added user 'raylinn' to file 'C: \ jboss \ standalone \ configuration \ mgmt-users.properties'
Added user 'raylinn' to file 'C: \ jboss \ domain \ configuration \ mgmt-users.properties'
Press any key to continue...

Now you can enter http: // localhost: 8080/in your browser to enter the JBoss welcome page. Click the "Administration Console" link on the page to see the management interface, 1-5:

Figure 1-5 Admin Console page.

Create a virtual database in Teiid. You can use a designer or an XML file called dynamic VDB to create a virtual database. First, we know how to use dynamic VDB. Later we will learn how to use the designer. NOTE: If your application needs to use the view layer on top of the data source, you must use the designer.

We use a portfolio as an example to demonstrate how Teiid integrates multiple data sources and returns results from multiple data sources using a single query. With data Federation technology, it is easy to integrate new data sources without exposing any access information.

In this example, we will use two data sources: Account Information (relational) stored in the H2 database, and the stock price (non-relational) stored in the CSV file ). We will create a dynamic VDB named DynamicPortfolio. In this XML file, we will set a model for each data source: Account and MarketData. When you deploy the VDB to Teiid for running, the application can access it using JDBC.

Figure 1-6 access the data source through dynamic VDB

The query statement is associated with the data in the model Accounts and MarketData. In the background, Teiid simultaneously accesses the relational and non-relational data sources, calculates the investment profit, and returns the result. DynamicPortfolio's content is as follows (portfolio-vdb.xml ):

<? Xml version = "1.0" encoding = "UTF-8" standalone = "yes"?>
<Vdb name = "Portfolio" version = "1">
<Description> The Portfolio Dynamic VDB </description>
<Property name = "UseConnectorMetadata" value = "true"/>
<Model name = "MarketData">
<Source name = "text-connector" translator-name = "file" connection-jndi-name = "java:/marketdata-file"/>
</Model>
<Model name = "Accounts">
<Property name = "importer. useFullSchemaName" value = "false"/>
<Source name = "h2-connector" translator-name = "h2" connection-jndi-name = "java:/accounts-ds"/>
</Model>
</Vdb>

Here we will explain the above vdb xml structure:

• The element "vdb" defines the name and version of the virtual database. In this example, the "Portfolio" version is 1.
• The element "model" indicates the integrated schema. In this example, we define two data sources. "MarketData" indicates the Schema of the text file that stores the stock information, and "Accounts" indicates the Schema of the HSQL database.
• The sub-element "source" of the "model" element defines the data source name (any name) and the converter name (which defines the data source type, for example, oracle, db2, mysql, h2, file, ws, etc.), the subsequent "connection-jndi-name" defines the JNDI name of the data source in the JBoss AS container
• You can also find some "property" elements in the "model" element to define how metadata is imported from the data source. For more information, see the dynamic VDB reference manual.

Of course, we also need to establish their JNDI lookup for these two data sources so that dynamic VDB can obtain database connection information. For details about how to compile the XML file for JNDI search, refer to the example and description in the docs/teiid/datasource directory. But note that the JNDI lookup of marketdata is to merge into the standalone-teiid.xml configuration file, which is not the same as the database.

Once the JNDI lookup deployment is successful, you can deploy dynamic VDB, just copy the portfolio-vdb.xml to the following folder,

<Jboss-install>/standalone/deployments

This action triggers the Teiid to automatically load the VDB when it is running, and verifies whether the data source is available. Once the data source is loaded successfully, the VDB status in the Admin Console is displayed as "Active ".

In addition to this method, you can also use the Admin Console or Admin Shell to deploy the virtual data Bureau to a remote server or cluster. The command line deployment method is provided here:

Bin/jboss-admin.sh
Command and run
# In stand alone mode
Deploy/path/to/my. vdb
# In domain mode
Deploy-all-server-groups/path/to/my. vdb

For more information about Teiid, click here.
Teiid: click here

This article permanently updates the link address: