What is ETL?

ETL is an important part of Bi. Let's take a look at the definition in wiki:

ETL is the abbreviation of extract-transform-load. It is the process of data extraction, conversion, and loading for filling and updating data warehouses. This is the data collection step before realizing business intelligence. After this step is completed, you can mine and analyze the data in the database.

For ETL, it is easy to understand what it is, that is, to organize scattered and difficult-to-use data, it turns into clear, easy-to-use, and (possibly at the same time) centralized data. In addition to the ETL process, you can perform further data analysis and utilization based on the report analysis system, multi-dimensional analysis system, and data mining system.

This process can be written through hard coddingProgramYou can also use various ETL tools.

For ETL, there are two common problems:

1. Why ETL instead of directly using data?

There are many reasons for this. For example, the raw data volume may be too large and needs to be extracted. servers with business data may face performance pressure and do not want analysis to affect performance. The heterogeneous and low-quality energy data can be achieved, some data may be difficult to use directly, such as data in the SAP system.

2. What are the main differences between different ETL tools and methods?

Since ETL is a process that can be implemented with hard codding, it is easy to ask "Why to use ETL tools ". In fact, hard codding is also an alternative ETL method. There are two main differences between tools and methods. One is the complexity of the design and establishment of ETL processes, for example, some work tools can be quickly established, modified, and debugged, while writing a program requires a lot of effort. Some work requires a lot of effort to configure the ETL tool. It is better to directly write an SQL statement, there are also great differences between different tools. Second, it is the efficiency of runtime. Some ETL work should be carried out frequently, while others should be performed at the initial time, some have large data volumes and high timeliness requirements, while others are vice versa. In this way, you may need to select an appropriate ETL Tool Based on the operational efficiency needs under different circumstances.

From the above two questions, we can understand the necessity and applicability of ETL. At the same time, we can also realize that ETL is a concept and should be selected (including not used) according to actual situations and needs) suitable ETL tools or methods.