Label:Data integration has become the enterprise in the pursuit of market share of the key technology components, and rely on manual coding in different ways, more and more enterprises choose a complete data integration solution to support its IT strategy, from big data analysis to cloud platform integration.A recent study by Dao Research compares the differences between several of the world's leading data integration solutions and the impact of these product technologies on the real business. T
Microsoft integration services is a platform that can generate high-performance data integration solutions, including extracting, transforming, and loading (ETL) packages for data warehouses.
Integration Services includes graphical tools and wizard used to generate and adjust packages; tasks used to execute workflow functions (such as FTP operations), execute SQL statements, and send emails; the data sources and targets used to extract and load data.
For everything, supervision is an effective way to improve itself, as is BI. In my personal experience, BI supervision can be divided into two types (Welcome to the discussion ):Runtime supervision(Runtime Monitoring)AndMonitoring of data warehouse health status(DW Healthy Monitoring):1. Runtime supervisionThe so-called runtime supervision refers to the process of monitoring data from the data source to the data warehouse. In general, it is to supervise the
decision-makers should be able to manipulate the data of the enterprise flexibly, observe the state of the enterprise from many aspects and multi-angle, and understand the change of the enterprise in multi-dimensional form. Using OLAP tools, we can make a connection between the dimension table and the fact table, and then do the aggregation operation to save the cube to achieve the objective of multi-angle analysis.Front-end display tools: Front-End display tool is to assist users to multi-angl
first of all, I admire eguid, whether it is technology or sharing, thanks to his series of Bowen on my first learning process help, so clearly indicated the reprint address:
http://blog.csdn.net/eguid_1/article/details/51725970
one, the local push to send the end1, Local: The use of JAVACV (Android and Java Platform recommended JAVACV), FFmpeg, OpenCV or JMF can be very convenient access to the local camera streaming media
JAVACV Series arti
in the past century, with the rapid development of computer and network communication technology, the traditional teaching mode of education is facing great challenge. The use of Internet for online teaching and the development of Web-based courses has become an important issue in the educational technology sector and the reform trend. Audio and video transmission in the distance education has been widely used and has been a good result. Among them, such as Noah's Point reader remote education u
1, the definition of ETLETL is "Extract"," Transform","Load" the initials of three words namely "extract "," Conversion "," Loading ", but we are often referred to as the daily data extraction. ETL is the core and soul of BI/DW (Business intelligence/Data Warehouse), integrating and improving the value of data according to the unified Rules , is responsible for the completion of data from the data source to the target data Warehouse conversion proces
required to handle the second type of modification
Mini Dimension (minidimension):
Extract a few fields from a common large dimension to form a small field dimension that can be used in a query with a field in a mini dimension
This design significantly improves query efficiency
type of fact:
Granularity fact table (additive Fact)
Cycle Snapshot fact table (semi-additive Fact)
Aggregation Snapshot fact table (non-additive Fact)
Non-factual fact table (factless Fact table)
Granularity fact table
1, Ali Open source software: datax
Datax is a heterogeneous data source offline Synchronization tool that is dedicated to achieving stable and efficient data synchronization between heterogeneous data sources including relational databases (MySQL, Oracle, etc.), HDFS, Hive, ODPS, HBase, FTP, and more. (Excerpt from Wikipedia)
2. Apache Open source software: Sqoop
Sqoop (pronunciation: skup) is an open source tool that is used primarily in Hadoop (Hive) and traditional databases (MySQL, PostgreSQ
ETL is the abbreviation of "extract", "transform", and "LOAD", that is, "extraction", "Conversion", and "loading ", however, we often call it Data Extraction for short.
ETL is the core and soul of Bi/DW (Business Intelligence/data warehouse). It integrates and improves the value of data according to unified rules, it is responsible for the process of converting data from the data source to the target data w
There are four data ETL modes based on the model design and source data:
Completely refresh, image increment, event increment, Image Comparison
There are four data ETL modes based on the model design and source data:
Completely refresh: Only the latest data is included in the data warehouse data table,The original data is deleted for each load, and the latest source data is fully loaded.. In this mode,
ETL 4: SQL server integration services
SSIS is Microsoft's upgrade to DTS on SQL server2005. It has to be said that Microsoft has spent a lot of effort on Bi, including providing multiple tools such as SSIs, SSAs, and SSRs, from creating a data warehouse to extracting data from metadata, to creating dimensions and mining structures, mining models, training, report model design, report design, and publishing, It is very convenient and powerful. Start
pushes the data from the data source. If the data source is protected and is forbidden, you can only use the data source to push the data.The following table summarizes the source data tables and their extraction modes used by the dimension and fact tables in this example.
Time stamp Mode
Snapshot mode
Trigger mode
Log mode
Ability to differentiate inserts/updates
Whether
Is
Is
Is
Multiple updates detected during
One of the goals of the data warehouse is the ability to provide timely, consistent, and reliable data for enhanced business functions.In order to achieve the above objectives, ETL must be continuously improved according to the following three standards:
Reliability
Availability of
Ease of management
Subsystem 22--Job Schedulersubsystem 23--Backup Systemsubsystem 24--Recovery and restart systemsubsystem 25--version control systemSubsyste
1 Log Table 1.1 ideasA log table is used to record the primary key of a table Yw_tablea the changed data in the Business library. Before the data enters the BI Library target table Bi_tablea, delete is based on the primary key recorded by the log table.1.2 Design 1.2.1 Log table structureCREATE TABLE LOG ( varchar), -- primary key 1 VARCHAR(20 ), - - primary key 2 VARCHAR, - - source table updatedate Date, -- update date loaddate- - Load Date );1.2.2
Tags: commercial int ase NSF process form color number BottomHere are a few words about some of their concepts:(1)db/database/Database -This is the OLTP database, the online things database, used to support production, such as the supermarket trading system. DB retains the latest state of data information, only one state! For example, every morning to get up and face in the mirror, see is the state, as for the previous day of the state, will not appear in front of your eyes, this is a db.(2)dw/d
Tags: sel note Select avoid IMG int Data Warehouse Problem toolbarFirst, prefaceThe company practical Hadoop constructs the Data warehouse, during the inevitable practical hivesql, in the ETL process, the speed has become the question which avoids can avoid. I have a few data tables associated with running 1 hours of experience, you may feel indifferent, but many times ETL will be multiple hours, very waste
Tags: Options import profile preparation Query str user Lin marginIntroduction to ETL: ETL (extract-transform-load abbreviation, that is, the process of data extraction, transformation, loading) Database to Database The following explains: Kettle Tool Implementation method Case Purpose : Import the EMP table from user Scott under User testuser. Preparation: first create a new table with the same structure a
Real-time streaming processing complete flow based on flume+kafka+spark-streaming
1, environment preparation, four test server
Spark Cluster Three, SPARK1,SPARK2,SPARK3
Kafka cluster Three, SPARK1,SPARK2,SPARK3
Zookeeper cluster three, SPARK1,SPARK2,SPARK3
Log Receive server, SPARK1
Log collection server, Redis (this machine is used to do redis development, now used to do log collection test, the hostname
WSE3.0 Framework provides data optimization transport mechanism, WSE3.0 build Web Service Security (4): Mtom message transmission optimization and file upload, download questions are introduced. WCF also provides streaming operations to support the transfer and processing optimization of large data objects, and today we WCF distributed development Step-win series (4): Use flow operations (streaming Operatio
The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion;
products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the
content of the page makes you feel confusing, please write us an email, we will handle the problem
within 5 days after receiving your email.
If you find any instances of plagiarism from the community, please send an email to:
info-contact@alibabacloud.com
and provide relevant evidence. A staff member will contact you within 5 working days.