Data Mining concepts and techniques (Han Jiawei) reading notes 4--data integration and transformation

1. Issues to consider for data integration

A. Pattern integration and object matching

B. Redundancy. Reason one: can be exported with one or a set of attributes, cause two: inconsistent attribute or dimension naming.

2. Correlation detection of attribute redundancy

A. Numerical attribute calculation correlation coefficient

Description: N is the number of Ganso, and Ai,bi is the value of the property, a, and A/b in Ganso I respectively. -a,-b are the mean of A and B, then the standard deviation of a A, then the sum of the AB cross product (that is, for each progenitor, attribute A multiplied by B). Note should: -1<=r<=1, if R is greater than 0, then A/b is positively correlated. means that the value of a increases with B's worth, and the greater the value, the stronger the correlation. R=0 said he didn't want to close. R<0,ab negative correlation means that one property prevents the appearance of another property.

In addition: the correlation of two attributes does not mean that one leads to the other one.

B. Classification (discrete) data through X2, chi-square test.

Set A has r values, B has a C value, then the R value of a and the C value of B form the columns and rows of a table. Order (AI,BI) represents the event that a takes value ai,b the value Bi.

Where Oij is the observed frequency (i.e. actual count) of the joint event (AI,BJ), and Eij is the desired frequency (AI,BJ), which can be calculated as follows:

where n is the number of data ancestors, COUNT (A=ai) is the number of Ganso with value AI, and count (B=BJ) is the number of Ganso with value BJ.

3. Meta-ancestor redundancy detection repetition

Inconsistencies usually occur between various replicas, in which the input errors and portions of the updated data appear without updating all occurrences.

4. Detection and processing of data value conflicts

The representations, scales, units, and encodings of different data source properties are inconsistent. The meaning of the same name attribute may be different for different data sources.

5. Content of Data Transformation (data preparation)

A. Smooth: de-noising

B. Aggregation (unlike clustering): aggregation and aggregation more is the synthesis of the existing data to obtain new attribute values (for example: to obtain annual income).

C. Data generalization: Use concept layering to replace raw data with high concepts. For example, the country replaces the street, the youth replaces the numeric age, etc.

D. Property constructs. Building new properties is added to the property set to help the mining process, similar to aggregation, but not just summarized, but with different purposes.

Data Mining concepts and techniques (Han Jiawei) reading notes 4--data integration and transformation