"Reading notes-data mining concepts and techniques" data preprocessing

The purpose of data preprocessing: improve the data quality, the three elements of data quality: accuracy, completeness, consistency.

Tasks for data preprocessing:

• Data cleanup
• Data integration
• Data specification
• Data transformation

Data cleansing-fill missing values, smooth noise, identify outliers, correct inconsistencies in data

• Missing values:

1. Ignore tuples
2. Manually fill in missing values
3. Use a global constant
4. Using the center measure of a property
5. Use the attribute mean or median of all samples that belong to the same class as a given tuple
6. Use most likely values (most popular)

• Noise data

1. Sub-box
2. Regression
3. Off-Group Point analysis

Data integration-consolidating data from multiple data stores

• Entity recognition issues
• Redundancy and correlation analysis
• Tuple repeats
• Monitoring and processing of data value conflicts

Data specification-The specification of the data set is represented, but still close to preserving the integrity of the original data

• Overview of Data Protocol policies

1. Dimensional regression
2. Quantity specification
3. Data compression

• Wavelet transform--linear signal processing technology, suitable for high dimensional data (HTTP://HI.BAIDU.COM/QINGSHUANGCII/ITEM/31E8831E65350DDE64EABF4C)
• Principal component Analysis--a dimensional regression method, suitable for sparse data
• Attribute subset Selection
• Regression and logarithmic linear models: Parametric data protocol
• Histogram
• Clustering
• Sampling
• Data Cube Aggregation

Data transformation and data discretization--data is transformed or unified into a form suitable for mining, easier to understand

• Overview of policies for data transformations
• Standardization
• Split-Box discretization
• Histogram analysis discretization
• Clustering, decision tree and correlation analysis discretization
• Conceptual layering of nominal data

"Reading notes-data mining concepts and techniques" data preprocessing