Data quality analysis
Dirty data includes: missing value, outliers, inconsistent values, duplicate data and data containing special symbols;
1. Missing value handling
Statistical missing rate, missing number
2. Outlier Handling
(1) Simple measurement analysis
(2) 3Q principle
In the case of normal distribution, the small probability event is an outlier
If the non-obedience is too distributed, it can be analyzed by the standard deviation from the original average number of times.
(3) Box line diagram analysis
Use describe () to describe
Main data exploration functions
Summary of 1.Pandas Common functions
Import data
Exporting data
View, check Data
Data selection
Data cleanup
Data processing: Filter, Sort, and GroupBy
see which values are specificids = [1,4,3,3,4,2,3,4,5< 6 , 1 c22/> ] news_ids = list(set(ids))news_ids.sort(ids.index)
Data consolidation, data statistics
2. Expand statistical Feature functions
Cumulative statistical Feature function
- Cumsum: The number of the previous 1-n is given in turn
- Cumprod: The product of the number of previous 1-n is given in turn
- Cummax: The maximum value of the number of previous 1-n is given in turn
- Cummin: The minimum value of the number of previous 1-n is given in turn
- Rolling_sum (): Sum (by column)
- Rolling_mean (): Arithmetic average
- Rolling_var (): Variance
- ROLLING_STD (): Standard deviation
- Rolling_corr (): Correlation coefficient matrix
- Rolling_cov (): Covariance matrix
- Rolling_skew (): skewness (third-order moment)
- Rolling_kurt (): kurtosis (four-order moment)
Python data exploration