Preprocessing in data mining--taking the problem of telecom customer churn as an example _ data mining

Data preprocessing

STEP1: Data sampling: Because in the establishment of customer churn model process, the loss of customers often accounted for the proportion of all customers are very small, at this time, the best way is to retain the entire loss of customer population, but not the loss of customer population sampling, so that customer churn and non-customer churn in the 1:1~1:2

STEP2: Data Discovery (missing value and outliers value)

STEP3: Create the missing variable indicator; For each missing variable, the value is missing, the value is 1, and when it is not missing, it is 0, which creates the corresponding missing variable

STEP4: The sample data is divided into training sets and test sets

Strata ins can guarantee that the number of customers lost in the training set and test set is roughly the same as the number of non-lost customers

Actually think of it as a training set, a correction set, and a test set. The training set is used to establish the model, the correction set is used to adjust the parameter, prevent the fitting and so on, the test set is used to judge the model.

STEP5: Fill in the training data set, output data train1 (fill with mean value)

STEP6: Attribute variable compression (occupation occupation):

First, we get the ratio or probability of each level of the attribute variable, and then we can cluster the relative similarity according to the probability.

But the card side value is getting smaller (the total card side value is a constant, each time, the card side value reduces some), until finally divides into a class, the card square value changes to 0; so we need to find a balance between the degrees of freedom and the value of the card square.

STEP7: Continuous variable compression (elimination of collinearity)

STEP8: Variable selection: Eradication of Spearman and hoeffding correlation coefficients to select some variables with very weak predictive power

STEP9: According to the definition of logistic regression, we can know that logit (p) and the independent variable is a linear relationship, so we also need to do some processing of the numerical variable (the specific operation is: The value of the variable is divided into 100 groups, the average of each group of mean_x, To see if mean_x and Logit (p) is a linear relationship, is not a linear relationship, directly replaced with 1,2,3,4 ... 100)

STEP10: Model development: Building related models