Preprocessing in data mining--taking the problem of telecom customer churn as an example _ data mining

Source: Internet
Author: User

Data preprocessing

STEP1: Data sampling: Because in the establishment of customer churn model process, the loss of customers often accounted for the proportion of all customers are very small, at this time, the best way is to retain the entire loss of customer population, but not the loss of customer population sampling, so that customer churn and non-customer churn in the 1:1~1:2

STEP2: Data Discovery (missing value and outliers value)

STEP3: Create the missing variable indicator; For each missing variable, the value is missing, the value is 1, and when it is not missing, it is 0, which creates the corresponding missing variable

STEP4: The sample data is divided into training sets and test sets

Strata ins can guarantee that the number of customers lost in the training set and test set is roughly the same as the number of non-lost customers

Actually think of it as a training set, a correction set, and a test set. The training set is used to establish the model, the correction set is used to adjust the parameter, prevent the fitting and so on, the test set is used to judge the model.

STEP5: Fill in the training data set, output data train1 (fill with mean value)

STEP6: Attribute variable compression (occupation occupation):

First, we get the ratio or probability of each level of the attribute variable, and then we can cluster the relative similarity according to the probability.

But the card side value is getting smaller (the total card side value is a constant, each time, the card side value reduces some), until finally divides into a class, the card square value changes to 0; so we need to find a balance between the degrees of freedom and the value of the card square.

STEP7: Continuous variable compression (elimination of collinearity)

STEP8: Variable selection: Eradication of Spearman and hoeffding correlation coefficients to select some variables with very weak predictive power

STEP9: According to the definition of logistic regression, we can know that logit (p) and the independent variable is a linear relationship, so we also need to do some processing of the numerical variable (the specific operation is: The value of the variable is divided into 100 groups, the average of each group of mean_x, To see if mean_x and Logit (p) is a linear relationship, is not a linear relationship, directly replaced with 1,2,3,4 ... 100)

STEP10: Model development: Building related models

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.