Chapter II: Data Mining Overview

Data mining refers to the non-trivial process of automatically extracting useful information hidden in data from data collection, which is represented by rules, concepts, laws and patterns, etc.

2.1 Development History of data mining

.....

2.2 Key differences between data analysis and data mining

Compared with the traditional statistical analysis technology, data mining has the following characteristics:

• Data mining is good at processing big data (dozens of millions of rows or more)
• Data mining is often used in practical applications with data mining tools
• The trend in data analysis applications is to crawl data in large databases

Data mining is the extension and development of statistical analysis technology

Differences in data mining and statistical analysis:

One of the basis of statistical analysis is probability theory, statistical analysis of data needs to make assumptions about the relationship between data distribution and variables, determine what probability function to describe the relationship between variables, and how to test the statistical significance of the parameters, data mining applications do not need to make any assumptions about the data release, The algorithms in data mining automatically look for the relationship between variables, and the relative Yu Hai data mining has obvious application advantages.

Statistical analysis often manifests in the prediction as one or a set of function relations, the data mining in the prediction application focus on the prediction results, many times do not produce a definite functional relationship from the results, sometimes do not know those variables work, how to function

2.3 Key mature technologies of data mining and the main application in data operation

2.3.1 Decision Tree: is a very mature, widely used data mining technology, modeling process similar to the growth process of the tree, the analysis of the data sample first set into a root, through the layers of branches, eventually formed N nodes, each node represents a conclusion

The 3 most commonly used decision tree algorithms are

• CHAID (Chi-square automatic mutual relationship detection): According to the local optimal principle, using chi-square test to select the most influential independent variables of the corresponding variables, the premise is that the dependent variable is a category variable
• CART (Classification and regression tree): Based on the overall optimal principle, using the Gini coefficient and other non-purity indicators
• ID3 (including c4.5\c5.0)

The difference between chaid and cart:

Chaid local optimal principle, using chi-square test to select the most influential independent variables of the corresponding variables

Cart based on the overall optimal principle, the use of Gini coefficient, etc.

2.4 Characteristics of data mining applications in the Internet industry

Chapter II: Data Mining Overview