Concepts related to data mining

1. Differences between statistics and data mining:

Statistics mainly uses probability theory to establish mathematical models. It is one of the common mathematical tools used to study random phenomena.

Data Mining analyzes a large amount of data, discovers internal links and knowledge, and expresses this knowledge using models or rules.

Although some analysis methods (such as regression analysis) are the same, data mining and statistics are essentially different:

A major difference lies in the scale and nature of processing objects (datasets. Data mining often faces databases of GB or even TB magnitude, but it is difficult to process such large data sets using traditional statistical methods. Traditional statistical processing often collects data for specific problems (or even optimizes it through experimental design) and analyzes data to solve specific problems. Data Mining is often a secondary process of data analysis, the data it uses may not have been specially collected for the current research, so its applicability and pertinence may not be strong. In the process of data mining, we need to pre-process abnormal data and conflicting fields to improve the data quality as much as possible, and then perform data mining for the pre-processed data.

Another difference is that, in the face of complicated massive data structures, data mining often requires a variety of mathematical models and mathematical tools other than traditional statistics to establish models or rules that are most suitable for describing objects.

In short, the hypothesis test (or significance test) method is often used in biomedical research. It focuses on the hypothesis-driven (hypothesis-driven), that is, the hypothesis is proposed and tested; data mining does not have such a function. It is mainly data-driven, that is, discovering rules from data and gaining knowledge.

2. Data Mining Classification

Data Mining is divided into two types: prediction and descriptive. Prediction data mining uses known results from historical data to deduce or predict the possible values of unknown data. A descriptive model is a pattern or link used to identify data. It aims to explore the inherent nature of the analyzed data. Different data mining methods can be used based on the nature of objects and the specific problems to be solved.

Predictive data mining methods include classification, regression analysis, and time series analysis. Descriptive data mining methods include clustering) association rule analysis and sequence analysis.