Data Mining Overview

Recently read more about the big data processing aspects of knowledge, but for example, hadoop,spark,storm and other platforms are mostly for data storage and management operations, not for data analysis and processing. So here is another kind of data processing, data mining. Learning data mining is also very accidental, first of all I have been doing is the data aspects of the work, data mining is equivalent to the next step after the data processing, learning the basic knowledge of information mining, to understand some commonly used data mining algorithms, for me is also a good thing.

Since my current level is just an entry level, I'll simply talk about the basic concepts of data mining. Data mining, in English, is: The data miming, also known as KDD, the re-discovery of knowledge, data mining, as the name implies, is to find out from the information for people, meaningful things. Data mining is ubiquitous, the most common is when shopping online, people will recommend some of the products that may interest you. Professionally speaking, this is called BI (Business Intelligence). There are many, for example, banks using data for fraud detection. Here are the general steps for data mining:

1. Data cleansing (data preprocessing)

2. Data integration (data preprocessing)

3. Data selection (data preprocessing)

4. Data Transformation (data preprocessing)

5. Data Mining

6. Model evaluation

7. Knowledge representation

The first four steps are also used as data preprocessing operations. Data preprocessing has many effects, such as the removal of noise data or the processing of outliers, as well as the normalized operation of data. Perhaps you will ask, where our huge data exist, like the general system, is in the relational database, but at this time there is a problem, data mining for the data is often very large demand, which requires a lot of data, so we are called Data Warehouse concept, Organize a number of databases to form a data warehouse, and then we will analyze the Data Warehouse OLAP contact analysis processing. The Data warehouse is a data cube in the form of data representation of the situation.

What is digging data

What does data mining dig up, first one is the frequent pattern of mining, this is good to understand, here involves a lot of frequent itemsets mining algorithms, such as the Apriori algorithm, there are many correlation and correlation elements. There is a mining often mentioned things called classification , classification algorithm in the data mining is also very important, is often said to be the Bayesian classification algorithm, based on the probability of statistical algorithm, and then on the basis of the classification algorithm has emerged cluster analysis, there is the back of the K-means algorithm, K-Center point algorithm. For the previous 2 large modules of the mining system, there are corresponding to the advanced stage of mining analysis. For more specific data formats and more complex environments, there are different mining algorithms and methods.

the prospect of data mining

The future will certainly be a data explosion of the era, data mining will be a very hot field, he is a multidisciplinary interdisciplinary field, machine learning, neural network, statistics, for all walks of life will play a very important role.

Data Mining Overview