Get out of the misunderstanding of Data Mining

Data mining is an extraordinary process that identifies effective, novel, potential, useful, and ultimately understandable patterns from a dataset. It uses multiple disciplines such as statistical analysis, data warehouse, artificial intelligence, and information science to identify customer behavior patterns that are hard to gain insight from traditional business analysis methods, provide a detailed description of the customer and effectively predict the customer's behavioral trend. Currently, data mining is widely used in telecom operators. However, as the application goes deeper, deviations and misunderstandings frequently occur, including operations, processes, ideas, methods, and tools. The author tries to find out some common technical and business problems from the classic Data Mining Theory and the data mining practices of telecom enterprises, and provides methods to avoid or solve these problems.

◆ Data Mining: a perpetual motive behind it

Are we always motivated? No, but in this case, "Data Mining is omnipotent. It can calculate everything you want, as long as there is data ." -- This sentence is the theory of permanent motivation. In the classic theory of data mining, the model does not exist. When the model is not obvious, it is almost the same as the random distribution. Translated into the telecom language: if the data in the lost customer's system is no different from the features of the non-lost customer, the loss is unpredictable. There is only one way to avoid such errors: Understanding the classic theories and basic concepts of data mining.

◆ Hit rate Myth: uncover the mystery of the effect

When classification technology (decision tree and Neural Network) is used for prediction of loss, many engineers will promise a hit rate with Party. The hit rate concept is the percentage of real lost customers in the next month in the lost customer list provided by engineers. The higher the value, the better. This is because the prediction made by data mining software is actually a probability decision, and there must be wrong judgments. The exact description should be the hit rate within a certain percentage. For example, if Taobao has 0.1 million customers, the 3000-person list accuracy, 5000-person list accuracy, and 10 thousand-person list accuracy are scientific. Enterprises need to be realistic and determine an expected rate of coverage based on their business needs.

◆ Software and tools must be bought: Creating Chinese atomic bombs

Some business managers believe that software tools must be developed by large companies. They are complex, professional, and authoritative. However, whether it is suitable for the enterprise or the business that adapts to the research is not within the scope of the manager's key consideration. If we often move through professional data mining working groups, we will find that the real masters do not use commercial data mining software. Even if they are used, they can also be written using sasem.CodeSoftware. Their approach is to explore data and find the right one.AlgorithmSimple coding implementation, running data, and writing reports, because the depth of the application layer and the improvement of the effect are the most important. China can create its own atomic bombs. What about operators?

◆ Benchmark of engineers: a tool used by the smart database to speak at the conference

When a company recruited data mining engineers, there were a large number of applicants who claimed to have been engaged in data mining for 20 years. However, in fact, it was only 10 years for domestic researchers to study data mining. Many people think that statistics are also data mining. How can we measure the real strength and level of data mining engineers? First, let's look at his theoretical basis. Second, let's look at his methods. Engineers who do not have sufficient mathematical model capabilities, statistical analysis capabilities, and logical analysis capabilities are unqualified. Third, it depends on his innovative ability. In a word, a good data mining engineer can rely on a smart database. A less-excellent data mining engineer is just a talking tool.

◆ The model does not need to be updated: the flow is not corrupted, and the account is not informed

Some people say that when the model is ready, there is no need to update it. The Chinese saying "no corruption in the water, no corruption in the household" is not suitable for data mining. Customers are mobile, businesses are changing, policies are rolling, competitors are cunning, and the economy is fluctuating. Is the model fixed? No, a series of changes make the customer's behavior model change, and the model should also be updated in a timely manner. Otherwise, the loss of business opportunities will become mediocre.

◆ Same algorithms: suits and sneakers

Many people who only know the data mining technology will make a big splash: algorithms are the same. Isn't it just Clustering or pre-testing? We all know that wearing a suit cannot wear sneakers. Because of non-matching, in data mining practices, algorithms are rigid. No algorithm can be integrated, and the applicability of algorithms is very important, for data with many noisy points and most of them are singular points, it is best to use DBSCAN instead of K-means algorithms. If the data is distributed in a spherical cluster with a dumbbell-shaped adhesion cluster, k-means is far better than DBSCAN.

◆ Data mining is software: A brain conductor or a hand-directed brain

May someone say that data mining is not a software? Management Science has always been the best test ground for advanced technologies and concepts. From ERP, MRPII, OLAP, CRM (Customer Relationship Management) to DM (data mining), we will have such experience, management is not a software, but a system is more than a software. Once the CRM software is used, the customer relationship is improved. Once the data mining software is installed, the model is run, and it is not a real data mining system. Software tools are hands, while engineers are brains. What we need is a smart human brain that directs competent hands rather than hands-on or hands-directed brains. The effectiveness of data mining software is embodied in the wisdom of users.

◆ Statistics will be mined: Tigers will not climb trees

Some people say that statistics are very good, and mining is very good. First, we acknowledge that statistics is closely related to data mining. However, there is no special relationship between the two. In each development phase of data mining, only one subdirection development can be counted in the credit of statistics, that is, Bayesian Network. We hope that the statistical regression standard is neither too high nor too low: data mining engineers are capable of statistical analysis, statistical analysis is one of the basic capabilities and steps of data mining. Cats and tigers look really like each other, but tigers cannot carry trees.

◆ Business is more important than mining results: No doubt about the use of people, no doubt about the use of people

There is such a gossip network. After the mining results come out, there will be some deviations from the business experience. For example, there is a conflict in the model understanding. The business personnel believe that the results are incorrect and still revise the results according to the business experience.

The saying goes: people are not suspicious, and people are not suspicious. Since the cost is spent on data mining, we need to respect objective mining results. If there is a conflict with business experience, we need to check the various mining steps to find out where the problem is; if there is no problem, it is a kind of demeanor and wisdom that the past business has misunderstood and dared to face the reality and correct the previous deviations.

◆ Data mining is useless: Huainan Huaibei, orange and orange are different

Some render networks have completed data mining projects and found that the results are not good. Therefore, it is concluded that data mining is useless. In fact, the city of Huainan is orange, and the city of Huaibei is orange. The essential difference lies in the difference between 59 points and 60 points, but the two results are different. The success of Data Mining requires many conditions: first, a good data environment; second, excellent engineers; third, software tools to adapt to the target dataset; fourth, the analysis target dataset has non-random pattern information that can be mined. The method to solve the problem of data mining effectiveness is to check the steps of data mining.