A brief description of the basics of integrated learning

A brief description of the basics of integrated learning

What is integrated learning?

Integrated modeling is an important way to optimize the performance of the model. In general, it is worthwhile to apply an integrated learning approach to many of the models you build, and people are proving it again and again. They tend to use integrated learning in a data modeling contest such as Kaggle and benefit from it.

Integrated learning is actually a very broad concept, it is a definition of the benevolent see of the beholder. Next, this article will cover some of the basic concepts and ideas of integrated modeling, which should be enough to get you started to build your own integration model. At the same time, as always, we will try to make them look simpler.

First, let's take a quick look at the basics of integrated learning in one example. This example will tell us that in fact we do integrated modeling almost every day, but most of us are not aware of it.

For example: I want to invest in XYZ company, but I am not quite sure how it is performing. So I'm going to see if this company's stock price growth rate will be more than 6%, in order to get a conclusion, I decided to consult different areas and levels of people:

1. An employee of XYZ company: This person knows the internal workings of the company and has insider information about the company's operating mechanism. But he knows little about the company's exposure to external competition and industry innovation, and does not know how technological advancements will affect XYZ company's product evolution. According to historical experience, his judgment has a 70% correct rate.
2. XYZ Company's financial advisor: This person has a considerable insight into the overall competitive strategy of the enterprise in the market. However, he does not know exactly how the company's internal policies are coordinated. According to historical experience, his judgment has a 75% correct rate.
3. A stock trader: This person has a three-year observation experience of XYZ Company's stock, and he has a deep understanding of the seasonal trend in stock price movements and the performance of the overall market. At the same time, the person is more sensitive to the changes in stock prices. According to historical experience, his judgment also has a 70% correct rate.
4. An employee of the XYZ company's competitor: This person knows the internal operating mechanism of the competitor, which means he may know something about the impending impact of XYZ company. But he does not understand the dominant factors that affect the development of both companies. According to historical experience, his judgment is only 60% correct.
5. A market research team that belongs to the same class: The team makes a cross-cutting and trending analysis of the customer performance of XYZ Company's products, but because it focuses solely on the analysis of the customer on the one hand, the team is not aware of some of the changes that XYZ company itself could make. According to historical experience, their judgment has a 75% correct rate.
6. A social media expert: This person can help us understand XYZ Company's product market positioning and the changing trend of customer sentiment, but he is not aware of the many details beyond data marketing. According to historical experience, his judgment has a 65% correct rate.

This is all the consulting channels we have, and we may think of integrating this information to make a sensible decision.

Assuming that all predictions are independent, then now, when 6 experts agree that investing in XYZ company is a good decision, the joint accuracy of this decision is

1-30%*25%*30%*40%*25%*35%

= 1-0.07875 = 99.92125%

Suppose: the assumption that our predictions are independent of each other may be extreme, because the predictions that people make are more or less connected. Nevertheless, we can see that when we combine many predictions, we can become so sure.

Let's change this setting a little bit. This time we have 6 "experts" who are employees of XYZ company who work in the same department, and each of them has a 70% chance of making the right judgments about my investment.

So now, if you combine these "expert" suggestions, can I increase my confidence level to more than 99%?

Obviously not, because the predictions they make are based on very similar information. These employees ' forecasts are bound to be influenced by similar information, and the difference between their recommendations is most derived from their personal opinions about the company and some of the gossip they collect in the company.

What exactly is integration learning integrating?

Integration is the combination of many types of learning algorithms (independent models) to achieve the stability and prediction of the model to improve the ability of a learning. In the example above, the way we combine the predictions of all experts can also be called an integrated learning.

In this article, we will discuss some of the integrated technologies that are widely used in the industry. Before we talk about technology, let's look at how we've built so many learning algorithms. From the overall to the model-based approach, the differences between models can come from a variety of reasons.

The following are four reasons for the differences in the model, and the combination of the following factors leads to the difference between the models ' days:

1. Study of the overall difference

?

2. Model assumptions are different

3. Different modeling techniques

4. Different initial seed

Error (variance and bias) in integrated learning

Errors in any model can be disassembled into three elements by mathematical method, as follows:

This equation is extremely important for the following, to understand the mysteries behind the integration model, we first have to understand the cause of the error caused by the model. We will briefly introduce these errors and understand each of the integrated learning algorithms in this area.

Bias (Bias error) is used to quantify the mean of the difference between the predicted value and the actual value, and the advent of high errors means that our model may miss out on important trends, that is, the model's performance is not optimistic.

Variance (Variance) is the quantification of the degree of dispersion of the same individual predictions, and an overly generous model will over-fit your training overall, making it impossible to predict accurately the individual outside of the training set. The following figure may make you more aware of this (the red area is the true value, the blue point is the predicted value):

Photo Source: Scott Fortman

Often, when you increase the complexity of the model, you will find that the decrease in bias will directly lead to a reduction in the total error of the model, but it will be counterproductive to increase the complexity of the model after a certain point. In the end, your model will face an over-fitting problem with a high variance.

Theoretically, the optimal model should keep the balance between the two types of errors as far as possible, which is known as the tradeoff between error and variance management, and integration learning is one of the ways to implement this tradeoff.

Photo Source: Scott Fortman

Some commonly used integrated learning techniques

1. Bagging (Bagging): Bagging technology means that a similar learning algorithm is implemented multiple times in a small sample population, culminating in the average output of its predicted results. In the generalized bagging algorithm, you can even use different learning algorithms for different populations. As you would expect, this helps us to reduce variance errors.

2.boosting:boosting is an iterative algorithm that adjusts the weights of observations based on the predictions of the last classification. If the classification of an observed value is predicted to be wrong, then the algorithm will increase the weight of this observation, or vice versa. Typically, boosting can effectively reduce errors and build an effective predictive model, but it also faces the risk of overfitting.

3.Stacking: This is a very interesting model merging method, in this algorithm, we often use a learning algorithm to integrate the results of other learning algorithms, and the degree of bias or variance is directly dependent on the learning algorithm we use for merging.

Conclusion

As we all know, integration technology can be applied to the problem of each Kaggle predictive modeling contest, and how to choose the right integrated learning method sometimes goes beyond the science itself, more like an art rather than a technology. Through the accumulation of experience, perhaps in the near future, people will find a knack in the study of machine learning to deal with the choice of integrated learning methods in different situations.

Original Tavish Srivastava

Translation: sdcry!!

Original link: http://www.analyticsvidhya.com/blog/2015/08/introduction-ensemble-learning/

A brief description of the basics of integrated learning