Over-fitting (overfitting)

We have solved a theoretical problem before: can machine learning work? Now to solve another theoretical problem: overfitting.

As we have seen before, many times we have to nonlinear transform. But we can't determine the value of Q. Q too small, then the Ein will be very large, q too large, there will be over-fitting problems. As shown in the following:

So, what are the factors that overfitting specifically affected?

Now we have two other examples:

The data source for the first example is: a 10-th target function +noise; The second example is the data source: a 50-th objective function. Now we use the 2-th function (H2) and the 10-th function (H10) to fit two examples respectively. Let's predict the outcome.

I think: For these two examples, the H10 effect will be better. Because either for the first example or for the second example, there is no overfitting problem on the order of H10.

Here's the real result:

We can see that for two examples, the H2 effect is the best .

With this counterintuitive example, we can get a glimpse of overfitting.

Through this learning curve, we can see thatH10 can result well, but it is based on N is large enough, if N is very small, it is the result of H2 good !

Add one point: For the second example, obviously there is no noise, why H10 performance than H2 it?

because the complexity of 50-th is too high, H10 and H2 can not be accurately fitted. At this point the complexity of the objective function is equivalent to noise for H2 and H10.

We now think that the complexity level Q of data points N, noise, and target functions will affect overfitting.

Detailed instructions are described below.

From this objective function, the data is generated and then fitted with H2 and H10. When do we say over-fitting is going to happen? When the eout obtained by using H10 is greater than the eout obtained by using H2, it is inevitable that overfitting will occur, i.e. Overfit measure:eout (G10)-eout (G2).

For the first picture, q=20. We can easily see: 1) The smaller the n, the larger the noise, the more prone to overfitting.  2) when n is very small (here is n<80), it will inevitably occur over fitting; 3) when n is large, the larger the noise, the more prone to overfitting. N plays a decisive role.

For the second picture, the noise is fixed. We can easily see: 1) n about small, the larger the Q, the more prone to overfitting; 2) when n is very small, it is almost inevitable that overfitting will occur;

The first picture and the second picture are different: 1) for a red area at the lower left, it is certain that overfitting will occur because the order of the target function is small enough. But why does it not over-fit as n increases? 2) When the order is large enough, n is very small, over-fitting occurs, but once n is large enough (here n>100), there is no overfitting.

We turned noise into stochastic noise, and Q became deterministic noise.

Several influencing factors of overfitting, N (most important), Noise,q.

How to solve overfitting problem?

We have overfitting a car accident, the cause of the accident may be: driving too fast, there are a lot of holes in the road, there are too few signs on the road. This corresponds to the overfitting reason: DVC too large (q too Large), noise too much, the amount of data is too small.

How to avoid "the accident"? You can drive slower, avoid potholes on the road, or get some road markings. or step on the brakes and look at the dashboard more often .

Drive slowly: start with simple model;

Avoid road bumps: Data cleaning: Correction of the noise, pruning: Delete the data with noise.

Get more road signs: Get more data (which may not be possible), or use data hinting technology;

Brakes: regularization;

look at the instrument panel: validation.

The latter two will be described in detail later.

Over-fitting (overfitting)