Linear regression Linear regression (4) Local weighted regression

This article introduces the concepts of fitting and under-fitting, and introduces local weighted regression algorithms.

Over fitting and under fitting

Before in linear regression, we always put the individual x as our characteristic, but in fact we can consider that even the higher times of x as our characteristics, then we will get through linear regression is a multiple function.

We can imagine that when we only use X as our feature, our data may actually be presented as a two-time function, that is, our assumptions have some problems, then we will find the least square value is quite large. But if we have 10 sets of samples, and we choose a 10-dimensional feature, then we can imagine that the final curve we get will be a curve that passes through all 10 sets of samples, meaning that the value of the least squares is directly equal to 0.

You can see that the graph on the left and the graph on the right have some problems, but the two-time model hypothesis is what we want.

So, for a supervised learning model, too small a feature set makes the model too simple, too large a feature set makes the model too complex .

For cases where the feature set is too small, it is called under-fitting (underfitting);

For cases where the feature set is too large, it is called overfitting (overfitting)

Ways to solve this type of learning problem:

1) Feature Selection algorithm: A class of automation algorithms that select the features used in such regression problems

2) Non-parametric learning algorithm: Mitigating the need for selecting features, and leading to local weighted regression

Local weighted regression

Local weighted regression is a non-parametric learning algorithm, and the non-parametric learning algorithm is defined as an algorithm in which the number of parameters increases with the M (training set size) . Usually defined as the number of parameters increases linearly with M. In other words, what the algorithm needs will grow linearly with the training set, and the maintenance of the algorithm is based on the entire training set, even after learning.

Local weighted regression means: For a certain query point x, we have to do a local weighted regression to obtain the corresponding θ, the algorithm flow is:

finding the right Θ, minimizing , is our linear regression y for the current x query.

Which means that for each sample weight, there can be a variety of values,

A relatively standard and fair choice is:

is called the wavelength function, which controls the rate at which weights fall with distance. The smaller the bell, the narrower the Ω decays, and the larger it becomes, the slower it decays.

Omega can simply see that when the sample is closer to the inquiry point, the weight is the greater, the opposite the farther the smaller.

The pros and cons of this algorithm can be easily seen:

The advantage is that the local weighted regression alleviates the need of feature selection, and alleviates the problem of less fitting in a great extent.

The downside is that every time there is a new query, we have to iterate over the sample and re-converge the parameter θ, which can be very slow for efficiency.

Linear regression Linear regression (4) Local weighted regression