least squares and its implementation in Python

The least squares least square method, as the basis of the categorical regression algorithm, has a long history (presented by Mari Le Jende in 1806). It matches by minimizing the squared error and finding the best function of the data. By using the least squares method, the unknown data can be easily obtained, and the sum of the errors between the obtained data and the actual data is minimized. The least squares can also be used for curve fitting. Other optimization problems can also be expressed by minimizing the energy or maximizing the entropy using the least squares method.
What is the least squares? Don't worry, let's start with a few simple concepts.
Assuming that we now have a series of data points, then the estimate obtained by the fitted function h (x) given by us is, so how do we evaluate how well the fitting function we give to the actual function to be solved is relatively high? Here we define a concept: residuals, and we estimate that the degree of fit is based on the residuals. Here are three more norms:
? ∞-norm: Maximum of the absolute value of the residuals, which is the maximum of the residual distance in all data points
? 1-norm: absolute residuals and, that is, the sum of all data point residuals
? 2-norm: Sum of squares of residuals
The first two norms are the most easily thought out, the most natural, but not conducive to the differential operation, in the case of large amounts of data, the calculation is too large, not operable. Therefore, the 2-norm is generally used.
Having said so much, what does that norm have to do with fitting? The fitting degree, in layman's terms, is the similarity between our fitted function h (x) and the function y to be solved. Then the smaller the 2-norm, the more natural similarity is higher.


Thus, we can write the least squaresdefinitionThe
for the given data, the H (x) ∈h is solved in the assumed space H, so that the 2-norm of the residuals is minimized, i.e.

Geometrically, it is looking for a curve y=h (x) with the minimum distance squared from the given point. H (x) is called the fitted function or least squares solution, and the method of solving the Fitted function h (x) is called the least squares of curve fitting.


So what exactly should H (x) look like here? In general, this is a polynomial curve:


Here H (x,w) is an n-th polynomial and w is its parameter.
In other words, the least squares is to find such a group, making the smallest.

so how to find such a W, so that the fitting function h (x) and the target function y have the highest fitting degree? That is the least squares how to solve it, this is the key AH.
Suppose our fitting function is a linear function, namely:


(Of course, it can be a two-time function, or a higher-dimensional function, just as a solution example, so the simplest linear function is used)
So our goal is to find such a W,


Here the squared loss function for the sample
Students who have studied calculus should be more clear, this is a typical problem of solving extreme values, only need to find the partial derivative, and then the partial derivative is 0, you can solve the extremum point, namely:

Then we just need to solve this equation set to solve the w_i value.




============ Split Segmentation =============
Above we explain what is the least squares, and how to solve the least squares solution, we will use Python to achieve the least squares.
Here we select the target function as Y=sin (2πx), overlay a normal distribution as noise disturbance, and then use the polynomial distribution to fit it.


Code:

# _*_ Coding:utf-8 _*_# Author: yhao# blog: http://blog.csdn.net/yhao2014# email: [Email protected]import NumPy as NP # introduced Numpyim Port scipy as Spimport Pylab as Plfrom scipy.optimize import leastsq # introduces least squares function n = 9 # polynomial number # objective function Def real_func (x): R     Eturn Np.sin (2 * np.pi * x) # polynomial function def fit_func (p, x): F = np.poly1d (P) return f (x) # residual function def residuals_func (p, y, x): ret = Fit_func (p, x)-y return retx = np.linspace (0, 1, 9) # randomly selects 9 points as Xx_points = np.linspace (0, 1, 1000) # when drawing  Continuous Point y0 = Real_func (x) # target function y1 = [Np.random.normal (0, 0.1) + Y for y in y0] # Add function after the noise is being too distributed p_init = NP.RANDOM.RANDN (n) # random Initialization polynomial parameter PLSQ = LEASTSQ (Residuals_func, P_init, args= (y1, x)) print ' Fitting Parameters: ', plsq[0] # output fitting parameter Pl.plot (x_p Oints, Real_func (x_points), label= ' real ') Pl.plot (x_points, Fit_func (plsq[0), x_points), label= ' fitted curve ') Pl.plot (x, Y1, ' Bo ', label= ' with Noise ') pl.legend () pl.show ()

Output Fitting Parameters:


The image is as follows:

From the image, it is obvious that our fitting function is over-fitted, and we try to reduce the phenomenon of overfitting by adding a regularization term on the basis of the loss function:

To do this, we only need to add lambda^ p in the residuals function after the returned array

regularization = 0.1  # regularization coefficient lambda# residual function def residuals_func (p, y, x):    ret = Fit_func (p, x)-y    ret = np.append ( RET, NP.SQRT (regularization) * p)   # adds lambda^ p to the back of the returned array return    RET

Output Fitting Parameters:



The image is as follows:


It is obvious that the fitting objective function can be better under the proper regularization constraints.


Note that if the coefficients of a regularization item are too large, it can result in an under-fitting phenomenon (particularly high penalty weights at this time)
For example, when setting regularization=0.1, the image is as follows:

The apparent lack of fit at this time. Therefore, it is prudent to choose the regularization parameters.

Reference Documentation:

Using Python for machine learning-least squares

IamccmeMachine Learning Classical algorithm-----least squares-----least squares method of machine learning classical algorithm

least squares and its implementation in Python