"Linear algebra" least squares and projection matrices

In the previous article, "Orthogonal projection", we talked about orthogonal projections, and now we look at our familiar least squares from the perspective of orthogonal projection. I remember the first time I knew that the least squares method was in a freshman class, so let's start by looking at the least squares. 1, least squares recently machine learning compared to fire, machine learning many algorithms are classified information, such as support vector machine is based on known information to classify, neural network can find the relationship between input and output (of course, cannot give specific mathematical expression), both of these algorithms can find the relationship between input and output, Classification and regression are always complementary. Have time to write about the algorithms for machine learning.        To get to the bottom, the function of least squares is to find the relationship between input and output from a set of data. Principle:
The empirical equation is y=f (x), the equation contains some undetermined coefficients an, give the real value {(xi,yi) |i=1,2,... n}, the x, y values into the equation and then make the difference, can describe the error: Yi-f (xi), in order to consider the overall error, can take the sum of squares, The reason for the square is to consider that the error can be negative direct addition can cancel each other, so the memory error (note the choice of the error function there are many kinds, we choose the typical error function) is:

E=∑ (Yi-f (xi)) ^2

It is a multivariate function, having an n unknown amount, which is now required to be the minimum value. So it is necessary to satisfy the bias of each variable equals 0, so we get n equations:

n equations determine that n unknowns are constants that can be solved theoretically. The method of regression equation with this error analysis method is the least square.

2, least squares and projections

I do not like to see these theories, formula derivation, and more like to use examples to show the idea of the algorithm. For example, in a two-dimensional coordinate system, there are three points, (2,2), (3,2), how do you fit these points with a straight line?

First, we can assume that the line expression is as follows:

Then calculate the error function:

The error function e is obtained by the deviation of the coefficient, a, a, and the 0:

The value of the coefficients, a, a, a, and a linear expression is obtained by the above formula:

The curves obtained by the least squares are as follows:

Linear algebraic approach to least squares: Similarly, the expression of a fitted line is set to:

The purpose of fitting is to make the data points satisfy the above function expressions, namely:

The matrix is represented as follows:

The above formula through the Gaussian elimination of the element, you can find that there is no solution! We can find that the value of AA on the left side of the equation is a linear combination of the individual column vectors in matrix A, and if Aa=b has a solution, B must be within the column space of Matrix A. In the above example, the vector on the right is obviously not in its column space, so the equation has no solution. The idea of least squares is to find a vector p in the column space of matrix A, so that the error of P and B is minimized. Let's ask for B:

Aa=p is definitely solvable, because P is in the column space of Matrix A. To make the e vector the shortest length, when and only if p is a projection of B on the matrix column space! The general formula of the projection matrix in orthogonal projection is available:

Then aa=p the P into the formula can be:

To substitute a specific value into:

You can get:

The b,p,e vectors can be represented as follows:

The p,b is represented in the figure as follows:

"Linear algebra" least squares and projection matrices