geometric meaning and projection matrix of (mathematics) least squares

Main content:

1. What is least squares
2. The geometrical meaning of least squares
3. Orthogonal projection matrix

What is least squares?

Suppose we have n-pairs of data on our hands, {(Xi,yi): I=1...N}, in order to explore the relationship between the Y-variable and the X-variable, we want to match it with a polynomial, but how are the coefficients in the polynomial determined? It's definitely not going to work, and the least squares method tells us that the coefficients of this polynomial should make the sum of squares of the errors of each point the smallest.

(Baidu Encyclopedia) The Least squares (also known as the least squares method) is a mathematical optimization technique. It matches by minimizing the squared error and finding the best function of the data. By using the least squares method, the unknown data can be easily obtained, and the sum of the errors between the obtained data and the actual data is minimized. The least squares can also be used for curve fitting. Other optimization problems can also be expressed by minimizing the energy or maximizing the entropy using the least squares method.

The geometrical meaning of least squares

The geometrical meaning of the least squares: the geometric meaning in the least squares is the projection of a vector in the high dimensional space in the low subspace space.

From the definition above, we can hardly imagine the geometrical meaning of the least squares, so we can deduce by a simple example:

We fit the line by minimizing the sum of squared errors in the definition:

The error of each point indicates:

The sum of squares of the minimum error:

To solve the minimization problem above, we can get it by derivation, preferably in the form of matrix expression: Ax=b (here x means the coefficient a above)

The result is:

The above results can be obtained easily by the solution of the super-definite equation.

First of all, the vector expression form:

The parentheses indicate: it is a linear combination of two vectors [1, ..., 1]t and [X1, ..., xn]t, in other words, it is any point of the two-dimensional subspace (thought to be a planar one) made up of both vectors.

So the geometric meaning of the above equation: the expression vector [Y1, ..., yn]t (a point in space) to the distance of any point in the two-dimensional subspace; (length of the vector)

The geometric meaning of minimizing the square of the above (the minimization of the vector length): Looking at [1, ..., 1]t and [X1, ..., xn]t a point on a two-dimensional subspace that makes the vector [Y1, ..., yn]t to this point the smallest distance. How to find this point? Just make a geometric projection. such as

As shown, given a vector u in three-dimensional space, and a two-dimensional plane composed of vector v1,v2, the vector p is u to this plane of projection, it is a linear combination of v1,v2:

Using the vertical nature of the projection, we can get two equations about the coefficient c:

Make V = [V1, v2], p = c1v1 + c2v2, combine the above to form a matrix (more easily extended to a high-dimensional space) and get:

So the expression for the coefficient C is:

Did you find it familiar? and the same as the formula has wood!!!

Well, let's go back to the original example and see what the projection points and the projected space represent in relation to each other.

Replace the u in the diagram with [Y1, ..., yn]t, and replace the v1,v2 with [1, ..., 1]t and [X1, ..., xn]t, coefficients C1 and C2 which we require A0,A1.

Therefore, the geometric meaning of the least squares is a projection of a vector (determined by Y data) in a high-dimensional space in a low subspace space (determined by the number of x data and polynomial).

Orthogonal projection matrix

The above mentioned that the geometric meaning of the least squares is the projection in the space, in fact, the projection in the linear algebra is also the existence of its mathematical formula, you can contact the following mathematical knowledge to understand the geometric meaning of the least squares.

Zhang Chengji Space:

Projection matrix for Zhang Chengji space:

Projection interpretation of the least squares:

geometric meaning and projection matrix of (mathematics) least squares