A Brief Introduction to Least Square Method and Its Application

Whether linear or nonlinear fitting, the least square method is the basis for using them. The mathematical principles of the least square method can be explained a lot. For example, the viewpoint of using the extreme values of Multivariate functions can be used directly, or the viewpoint of distance between vectors and sub-spaces in linear algebra can be used. Because the second method is more intuitive, I will use the second method to describe the least square method, which is more intuitive and easy to understand. Instead, I will use the first method for verification.

1. Description of Linear Algebra

For ease of expression, here are some symbols, as shown in:

Figure 1

(1) A = [a1, a2]

(2) p is the combination of base vector a1 and a2 p = x1 "a1 + x2 'a2 = Ax'

(3) e = B-p

2. Mathematical Model (At is the transpose of A, x' is the x-unplug)

Figure 2

As shown in figure 2, we solve the non-linear homogeneous equation Ax = B. Obviously, vector B is not in column Space (Col Space) of column, that is, we cannot use the column vectors of A or their linear combinations to represent B, that is, this equation has no solution.

Although we cannot solve x on the above equation, we can find its optimal solution (approximate solution) x ", from 2, we can see that the projection p of B on Col Space is the closest vector that can be expressed in column Space of.

Here is a mathematical description:

(1) p = Ax ', which can be obtained from the orthogonal nature of Figure 1. a1t (B-Ax') = 0, a2t (B-Ax ') = 0

(2) [a1t; a2t] (B-Ax ') = 0 that is, At (B-Ax') = 0

(3) arrangement (2) formula AtAx '= Atb -- core formula of least square method

(4) x' = (AtA) \ Atb -- this is written in Matlab.

For (3), a simple description is given. Originally, Ax = B is unsolvable. However, by multiplying At on both sides of the equation, we can obtain the optimal solution x '.

3. One instance

Figure 2

Obtain the optimal solution of Ax = B

(1) draw a direct y = C + Dt to fit the three points)

(2) C + D = 1, C + 2D = 2, C + 3D = 2 => [1 1; 1 2; 1 3] [C; d] = [1; 2; 2]

(3) solving the optimal solution x' = [C "; D']

AtAx '= Atb => AtAx' = [1 1 1; 1 2 3] [1 1; 1 2; 1 3] [C '; d'] = [3C '6d '; 6c' 14D']

Atb = [1 1 1; 1 2 3] [1; 2; 2] = [5; 11]

Therefore, 3C '+ 6D' = 5

6c' + 14D '= 11

=> C '= 2/3 d' = 1/2 that is, the fitting curve is y = 2/3 + 1/2 t.

4. Multivariate function extreme value verification

Here, we verify 3. From figure 2, since it is the optimal solution

(1) To minimize | Ax-B | ^ 2, I .e. | Ax-B | ^ 2 = | e | ^ 2 = 0

(2) even if e1 ^ 2 + e2 ^ 2 + e3 ^ 2 = f (C, D) (C + D-1) ^ 2 + (C + 2D-2) ^ 2 + (C + 3D-2) ^ 2 = 0

(3) evaluate C and D to make f (C, D) = 0. Therefore, evaluate the partial direction of f to C and D to 0 respectively, and obtain 3C + 6D = 5, 6C + 14D = 11.

5. Compile a Matlab program to understand the linear and nonlinear fitting of the least square method.

Data Retrieval

X	-3	-2	-1	0	1	2	3
Y	4	2	3	0	-1	-2	-5

Least Square Fitting

(1) fit with y = a0 + a1 * x + a2 * x ^ 2

% The primary function clearx = [-3-2-1 0 1 2 3] '; y = [4 2 3 0-1-2-5]'; % obtain the coefficient matrix of the linear equation a = [zx_nh_f (x (1) zx_nh_f (x (2) zx_nh_f (x (3) zx_nh_f (x (4 )) zx_nh_f (x (5) zx_nh_f (x (6) zx_nh_f (x (7)]; B = y; A = A' *; B = A' * B; c = a \ B % to obtain the fitting function x_n =-3:0.; y_n = c (1) * 1 + c (2) * x_n + c (3) * x_n. ^ 2; plot (x, y, '*', x_n, y_n) grid

Function f = zx_nh_f (x) % sub-function zx_nh_f % ZX_NH_F Summary of this function goes here % Detailed explanation goes heref (1) = 1; f (2) = x; f (3) = x ^ 2; end

Expected result

C =
0.6667
-1.3929
-0.1310

<喎?http: www.bkjia.com kf ware vc " target="_blank" class="keylink"> VcD4KPHA + signature + CjxwPtb3uq/K/Signature + cjxwp133 vcd4kphbyzsbjbgfzcz0 = "brush: java;"> y_n = c (1) * 1 + c (2) * x_n; % Main Function

% F (3) = x ^ 2; % Delete

Reference:

MATLAB Numerical Analysis and Application-Song Ye Zhi

The concept of overfitting was previously just a rough view, but I didn't have a deep understanding. I want to take a good look at it later. So here I will not analyze it below, and I will write it later. The FPGA Project will be implemented after the holiday, and the progress of the FPGA project will also be written ~