Least squares least square_ least squares method

The first loss function that came into contact with a graduate student was least square. Recently studied again, make a summary of it.

The definition of wiki is enough. The formula is as follows E (W) =12∑n=1n{y−xwt}2 e (w) =12∑n=1n{y−xwt}2

Where Y y represents the Class-column vector, x x represents the characteristic row vector, and w w represents the regression or categorical parameter matrix. The Optimum w W is obtained by minimizing the Euclidean distance.

The first question I encountered was how the formula was obtained, and what the motivation was. I am personally inclined to explain it in the most plausible sense. Specifically as follows:

Suppose regression or classification model formulas are as follows: Y=wtx+ϵy=wtx+ϵ

Ε∼n (0,σ2) ϵ∼n (0,σ2) represents additive Gaussian noise, so Y∼n (wtx,σ2) y∼n (wtx,σ2). In this case, an independent observation x x is used to obtain a series of observations x= (X1,Y1) ..., (Xn,yn) x= (x1,y1) ..., (Xn,yn), then write the corresponding likelihood function p (y∣x,w,σ) =πnn=1n (wtx,σ2) p (y∣x,w,σ) =πn= 1NN (wtx,σ2)

Both sides of the natural logarithm, then ln (P (y∣x,w,σ)) =∑i=1nln (n (wtx,σ2)) ln (P (y∣x,w,σ)) =∑i=1nln (n (wtx,σ2))

Whereas n (wtx,σ2) =12πσ2√exp (− (y−wtx2) 2σ2) n (wtx,σ2) =12πσ2exp (− (Y−WTX) 22σ2)

So ln (P (y∣x,w,σ)) =−12σ2∑n=1n{yn−wtxn}2−12ln (2πσ2) ln (P (y∣x,w,σ)) =−12σ2∑n=1n{yn−wtxn}2−12ln (2πσ2)

Maximum likelihood function, solving W, W∗=argminw−12σ2∑n=1n{yn−wtxn}2−12ln (2πσ2) w∗=argminw−12σ2∑n=1n{yn−wtxn}2−12ln (2πσ2)

The second item in the formula has nothing to do with W W, it can be omitted, so w∗=argminw−12σ2∑n=1n{yn−wtxn}2 w∗=argminw−12σ2∑n=1n{yn−wtxn}2

Take off the σ2σ2 in the upper-style, which is the least squares we are familiar with.

When solving, we seek for the deviation of logarithmic likelihood function (the rule of the derivation of the attention matrix)

∇ln (P (y∣x,w,σ)) =−∑nn=1{yn−wtxn}xtn∇ln (P (y∣x,w,σ)) =−∑n=1n{yn−wtxn}xnt order 0, there are ∑n=1nynxtn=wt∑n=1nxnxtn∑n=1nynxnt= Wt∑n=1nxnxnt

The inverse of the matrices on both sides are: ∑NN=1XNYTN=∑NN=1XNXTNW∑N=1NXNYNT=∑N=1NXNXNTW

If Y y represents a class-scale matrix, and X x represents a feature matrix, there are XYT=XXTW xyt=xxtw w= (xxt) −1xyt w= (XXT) −1xyt

The formula above is called normal equation. The closed solution of W can be obtained, but as long as the experiment is known, if X x has a slightly larger dimension, the process of inversion is very, very slow, and consumes very very much resources. Therefore, W-w is generally solved by gradient descent method.

The maximum likelihood method proves the rationality of least squares in some degree, but in fact in history the least squares appear earlier than the former, so the rationality of least squares can be considered from other angles. For example, the geometric meaning of the least squares, this article is very good, read the benefit.

From:http://bucktoothsir.github.io/blog/2015/12/04/leastsquare/