Stanford Machine Learning---second speaking. multivariable linear regression Linear Regression with multiple variable

Original: http://blog.csdn.net/abcjennifer/article/details/7700772

This column (machine learning) includes linear regression with single parameters, linear regression with multiple parameters, Octave Tutorial, Logistic Regression, regularization, neural network, design of the computer learning system, SVM (Support vector machines), clustering, dimensionality reduction, anomaly detection, large-scale machine learning and other chapters. All of the content is from the Standford public class machine learning in the teacher Andrew's explanation. (Https://class.coursera.org/ml/class/index)

Second-------multivariable linear regression Linear Regression with multiple variable

(i), multiple Features:

Multivariate hypothesis: The output is determined by multidimensional input, that is, the input is a multidimensional feature. As shown: Price is output and the front four dimensions are input:

Suppose h (x) =θ0+θ1x1+ ... The so-called multi-parameter linear regression is each input x has (n+1) dimension [X0......XN]

(ii), Gradient descent for multiple Variables:

On the left is a gradient-decreasing single-variable learning method for the parameter, and the new algorithm on the right is the Multivariable learning method.

(iii), Gradient descent for multiple variables-feature Scaling

It is important to normalized feature, so feature scaling is used, all feature will be normalized to the [ -1,1] interval:

Normalization method: Xi= (xi-μi)/σi

(iv), Gradient descent for multiple variables-learning rate

Another key point in the gradient descent algorithm is the design of the machine learning rate: The design criteria guarantee that the cost function will be reduced after each iteration.

This is the case where the cost function has dropped smoothly:

This is the case where the cost function does not fall smoothly:

The reason, as shown in the picture on the right, is that because of the high learning rate, as the number of iterations increases, J (θ) will skip large, resulting in an inability to converge.

WORKAROUND: Reduce the learning rate

Summary: How to choose the learning rate:

Test α=0.001, convergence too slow (cost function drops too slow), test 0.01, past? That's 0.003 ...

(v), Features and polynomial Regression

Suppose our input is a size of a house, the output of which is the price of it, which is polynomial-fitted:

There are two options, two equations or three equations. Considering that the two-time equation is always at the highest point, with size↑,price↓, it is unreasonable to use three-time equations to fit.

Normalization here is one of the keys.

Or there is another fitting equation, the pink curve fits the following:

(vi), Normal equation

A method parallel to the gradient descent is normal equation, which uses a non-iterative approach to linear algebra, see:

We want to find the θ that minimizes the cost function by finding the parameter θ that makes the derivative take 0 o'clock:
This parameter can be obtained by the red box formula in the diagram:

Specifically: X is a matrix of MX (n+1) and y is the matrix of MX1

Why do x add a column 1? Since the x (i) 0=1 is often set;

The difference between gradient descent and normal equation is compared below:

(vii), Normal equation noninvertibility

We know that for a sample of M, each with n feature a training set, X is an MX (n+1) matrix, XTX is (n+1) x (n+1) square, then for the calculation of the parameter θ there is a problem, if | Xtx|=0, that is XTX do not seek inverse matrix? At this point can be redundant feature deletion (m<=n situation, feature too much):

Stanford Machine Learning---second speaking. multivariable linear regression Linear Regression with multiple variable