Reprint Deep Learning: Three (multivariance Linear regression practice)

Objective:

This article is mainly to practice multivariable linear regression problem (in fact, this article also on 3 variables), reference page: http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course= Deeplearning&doc=exercises/ex3/ex3.html. In fact, in the previous blog Deep learning: Two (linear regression practice) the solution of one-element linear regression problem is briefly introduced, but when it is solved by the gradient descent method, The learning rate given is fixed 0.7. The study rate in this experiment needs to be chosen by itself, so we should choose from small to large (for example, from 0.001 to 10) to determine which learning rate to use by observing the function curve between the loss value and the number of iterations. When learning rate alpha is available, the problem solving method is no different from the above.

The problem to be solved is to give 47 training samples, the Y value of the training sample is the house price, The X attribute has 2, one is the size of the house, the other is the number of the house bedroom. These training data are needed to learn the functions of the system, predicting a house size of 1650, and the price of a bedroom with 3 houses.

Experimental basis:

Dot (b): Represents the inner product of vector A and vector B.

And the theory of linear regression can be known as the loss function of the system as shown below:

Its vector representation is as follows:

When using the gradient descent method to solve the parameters, the formula for updating the parameters is as follows:

Of course it also has its own vector form (which can be embodied in the program).

Experimental results:

The results of the test learning rate are as follows:

Therefore, the choice of learning rate of 1 o'clock, can reach a rapid convergence rate, so the final program used in the learning rate of 1.

The results of the prediction using the gradient descent method and the formula method are as follows:

It can be seen that the results of both are consistent.

Experiment main program and code:

%%method One: Gradient Descent method x= Load (' Ex3x.dat '); y= Load (' Ex3y.dat '); x= [Ones (Size (x,1), 1) X];meanx= mean (x);%mean value Sigmax= STD (x);%Find standard deviation x (:,2) = (X (:, 2)-meanx (2))./sigmax (2); X (:,3) = (X (:, 3)-meanx (3))./sigmax (3); Figureitera_num= 100; %number of iterations attempted sample_num= Size (x,1); %number of training samples Alpha= [0.01, 0.03, 0.1, 0.3, 1, 1.3];%because almost every 3 times times the learning rate to test, so directly enumerated Plotstyle= {' B ', ' R ', ' G ', ' k ', ' b--', ' r--'};theta_grad_descent= Zeros (Size (x (1,:))); forAlpha_i = 1:length (alpha)%try to see which learning rate is best theta= Zeros (Size (x,2), 1); %the initial value of the theta is assigned a value of 0 Jtheta= Zeros (Itera_num, 1);  fori = 1:itera_num%calculates the parameter Jtheta (i) after the number of iterations of a learning rate alpha Itera_num= (1/(2*sample_num)). * (x*theta-y) ' * (x*theta-y);%jtheta is a line vector grad = (1/sample_num). *x ' * (x*theta-y); theta = Theta-alpha (alpha_i). *grad; End Plot (0:49, Jtheta (1:50),Char(Plotstyle (alpha_i)), ' LineWidth ', 2)%It is important to use the CHAR function to convert hold onif(1 = = Alpha (alpha_i))%The result of the experiment is that the alpha 1 o'clock is the best, then the theta value after the iteration is the desired value theta_grad_descent=Theta Endendlegend (' 0.01 ', ' 0.03 ', ' 0.1 ', ' 0.3 ', ' 1 ', ' 1.3 '); Xlabel (' Number of iterations ') Ylabel (' Cost function ')%here is the prediction formula Price_grad_descend= Theta_grad_descent ' *[1 (1650-meanx (2))/sigmax (2) (3-meanx (3)/sigmax (3))] ' %%method Two: normal EQUATIONSX= Load (' Ex3x.dat '); y= Load (' Ex3y.dat '); x= [Ones (Size (x,1), 1) x];theta_norequ= INV ((x ' *x)) *x ' *yprice_norequ= Theta_norequ ' *[1 1650 3] '

Resources:

http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course=DeepLearning&doc=exercises/ex3/ Ex3.html

Deep Learning : Two (linear regression exercises)

Tornadomeet Source: Http://www.cnblogs.com/tornadomeet Welcome to reprint or share, but be sure to declare the source of the article.

Reprint Deep Learning: Three (multivariance Linear regression practice)