Robot Learning Cornerstone (Machine learning foundations) Learn Cornerstone Job three q18-20 C + + implementation

Hello everyone, I am mac Jiang, today and everyone to share the coursera-ntu-machine learning Cornerstone (Machines learning foundations)-Job three q18-20 C + + implementation. Although there are many great gods in many blogs have given the implementation of Phython, but given the C + + implementation of the article is significantly less, here for everyone to provide a C + + implementation of the idea! Although my code can get the correct answer, but there may be some ideas or details are wrong, if you bo friends found, please timely message correction, thank you! Again, Bo owners to provide the implementation of the code is not to let you pass the test, but for students who have difficulties in learning to provide a solution, I hope that my article on your study has some help!

The source of this article: http://blog.csdn.net/a1015553840/article/details/51085835

For additional questions, see the summary: http://blog.csdn.net/a1015553840/article/details/51085129

This part is mainly about using gradient descent to achieve logistic regression.

(1) Logistic regression

Logistic regression hyphothesis:

Logistic regression single point error:

Total cost of logistic regression:

(2) Gradient descent method

Batch gradient descent for logistic regression:

Stostic gradient descent for logistic regression: n-samples are not required for averaging, directly taking one of the gradients

(3) The implementation steps of logistic regression

1. Question 18th

(1) Test instructions: Download training samples and test samples from two websites for logistic regression, respectively. Take iteration step ita = 0.001, iteration count t=2000, Seek eout

(2) Implementation: Note that the best way to achieve total W initialization is all 0, because the X of the optimal solution is near 0. Because t=2000 only iterates 2000 times too little, if the initialization of the other number is 1, then 2000 iterations is not found the optimal solution, at least tens of thousands of times to iterate! The default initialization w value is not given in the topic, but I try to initialize it all to 0. This also shows that the logistic regression iteration is thousands of times large, and it takes tens of thousands of times to get a better solution, or a gradient descent method with advanced optimization.

#include "stdafx.h" #include <iostream> #include <fstream> #include <vector>using namespace std;# Define demension 20//Data dimension struct record{//data format double x[demension+1];int y;}; struct weight{//parameter format double w[demension+1];}; int sign (double x) {//signif (x > 0) return 1;else return-1;} void GetData (FStream &datafile,vector<Record> &data) {//Read data while (!datafile.eof ()) {Record temp;temp.x [0] = 1;for (int i = 1; I <= demension; i++) datafile>>temp.x[i];d atafile>>temp.y;data.push_back (temp);} Datafile.close ();} Double sigmoid (double x) {//sigmoid function, logical function, S-shape function return 1.0/(1.0 + exp (-X));} Double Vectormul (double *a,double *b,int demension) {//Two vectors multiplied return inner product Double temp = 0.0;for (int i = 0; I <demension; i++) tem p + = a[i] * B[i];return temp;} void Calcubatchgradient (vector<record> &data,weight weight,int n,double *grad) {//batch gradient descent method for (int i = 0; i < N i++) {Double temp = sigmoid ( -1 * VECTORMUL (weight.w,data[i].x,demension+1) * (double) data[i].y); for(int j = 0; J <= Demension; j + +) Grad[j] + = -1.0 * Temp * data[i].x[j] * DATA[I].Y; }for (int i = 0; I <= demension; i++) grad[i] = grad[i]/N;} void Calcustochasticgradient (Record data,weight weight,double *grad) {//Random gradient descent method Double temp = sigmoid ( -1 * VECTORMUL (Weig HT.W,DATA.X,DEMENSION+1) * (double) data.y); for (int j = 0; J <= Demension; j + +) Grad[j] + = -1.0 * Temp * data.x[j] * data . Y;} void Updatew (Weight &weight,double ita,double *grad) {//Take advantage of the resulting gradient update parameter weightfor (int i = 0; I <= demension; i++) {weigh T.w[i] = weight.w[i]-(ITA * grad[i]);}} Double Calculgerror (vector<record> &data,weight weight,int N) {//Calculate error calculation method for logistic regression double error = 0.0;for (int i = 0; i < N; i++) {error + = log (1 + exp (-DATA[I].Y * VECTORMUL (weight.w,data[i].x,demension+1)));} return double (error/n);} void Logisticregression (vector<record> &data,weight &weight,int n,double ita,int iteration) {//Logistic regression for ( int i = 0; I < iteration; i++) {//Use batch gradient descent method to calculate logistic regression double Grad[demEnsion+1] = {0.0};calcubatchgradient (Data,weight,n,grad); Updatew (Weight,ita,grad);cout<< "iter =" <<i << ", the logistic regression error of the training sample ein =" <<calculgerror (data,weight,n) <<endl;}   /*int i = 0; Using stochastic gradient descent method to calculate logistic regression while (I < iteration) {Double grad[demension+1] = {0.0};calcustochasticgradient (data[i%n] , Weight,grad); Updatew (Weight,ita,grad);cout<< "iter =" <<i<< ", training sample Logistic regression error ein =" << Calculgerror (Data,weight,n) <<endl;i++;} */}double calcuerror (vector<record> &data,weight weight,int N) {//Use logistic regression to do two-tuple classification, calculate 0/1 Error double = 0.0;for ( int i = 0; i < N; i++) {if (sign (Vectormul (data[i].x,weight.w,demension+1)) = Data[i].y) error++;} return double (error/n);}  void Main () {vector<record> trainingdata;      Training sample vector<record> TestData; Test sample FStream File1 ("TrainingData.txt");//Read the Training sample data fstream file2 ("TestData.txt");//Read Test sample Data if (File1.is_open () && File2.is_open ()) {getData (file1,trainingdata); GetData (file2,testdata);} Else{cout<< "Can not open file!" <<endl;exit (1);}  int train_n = Trainingdata.size ();//Training sample number int test_n = Testdata.size ();//test sample number Double ITA = 0.001;//step Itaint interation = 2000;//Iteration count weight weight;//Logistic regression parameter for (int i = 0; I <= demension; i++)//parameter is initialized to 0, note that if it is all 1 iterations 2000 times is not the result, because the optimal solution is near 0 To get results iteration must be around tens of thousands of times weight.w[i] = 1;logisticregression (trainingdata,weight,train_n,ita,interation);cout<    < "0/1 Error in training sample ein =" <<calcuerror (trainingdata,weight,train_n) <<endl; cout<< "test Sample 0/1 Error eout =" <<calcuerror (testdata,weight,test_n) <<endl;}

(3) Answer: 0.475

2. Question 19th

(1) Test instructions: Change the step ita=0.001 of the 18th question to 0.01, ask Eout

(2) Analysis: This is more simple, as long as the main function of the ITA changed to 0.01 can be

(3) Answer: After the iteration ein = 0.195, eout = 0.22; If the iteration 20,000 times, ein=0.172,eout=0.182 at this time basically to achieve the local optimal!

The answer is 0.220.

3. Question 20th

(1) Test instructions: Ita takes 0.001, iterates 2000 times, uses the stochastic gradient descent method (Stostic Gradieng descent), and the eout after iteration 2000 times

(2) Analysis: I have written a random gradient descent calculation method in the procedure given in the 18 questions! The 18th question uses the batch gradient descent, as long as in the logisticregression this function, the above comments out, the following comments in the code out of the random gradient descent method of the logical regression implementation.

(3) Answer: A. Iterate 2000 times, ein=0.466,eout = 0.475, so the answer is 0.475

B. If the iteration 20,000 times, ein=0.186,eout=0.196, so say 2000 times too little! It takes tens of thousands of times to reach the optimal solution.

For this problem, the answer is 0.475 because of t=2000.

The source of this article: http://blog.csdn.net/a1015553840/article/details/51085835

For additional questions, see the summary: http://blog.csdn.net/a1015553840/article/details/51085129

Robot Learning Cornerstone (Machine learning foundations) Learn Cornerstone Job three q18-20 C + + implementation