Softmax regression--Identifying Minst Database

Softmax regression is a generalized version of the logistic regression.

But the logistic regression is a 2 classification problem, and Softmax is a multi-classification problem, that's all.

The Softmax regression was degenerate into a logistic regression in the case of 2 classification.

The cost function of logistic regression

Considering the weight decay, modifying the cost function of the Softmax regression

The weight decay here is necessary because the original cost function is not a strictly convex function and has an infinite number of minima. Now this weight decay is added, and the function becomes a strictly convex function. The L-BFGS algorithm can guarantee the global optimal solution of convergence.

The derivative of the new function is as follows

Next is programming, to pay attention to a few small problems

1. To put the following files into the path also, in order to invoke:

Computenumericalgradientloadmnistimagesloadmnistlabelstrain-images.idx3-ubytetrain-labels.idx1-ubyteminfunc </span>

2. The name of the data file in the main program needs to be modified

Switch

Train-images.idx3-ubyte/train-labels.idx1-ubyte</span>

3. Note the dimensions of several important matrices

Data:784*60000theta:10*784m:10*60000groundtruth:10*60000</span>

4. Use Bsxfun function, less memory, improve speed.

5. Note how to prevent data overflow processing.

6.gred records the number of occurrences of the maximum probability, not the number.

7. Commonplace, vectorization Programming .

The core code is given below:

Softmaxcost

m=theta*data;% gets the weighted data matrix M=bsxfun (@minus, M, Max (M, [], 1));% prevents data overflow hypothesis=bsxfun (@rdivide, exp (m), SUM (exp (m))); Get the probability matrix cost=-(Groundtruth (:) ' *log (hypothesis (:)))/numcases+lambda/2*sumsqr (theta);% cost function thetagrad=-( groundtruth-hypothesis) *data '/numcases+lambda*theta;% gradient function </span>

Softmaxpredict

[~,pred]= Max (theta*data);% record the maximum probability of the sequence number instead of the maximum value </span>

accuracy:93.272%, can see Softmax easy to achieve, the effect is also very good.

Welcome to participate in the discussion and follow This blog Weibo personal home Subsequent content continues to update Oh ~

reproduced please respect the author's labor, complete reservations The above text as well article links , thank you for your support!