-logistic regression algorithm for machine learning algorithm

Logistic regression algorithm debugging First, the principle of the algorithm

Logistic regression algorithm is an optimization algorithm, which is mainly used for classification problems with only two kinds of labels. The principle is to use a straight line to fit some data points and divide the data set. Broadly speaking, this is also a multivariate linear regression method, the difference is that the algorithm needs to find out the maximum possible to divide the two categories and not according to the linear relationship to predict the value of the dependent variable. The core part of the logistic regression algorithm is the sigmoid function:

WhereXI is the first feature of the DataSet . Define loss function loss function:

The smaller the loss function, the better the effect of curve fitting. The optimal solution of W is obtained by updating the coefficient wof x with the gradient upward method . The principle of gradient upward method is that the best way to obtain the extremum of a function is to follow the gradient direction of the function, and the gradient can be solved by partial derivative. the gradient grad of the loss function of Logistic regression algorithm is .

By constantly updating the weights of x iterations a certain number of times you can get the desired W.

Second, algorithm debugging 2.1 gradient descent method algorithm

first define the functions that import the data Loaddataset (), which makes the data import more convenient. For the convenience of operation, the code in the book changes slightly, the input file location into a direct input matrix. The following two lines in the gradient descent algorithm are derived from the solution of the loss function, and the process is not complicated.

Run Loaddataset to test the input algorithm, the result is as follows:

2.2 Draw a decision boundary

after the test of the gradient ascending algorithm is feasible, a set of regression coefficients is actually solved. In order to understand the logistic algorithm better, we need to draw the dividing line. Define the function Plotbestfit and enter the code in the Pycharm. Notable in this program is the Geta function, which is a function in the Maxtri module in the NumPy Library, which works just as opposed to the mat, which transforms the matrix into an array. If it is removed, it does not work, mainly because each line of the matrix can be only 1 in length and not as an array. After passing the test result 1.

Figure 1

2.3 Random Gradient ascent algorithm

due to the construction of gradient ascending algorithm every time the entire data set, when the amount of data is very large, the computational capacity is very large. The following is an improved algorithm for the ascending algorithm of the ladder, the random gradient ascending algorithm. The principle is to update the regression coefficients with only one sample at a time, and then incrementally update the new sample input. Defines the STOCGRADASCENT0 function, which is expected to be very similar to the gradient ascent algorithm. Enter the code to verify that the following error occurred between validation:

This problem occurs because of the "*" problem of the operation symbol, in the calculation between the list, can only be used to multiply the integer to the other list, this may be because a number in python when multiplying a matrix will first copy the number into a list of length and the same list, and then multiply. But the problem with this is that you can only work on integers when you generate a copy list. The modified method is to convert the list to a matrix, and then the operation. The places to be modified are:

after debugging can be good output results,2 shows:

Figure 2

The result is not as good as before, because some coefficients converge very slowly, and after 500 iterations they do not reach a stable value, so the next step needs to be improved.

2.4 Improved stochastic gradient rise method

To prevent the coefficients from bouncing back and forth during the iteration, they converge to a certain value. The method of random ascending gradient is improved, the α is adjusted on the one hand, and the fluctuation is reduced, on the other hand the random selection sample is used to update the regression coefficients, which will reduce the periodic fluctuation. Then debug the program, enter the code to run the problem is not found, can directly run the results,3:

Figure 3

It can be seen that the effect of classification is very good, and the data set is completely classified to prove the effectiveness of the algorithm.

2.5 Logistic Regression classification algorithm

The logistic optimization algorithm has been found out, the following need to use logistic regression classification test. First, the logistic classifier is constructed. Because of the watermelon dataset selected, the data in the textbook is not used, so the code is slightly different. But the method is the same. The main program code is as follows:

defClassifyvector (inx,weights): Prob=sigmoid (SUM (inx*weights)) ifprob>0.5:return1.0Else:return0.0defColictest (Testset,weights,labelset):#input test set, regression factor, labelErrorcount =0 N=shape (Testset) [0] forIinchrange (n):ifClassifyvector (Array (testset[i]), weights)! =Labelset[i]: Errorcount= Errorcount + 1errorrate= errorcount*1.0/Nreturnerrorratedefmultitest (testset,weights,labelset): Numtests=10;errorsum=0.0 forKinchRange (numtests): Errorsum=errorsum+colictest (Testset,weights,labelset)Print('average error rate is%s'% (errorsum/numtests))

Here we are using the watermelon data set, because the sample less predictable effect is not very good, after using three algorithms, the error rate of about 55%. If the data set as a training set and as the test set accuracy rate of about 75%, by modifying the number of iterations the final accuracy rate will converge to 84%.

Iii. Summary

first, through the debugging of the algorithm, the principle of the algorithm and implementation methods have a further understanding. Theoretically,the logistic function is a more ideal algorithm. In the algorithm, some improvements are made to the gradient rise method, which makes the convergence speed faster, and the feasibility of the algorithm is greatly improved. But through the example analysis, the classification effect is not as good as the expected effect. There is still room for improvement in the algorithm, which deserves further exploration.

-logistic regression algorithm for machine learning algorithm