The Sklearn realization of 3-logical regression (logistic regression) in machine learning course

0. Overview

The linear regression can not only be used to deal with the regression problem, but also can be converted to the classification by comparison with the threshold value , but the output range of the assumed function is not limited. Such a large output is classified as 1, and a smaller number is divided into 1, which is odd. The output range of the hypothetical function of logistic regression is 0~1.

When the data set contains error points, the corresponding error of using linear regression will be very large.

logical regression is actually a classification algorithm, but it is called logistic regression because of historical reasons.

The assumption function of logistic regression is based on the assumption function of linear regression and the compound function formed by the S-shape function .

Although the cost function of logistic regression and the cost function of linear regression are the same in form, the assumption function is different, so it is actually different.

logical regression is a very powerful, and possibly even most widely used, classification algorithm in the world. 1. Classification issues

logical regression is a kind of classification algorithm

In the classification problem, the variable y to predict is a discrete value

Logical regression is one of the most widely used learning algorithms 2. Hypotheses indicate

assumption function Requirements : Due to logical regression this classifier requires an output value between 0 and 1, so it is necessary to think of an assumption that satisfies the predicted value between 0 and 1.

the hypothetical model of logistic regression :

where x represents the eigenvector, and G represents the logical function (logistic functions) is a commonly used logical function of the S-form (sigmoid function).

S-shaped functions (sigmoid function):

The image of the function is:

Together, we get the assumption that the logical regression model:

The function of hθ (x) is to compute the probability of the output variable =1 (estimated probablity) for a given input variable based on the selected parameter

Assuming that the value of the function is 0.7, then the probability y of 70% is a positive class. 3. Decision-making boundaries

decision Boundaries (decision boundary) is determined by a hypothetical function and can theoretically be any curve. 4. Cost function

For the linear regression model, the cost function is the sum of the squares of all the model errors. But in the logical regression , assuming the function is brought into the error square and the model, the assumption function will be a non convex function (Non-convex functions), so there are many local minima, which will affect the gradient descent algorithm to find the global minimum value.

the cost function of logistic regression :

, in which

The cost of the build (hθ (x), y) is simplified as follows:

The cost function of logistic regression is a function of the input variable theta 0, theta 1 、...、 theta N.
Bring in the cost function to get:

After we get such a cost function, we can use the gradient descent algorithm to obtain the parameter which can make the cost function minimum.

Gradient Descent :

After the derivation is obtained:
5. Simplified cost function and gradient descent

The gradient descent formulae for logistic regression and linear regression appear to be the same:

But its hypothetical function is different.
Linear regression assumption function:

Logical regression assumption function:
6. Advanced Optimization

In addition to the gradient descent method, there are conjugate gradient method, BFGS (variable scale method) and L-BFGS (limited variable scale method), the advantage of these three algorithms is not to manually select the learning rate, faster than the gradient descent method, the disadvantage is more complex. 7. Multi-Category classification: One-to-many

We will talk about how to use logical regression (logistic regression) to solve multiple classification problems, specifically, a classification algorithm called "One-to-many" (One-vs-all) can be used.
Suppose that a training set has three categories that can be converted into three two-yuan classification problems.
Finally, when we need to make predictions, we run all the classifiers and then select the highest possible output variable for each input variable.
The sum of probabilities of various probabilities is 1. 8. Code

(1) Original model

"" "Function: Logical regression Description: Author: Tang Tianze Blog: http://blog.csdn.net/u010837794/article/details/Date: 2017-08-14" "," "Import the package required for the project" "" Imports Nump Y as NP import matplotlib.pyplot as PLT # using Cross-validation method, the dataset is divided into training set test set from sklearn.model_selection import Train_test_split F  Rom sklearn import datasets from Sklearn.linear_model import logisticregression # loading Iris DataSet Def load_data (): Diabetes  = Datasets.load_iris () # splits the dataset into training sets and test sets X_train, X_test, y_train, y_test = Train_test_split (Diabetes.data, Diabetes.target, test_size=0.30, random_state=0) return X_train, X_test, Y_train, Y_test # use logisticregression to investigate linear back Homing predictive Capability def test_logisticregression (X_train, X_test, Y_train, y_test): # Select Model CLS = Logisticregression () # Put the data Give model training Cls.fit (X_train, Y_train) print ("coefficients:%s, intercept%s"% (Cls.coef_,cls.intercept_)) print ("Res Idual sum of squares:%.2f "% Np.mean (cls.predict (x_test)-y_test) * * 2)) print (' Score:%.2f '% cls.score x_test EST)) if __name__== ' __mAin__ ': X_train,x_test,y_train,y_test=load_data () # Generates a dataset test_logisticregression for regression problems (x_train,x_test,y_train,y_ Test) # Call Test_linearregression

coefficients:[[0.40051422 1.30952762-2.09555215-0.9602869]
[0.3779536-1.39504236 0.41399108-1.09189364]
[ -1.66918252-1.18193972 2.39506569 2.00963954]], intercept [0.24918551 0.81149187-0.97217565]
Residual sum of squares:0.11
score:0.89

(2) using multiple classification parameters on the basis of (1)

"" "Function: Logical regression Description: Author: Tang Tianze Blog: http://blog.csdn.net/u010837794/article/details/Date: 2017-08-14" "," "Import the package required for the project" "" Imports Nump Y as NP import matplotlib.pyplot as PLT # using Cross-validation method, the dataset is divided into training set test set from sklearn.model_selection import Train_test_split F  Rom sklearn import datasets from Sklearn.linear_model import logisticregression # loading Iris DataSet Def load_data (): Diabetes  = Datasets.load_iris () # splits the dataset into training sets and test sets X_train, X_test, y_train, y_test = Train_test_split (Diabetes.data, Diabetes.target, test_size=0.30, random_state=0) return X_train, X_test, Y_train, Y_test # use logisticregression to investigate linear back Homing predictive Capability def test_logisticregression_multiomaial (X_train, X_test, Y_train, y_test): # Select Model CLS = Logisticregression ( Multi_class= ' multinomial ', solver= ' Lbfgs ') # submit data to Model training Cls.fit (X_train, Y_train) print ("coefficients:%s, inte  Rcept%s "% (Cls.coef_,cls.intercept_)) print (" Residual sum of squares:%.2f "% Np.mean (cls.predict (x_test)-y_test) * * 2) Print (' Score:%.2f '% Cls.score (x_test, y_test)) if __name__== ' __main__ ': X_train,x_test,y_train,y_test=load_data () # Generating datasets for regression problems Test_logisticregression_multiomaial (x_train,x_test,y_train,y_test) # calls Test_linearregression

coefficients:[[-0.39772352 0.83347392-2.28853669-0.98142875]
[0.54455173-0.29022825-0.23370111-0.65566222]
[ -0.14682821-0.54324567 2.5222378 1.63709097]], intercept [8.99974988 1.54361012-10.54336001]
Residual sum of squares:0.02
score:0.98

(3) Consideration of regularization coefficient

"" "Function: Logical regression Description: Author: Tang Tianze Blog: http://blog.csdn.net/u010837794/article/details/Date: 2017-08-14" "," "Import the package required for the project" "" Imports Nump Y as NP import matplotlib.pyplot as PLT # using Cross-validation method, the dataset is divided into training set test set from sklearn.model_selection import Train_test_split F  Rom sklearn import datasets from Sklearn.linear_model import logisticregression # loading Iris DataSet Def load_data (): Diabetes  = Datasets.load_iris () # splits the dataset into training sets and test sets X_train, X_test, y_train, y_test = Train_test_split (Diabetes.data, Diabetes.target, test_size=0.30, random_state=0) return X_train, X_test, Y_train, Y_test # use logisticregression to investigate linear back
    Homing predictive Capability def Test_logisticregression_c (X_train, X_test, Y_train, y_test): Cs=np.logspace ( -2,4,num=100) scores=[]

        For C in Cs: # Select Model CLS = Logisticregression (c=c) # submit data to Model training Cls.fit (X_train, Y_train) Scores.append (Cls.score (X_test, Y_test)) # # Drawing Fig=plt.figure () Ax=fig.add_subplot (1,1,1) ax.pl OT (cs,scores) ax.set_xlabel (r "C") Ax.set_ylabel (r "Score") Ax.set_xscale (' Log ') Ax.set_title ("Logisticregression") plt.show () If __name__== ' __main__ ': X_train,x_test,y_train,y_test=load_data () # Generates a dataset for regression problems Test_logisticregression_c (X_trai N,x_test,y_train,y_test) # Call Test_linearregression

9. Summary 10. Reference materials

[1] ng CS229
[2] Hangyuan Li "Statistical Learning method"
[3] The Chinese School of "Python War Machine Learning"