Logistic Study Notes

1. Differences between logistic regression and general linear regression models:

(1) the relationship between result variables and dependent variables of linear regression or between response variables and independent variables is assumed to be linear, while the relationship between the two in logistic regression is non-linear;

(2) The premise assumption is different. In linear regression, it is usually assumed that for a value of the independent variable X, the observed values of the dependent variable Y are normally distributed, but in logistic regression, the dependent variable Y follows two or more distributions;

(3) Logistic regression does not contain any residual items.

2. Model Selection:

Model Selection is very useful in practical application. Generally, model-related data is divided into three parts: training data, verification data, and test data, as shown in:

 

Training set

Validation Set

Test Set

 

Both the training data and validation data are existing sample data, that is, the observed data. Test data is the data generated in future practical applications and is unknown in advance.

The question of model selection is how to verify whether a model is good. The final performance of a model depends on its performance in the test dataset. Therefore, when no test data is observed, we can only use the verification data set to replace it for testing. Generally, the cross verification method is used, for example, LOOCV, that is, leave one method for cross verification, and similar K-fold Cross verification. The main purpose of cross-validation is to prevent trained models from over-fitting. However, in today's world, because the data is massive, the cross-validation method is getting fewer and fewer, because if the training data set is very large, there is generally no overfitting phenomenon.

There are also some ways to evaluate the model quality directly without verification, such as AIC, Bic, MDL, SRM, etc.

3. Parameters of interest in statistical results:

In commercial practice, the above statistics are most concerned with the c statistic, followed by the likelihood chi-square statistic, followed by the HL statistic, and seldom concerned with AIC and rsquare, this is very different from multivariate linear regression. The root cause is that multivariate linear regression is a prediction model. The value of the target variable has actual numerical significance, while logistic is a classification model, the value of the target variable is a classification identifier, so the relative consistency between the observed value and the predicted value is more important than absolute consistency.

4. Differences and relationships between logistic regression and SVM:

Both methods are common classification algorithms. as a very important task in the field of data mining, classification aims to learn a classification function or classification model (or classifier ).

The difference between objective functions is that logistic regression uses logistical loss and SVM uses hinge loss. The two loss functions aim to increase the weights of data points that have a greater impact on classification and reduce the weights of data points that have a smaller relationship with classification. SVM only takes support vectors into account, that is, the most relevant few points of classification, to learn classifier. Logistic regression uses nonlinear ing to greatly reduce the weights of vertices that are far away from the classification plane and increase the weights of data points that are most relevant to the classification. Both have the same fundamental purpose. In addition, different regularization items, such as L1 and L2, can be added to both methods as needed. Therefore, in many experiments, the results of the two algorithms are very close.
However, logistic regression is relatively simpler, easier to understand and implement, especially for Large-Scale Linear classification. SVM is more complex to understand and optimize. However, the theoretical basis of SVM is more solid, and there is a set of theoretical basis for minimizing structural risks, although the average user will not pay much attention to it. Another important point is that after SVM is converted to a dual problem, classification only requires calculation of the distance from a few support vectors, which has obvious advantages in complex Kernel Function compute, this greatly simplifies the model and computing workload.

5. prior probability and Posterior Probability

Prior probability a prior probability describes a variable in the absence of a fact. A posterior probability is a conditional probability after a fact is taken into account.

6. Rationality of Logistic regression model selected in medical disease research:

(1) The model determines that you are not ill, but you have a low probability of illness. The advantage of Logistic is that logistic controls FPR well, so when the costs of the two types of errors are inconsistent, you must carefully analyze the selection.

7. Source and definition of ROC curve

Many modern classifiers do not simply give a 0 or 1 classification decision, but give a classification tendency, such as the classification probability output by Bayesian classifier. For these classifiers, when you take different thresholds, you can get different classification results and classifier evaluation indicators. We can also see that with the decrease of the threshold (more customers will be treated as positive instances), recall and 1-specificity also increase (that is, specificity decreases accordingly ). Then we need to make a dynamic evaluation. A reasonable way is to plot a series of recall and specificity generated based on different thresholds to Cartesian coordinates, you can see their corresponding relationships more clearly. Because the direction of recally and specificity is just the opposite, we plot sensitiousness and 1-specificity to the same graph, and their corresponding relationship gets the legendary ROC curve, the full name is faster er operating characteristic curve. The Chinese name is "receiver operation characteristic curve ".