27-How to measure the performance of the classification algorithm (scoring metrics for classification)

The last two days have embraced the city of Beijing, feeling the atmosphere is rough, the whole person is more energetic. The Forbidden City is big, the Summer Palace is bigger, I bought the joint ticket, the result is impossible to finish. Beijing people are also enthusiastic, sitting on the boat, the big uncle kept greeting: This side can be taken 17 hole bridge, Vision good area; There you can shoot the Jade Belt bridge; sit and I'll take a few shots for you; you hold the tower in your hand. Then walk on the road, ask a girl where is the East Gate, a walking Fengfenghuohuo aunt confessed: Right in front of the turn ... The enthusiasm of the people of Beijing is enough to withstand the long winter.

Come back also to study hard, today to learn the performance of the measurement classification algorithm is good or bad several important indicators:)

First affirm the following abbreviation: p:positive; n:negative; Tp:true Positive; Fp:false Positive; Tpr:true Positive rate; TN, FN, FPR: You know.

1. Accuracy rate (accuracy)

For a given test data set, the classifier correctly classifies the ratio of the number of samples to the total number of samples.

A = (tp + tn)/(P+N) = (tp + TN)/(TP + FN + FP + TN); It reflects the ability of the classifier to judge the whole sample-positive judgment is positive and negative judgment is negative.

2. Confusion matrices (confusion matrix)

3. Accuracy rate (Precision), recall rate (Recall) and F1-measure

Accuracy rate (Precision):

PRE = tp/(TP+FP); Reflects the proportion of the true positive sample in the positive case determined by the classifier.

Recall (Recall), also known as True Positive rate:

REC = tp/(TP+FN); It reflects the proportion of the positive cases that are correctly judged to account for the total.

F1-measure, which can be understood as the geometric average of the accuracy and recall rate:

F1 = 2 * PRE * rec/(PRE+REC);

Another useful function is Classification_report, which provides accurate rates, recall rates, and fscore calculations for all classification categories (all values of y).

The above several metrics are especially suitable for two kinds of situations which are often encountered in practice:

1. Category of imbalance: a category that appears to be much more frequent than other categories;

2. The cost of non-equivalence: that a classification error requires a higher cost than other classification errors (such as the diagnosis of cancer patients as non-cancer, the consequences are disastrous).

4. ROC & AUC

First understand the two formulas:

TPR = TP/(TP + FN)--can be understood as sensitivity to positive cases

FPR = FP/(fp + TN)--can be understood as blind sensitivity to positive cases

Wiki definition:

The ROC curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various Thresho LD settings.

The ROC curve draws the correspondence between FPR (x) and TPR (y) when the threshold changes. When the test sample is a positive sample of the probability value is greater than a certain threshold (commonly used 0.5), that is, it is a positive sample, where the threshold is threshold.

For example: If the threshold is very low (such as 0.1), then it is easy to test the sample is a positive sample of the probability value >0.1, so FPR will be very high (too sensitive to the positive case), and TPR is very high, assuming threshold=0.1, fpr=tpr= 1, you can draw the first point in the coordinate system (threshold); In the same vein, with the increase of the FPR and TPR will gradually decrease, and finally even 0. The shape of the curve is usually as follows:

If the classifier works well, the curve should be close to the upper-left corner (TPR large, FPR small).

The AUC (area under Curve) is defined as the size of the ROC curve, which obviously does not have a value greater than 1. Since ROC curves are generally located above the line of y=x (corresponding to random guesses), the AUC has a value range between 0.5 and 1. The AUC value is used as the evaluation criterion because many times the ROC curve does not clearly explain which classifier works better, and as a numeric value, it is better to have a larger classifier for the AUC.

Next week to learn the decision tree, dry more, please look forward to:)

27-How to measure the performance of the classification algorithm (scoring metrics for classification)