New understanding of AUC KS

Turn https://www.zybuluo.com/frank-shaw/note/152851 New Understanding: I think the AUC, and KS similar. The AUC is a threshold based on the predicted probability (from large to small) and can be divided into no more than n threshold values for the sample count. You can get n recall and precision the ROC curve by connecting these points to a line. The AUC is the area under the ROC. The point closest to the upper left corner is the best threshold. The thresholds for 1 and 0 were obtained (0,0) and (at) points respectively. The sample point is just a point on the curve, theoretically infinitely large, impossible to depict. Transverse: false Yang rate, longitudinal: recall. n points for the corresponding threshold segmentation of the recall and PRECISION.KS can be considered to be based on the predicted probability (from large to small sort) as a threshold, the horizontal axis of 0 to 1 threshold, vertical n recall and precision points, plotted two curves respectively, the absolute difference between them the maximum value is the KS value, the horizontal axis is the most A good threshold value.  And according to the financial definition: KS is 1, according to the predicted value of the partition to find the original Y value of the ratio (Division of good/bad/better than the overall good or bad).   2. The maximum value of the absolute difference of the ratio-bad proportion is the KS value. For example: 78217498 I don't think it's right. If the maximum value may be good or bad for a partition, such a value is meaningless. Only the absolute difference of the recall-false positive rate is the most practical significance. Objective

Most of this article comes from the following two blogs:
http://blog.csdn.net/dinosoft/article/details/43114935
http://my.oschina.net/liangtee/blog/340317

Introduction

Suppose you have the following two classifiers, which one is good? (There are 90 samples of Class A and 10 samples in class B.) ）

,	Class A sample	Class B Samples	Classification Accuracy
Classifier C1	A*90 (100%)	A*10 (0%)	80D
Classifier C2	A*70 + b*20 (78%)	A*5 + b*5 (50%)	75%

Classifier C1 all the test samples are divided into a class, the classifier C2 the Class A 90 samples to 70, Class B 10 samples 5.

The C1 classification accuracy of 90%,C2 is 75%, but intuitively, we feel that C2 is more useful. But according to the correct rate to measure, then certainly C1 effect is better. Then it's inconsistent with what we think. In other words, there are times when it is inappropriate to rely solely on the correct rate.

We also need an evaluation index, can objectively reflect the positive sample, negative sample comprehensive prediction ability, but also to consider eliminating the impact of sample tilt (in fact, is the idea of normalization, the actual is very important, such as PV is always far greater than click), this is the AUC indicator can solve the problem.

ROC

In order to understand the AUC, we need to understand ROC first.
Take a look at the results of a common two classification problem, with 4 combinations of predicted and actual values, as shown in the following table:

We define a variable:

As you can see, TPR represents the proportion of the quantity that is predicted to be correct and actually classified as positive and all positive samples. -What is the correct forecast ratio in the actual positive sample?

FPR represents the proportion of the quantity that is predicted to be wrong and is actually classified as negative to all negative samples. -What is the percentage of false predictions in the actual negative sample?

You can substitute the above two classifiers to get the following table (classifier C1):

,	forecast a	Forecast B	Total
Actual A	90	0	90
Actual B	10	0	10

TPR = FPR = 1.0.

Classifier C2:

,	forecast a	Forecast B	Total
Actual A	70	20	90
Actual B	5	5	10

TPR = 0.78, FPR = 0.5

Then, to TPR as the ordinate, FPR for horizontal drawing, you can get:

The medium blue represents the C1 classifier, and the green represents the C2 classifier. Can know, this time the green point is closer to the upper left corner, can be regarded as the classification effect is better. So the evaluation criteria to be close to the upper left corner is a good classifier (considering the combination of positive and negative samples of the ability).

A series of such points form a curve, which is the ROC curve. The area under the ROC curve is the AUC (areas under the curve of ROC). This is the origin of the AUC indicator.

How to draw Roc curves

For a particular classifier and test data set, it is obvious that we can only get a classification result, that is, a set of FPR and TPR results, and to get a curve, we actually need a series of FPR and TPR values to get such a curve, and how to get it?

One of the important functions of the classifier is "probability output", that is, the classifier thinks that a sample has a large probability of being a positive sample (or negative sample), to dynamically adjust whether a sample is a positive or negative sample (remember when Ali race when there is an indication of the probability of a positive sample of the column? ）

If we have obtained the probability output of all samples (the probability of a positive sample), the question now is how to change this threshold (probability output)? We sort from large to small according to the probability values of each test sample that belong to a positive sample. is an example of a total of 20 test samples, the "Class" column represents the true label for each test sample (P for positive sample, N for negative sample), and "score" indicates the probability that each test sample belongs to a positive sample.

Next, we take the "score" value as the threshold from high to low, and when the probability of a test sample being a positive sample is greater than or equal to this threshold, we consider it a positive sample, otherwise a negative sample. For example, for the 4th sample in the diagram, whose "score" value is 0.6, the sample 1,2,3,4 are considered positive samples because their "score" values are greater than or equal to 0.6, while the other samples are considered negative samples. Each time a different threshold is selected, we can get a set of FPR and TPR, a point on the ROC curve. In this way, we get a total of 20 sets of FPR and TPR values, which are drawn in the ROC curve results such as:

When we set the threshold to 1 and 0 o'clock, we can get (0,0) and (two) points on the ROC curve respectively. By connecting these (FPR,TPR) pairs, the ROC curve is obtained. The more the threshold value, the more smooth the ROC curve.

--When Ali was playing, he thought Roc was useless!!!! really is Youyanwuzhu Ah!!! Still have doubts is: how according to Roc to judge the result of good change? See which classifier is closer to the upper left corner. At the same time, according to the ROC to determine the delineation of positive samples of the probability boundary selection where more appropriate!!! It turns out this is!!!!!!!!!

Why Use Roc

Why use ROC and AUC, since so many criteria have been evaluated? Because the ROC curve has a good feature: when the distribution of positive and negative samples in the test set changes, the ROC curve can remain unchanged. There is often a class imbalance in the actual data set (class imbalance), where negative samples are much more than positive samples (or vice versa), and the distribution of positive and negative samples in the test data can vary over time. Is the contrast between the ROC curve and the Precision-recall curve:

In, (a) and (c) are the ROC curves, (b) and (d) are precision-recall curves. (a) and (b) show the results of classifying them in the original test set (distribution balance of positive and negative samples), (c) and (d) The result of the classifier by increasing the number of negative samples in the test set to the original 10 times times. It can be seen clearly that the ROC curve remains essentially the same, while the Precision-recall curve changes considerably.

New understanding of AUC KS