Yolo Learning Recall (Recall), accuracy rate (Precision), average correct rate (average_precision (AP)), and (Intersection-over-union (IoU)) __recall

Summary

In the process of training YOLO V2, the system will show some values of evaluation training effect, such as Recall,iou and so on. In order to be afraid of forgetting later, now put their own understanding of these several measures of the way to record.
This article first assumes a test set, and then around the test set to introduce the methods of calculation of these measures. the wild Goose and the plane

Assuming there is a test set, the images in the test set are made up of two pictures of geese and airplanes, as shown in the following illustration:

Let's say your classification system ends up being able to take out pictures of all the planes in the test set, not the wild goose.

Now make the following definition:
True Positives: The picture of the airplane was correctly identified as an airplane.
True negatives: wild Goose Pictures are not recognized and the system rightly thinks they are geese.
false positives: The picture of the wild goose was wrongly identified as an airplane.
False Negatives: pictures of airplanes are not recognized, and the system mistakenly thinks they are geese.

Suppose your classification system uses the above assumptions to identify four results, as shown in the following illustration:

So in the four images identified:
True Positives: There are three planes that draw the green box.
False Positives: There is a, painted red box of wild Goose.

Not identified in the six photos:
True Negatives: There are four pictures of these four wild geese, and the system correctly does not recognize them as airplanes.
False Negatives: There are two, two planes are not recognized, the system mistakenly think they are geese. Precision and Recall

Precision is actually the ratio of True positives in the recognized picture:

where n stands for (True positives + False positives), which is how many photos the system has identified altogether.
In this example, the True positives is 3,false positives is 1, so the precision value is 3/(3+1) = 0.75.
This means that in the identified results, the picture of the aircraft accounted for 75%.

Recall is the ratio of the number of aircraft correctly identified to the number of all aircraft in the test set:

The denominator of the recall is (True positives + False negatives), and the sum of the two values can be understood as a total number of photos of the plane.
In this example, the True positives is 3,false negatives is 2, then the recall value is 3/(3+2) = 0.6.
This means that 60% of all aircraft images are correctly identified as aircraft. Adjust threshold value

You can also change the value of precision or Recall by adjusting the threshold to choose how many images the system recognizes.
Under a certain threshold (blue dotted line), the system identifies four images, as shown in the following figure:

The four images that the classification system considers to be greater than the threshold (above the blue dotted line) are more like airplanes.

We can choose to let the system recognize how many pictures can be made by changing the threshold (or as the blue dotted line up and down), and of course the threshold change will cause precision and recall values to change. For example, put the blue dotted line under the first picture, which means that the system only identifies the top plane picture, then the precision value is 100%, and the recall value is 20%. If you put the blue dotted line under the second picture, which means that the system only recognizes the top two pictures, the precision value is 100%, and the recall value is 40%.

The following figure is the change of precision and recall under different threshold conditions:
Precision-recall Curve

If you want to evaluate the performance of a classifier, a better way to do this is to observe the changes in precision and recall values when the threshold changes. If the performance of a classifier is better, then it should behave as follows: The identified image of the aircraft accounted for a larger proportion, and before identifying the wild goose, as much as possible to correctly identify the aircraft, that is, let recall value increase while maintaining the value of precision at a very high level. The poor performance classifier may lose a lot of precision value in order to improve the recall value. In general, the Precision-recall curve is used in the article to show the trade-off between precision and recall.

The above diagram is the Precision-recall curve of the classifier, which can achieve 40%recall without loss of precision. And when the recall reaches 100%, the Precision is lowered to 50%. approximated Average Precision

Compared with the graphs, at some times a specific value can show the performance of the classifier more visually. Typically, the Average precision is used as the metric, and its formula is:

In this integral, where p represents precision, R represents Recall,p is a function with R as the parameter, which is equal to taking the area under the curve.

In fact, this integral is extremely close to this value: each threshold value (precision value) is multiplied by (the change of the recall value), then the product value obtained under all thresholds is added. The formula is as follows:

In this formula, N represents the number of all the pictures in the test set, P (k) indicates the value of the precision when the K picture is recognized, and Delta R (k) indicates the change of the value of the number of images from K-1 to K (by adjusting the threshold) recall values.

In this case, the value of approximated Average precision
= (1 * (0.2-0)) + (1 * (0.4-0.2)) + (0.66 * (0.4-0.4)) + (0.75 * (0.6-0.4)) + (0.6 * (0.6-0.6)) + (0.66 * (0.8-0.6)) + (0.5 7 * (0.8-0.8)) + (0.5 * (0.8-0.8)) + (0.44 * (0.8-0.8)) + (0.5 * (1-0.8)) = 0.782.

= (1 * 0.2) + (1 * 0.2) + (0.66 * 0) + (0.75 * 0.2) + (0.6 * 0) + (0.66 * 0.2) + (0.57 * 0) + (0.5 * 0) + (0.44 * 0) + (0.5 * 0.2) = 0.782.

It can be seen from the calculations that the recall values do not change (red values) and do not contribute to the increase of average precision values. interpolated Average Precision

Unlike approximated Average Precision, some authors choose another standard for measuring performance: interpolated Average Precision. This new algorithm no longer uses P (k), which means that the value of the precision when the system recognizes the K picture is multiplied by the recall change value. Instead use:

That is, each time you use the precision in all thresholds, the precision value of the maximum value is multiplied by the recall change value. The formula is as follows:

The image below is approximated Average Precision compared to interpolated Average Precision.
It should be noted that in order to make the feature more obvious, the parameters used in the picture have nothing to do with the example mentioned above.

It is obvious that approximated Average precision is very close to the precision curve, and the Average precision value calculated by using interpolated Average precision is significantly higher than that of approximated The Average precision method is calculated to be high.

Some important articles are used interpolated Average Precision as the measurement method, and directly called the calculated value of Average Precision. PASCAL Visual Objects Challenge has been using this metric system since 2007, and they believe that this method can effectively reduce the jitter in the Precision-recall curve. So when you compare average Precision values in an article, it's best to make sure that they're using that metric. IoU

IOU This value, you can understand that the system predicted the box and the original picture in the mark of the box overlap degree.
The calculation method is the detection result and the intersection of the Ground Truth, which is the accuracy rate of the detection:
Iou=detectionresult⋂groundtruthdetectionresult⋃groundtruth IoU = \frac{detectionresult⋂groundtruth} { Detectionresult⋃groundtruth}

As shown in the following illustration:
The blue box is: Groundtruth
The Yellow box is: Detectionresult
The green box is: Detectionresult⋂groundtruth
The red box is: Detectionresult⋃groundtruth

to say.

1, this article refers to the following blog
https://sanchom.wordpress.com/tag/average-precision/
http://blog.csdn.net/eddy_zheng/article/details/52126641
2, in the Training YOLO V2, will appear these several parameters, therefore summarizes in this, saves later forgets.
3, this article is just a learning note, the content may have errors, only for reference.
4, if you find the error in the article, please comment, thank you.
5, and then continue to put the training YOLO problems in the process of writing on the blog.