Evaluation indicators for information retrieval (Precision, Recall, F-score, MAP)

Previously wrote a blog called Machine Learning Combat notes non-equilibrium classification problem: http://blog.csdn.net/lu597203933/article/details/ 38666699 the precision and Recall and ROC are explained, the difference is Precision,recall, F-score, MAP is mainly used for information retrieval, and Roc The curve and its metric AUC are mainly used for classification and identification,ROC 's detailed introduction to the above blog, here precision, Recall and the previous blog calculation results are actually the same, but here from the perspective of retrieval to understand.

A: Precision, Recall, F-score

Information retrieval, classification, identification, translation and other fields two most basic indicators are recall (Recall rate) and accuracy (precisionrate), recall rate is also called recall rate, accuracy is also called precision ratio, concept formula:

Recall rate (Recall) = The total number of related files/systems retrieved by the system

Accuracy (Precision) = The total number of retrieved files/systems retrieved by the system

The diagram shows the following:

Note: (1) The accuracy rate and recall rate are mutual influence, under the ideal situation is certainly to do both high, but generally accurate rate is high, recall rate is low, recall rate is low, accurate rate is high, of course if both are low, that is what place problem.

(2) If it is to do a search, that is to ensure that the recall of the situation to improve the accuracy rate, if the disease monitoring, anti-litter, is the accuracy of the conditions, to improve the recall.

Therefore, when both are required high, it can be measured by F1 (or F-score). The calculation formula is as follows:

F1= 2 * p * r/(P + r)

(1)   The formula is basically the case, but how to calculate the graph 1 a b Span style= "Color:rgb (51,51,51)" >, c , d This requires manual labeling, the manual labeling of data takes a lot of time and is boring, if only to do experiments can use the ready-made corpus. Of course, there is a way to find a more mature algorithm as a benchmark, using the results of the algorithm as a sample to compare

(2) The image of the intuitive understanding is Recall requirements are all, rather than kill 1000, can not let a person, so Recall will be very high, but Precision will be the lowest. For example, all the samples are judged as a positive example, this is Recall will be equal to 1, but a lot of negative samples are treated as a positive example, in some cases is not applicable, such as mail filtering, at this time, the requirement is accurate rate, It is not a recall rate, and it is definitely the worst-case scenario for all messages to be treated as spam ( recall=1 at this point ).

If there is no evidence that you are guilty, then you are guilty and the recall rate will be high; if there is no evidence that you are guilty, then you are not guilty, the recall rate will be very low, not all, many people go unpunished;

Two: MAP

MAP: Full name mean average precision (average accuracy rate). MAP is to solve P , R , f-measure The single point value is limited , taking into account the ranking of the results of the search.

The calculation is as follows:

Suppose there are two themes, topic 1 has 4 related pages, and theme 2 has 5 related pages. A system retrieves 4 related pages for topic 1, Rank 1, 2, 4, 7, and 2 related pages for topic 3, with Rank 1,3,5. For Topic 1, the average accuracy rate is (1/1+2/2+3/4+4/7)/4=0.83. For topic 2, the average accuracy rate is (1/1+2/3+3/5+0+0)/5=0.45. Then map= (0.83+0.45)/2=0.64. ”

Reference documents:

1:http://blog.csdn.net/marising/article/details/6543943 Information Retrieval (IR) evaluation indicators-accuracy rate, recall, F1, MAP, ROC, AUC

2:http://blog.sina.com.cn/s/blog_662234020100pozd.htmlmap (Mean Average Precision)

3:http://wenku.baidu.com/view/ef91f011cc7931b765ce15ec.html

Evaluation indicators for information retrieval (Precision, Recall, F-score, MAP)