Common performance evaluation indicators for Information Retrieval

1. Recall and precision

Traditional Information Retrieval and evaluation indicators mainly include the recall and precision ratio,

The query completion rate refers to the ratio of the number of relevant documents in the returned results to the total number of relevant documents in the system, which mainly reflects the integrity of the retrieval system recall results.

Precision ratio refers to the ratio of the number of relevant documents in the returned results to the total number of results, reflecting the accuracy of the query results of the retrieval system.

Recall = num (query result set documentation)/num (system's general Documentation)

Precision = num (relevant documents in the query results)/num (total documents in the query results)

 

The greater the total number of results, the more likely the query is to be complete, the higher the query completion rate, and the larger the number of false checks, the higher the precision. And vice versa.

A comprehensive query of comprehensive and accurate indicators: F1 (recall, precision) = 2recall * precision/(recall + precision)

 

Calculation of the full query rate (from interactive encyclopedia ):

 

Measure an Information Retrieval System

An indicator of the success degree of the relevant literature from a specific collection. The value is equal to w/X. In the formula, W indicates the number of documents with the specified caliber, X is the number of documents that are actually related to a subject in all N documents included in a specific search system. This indicator was initially set by J. W. Perry

And a. Kent

Proposed by others in 1956. F. W. lanster

In 1979, the intelligence retrieval system-characteristics, tests and evaluations

In this book (version 2), N documents are expressed as the sum of A + B + C + D, and the following 2 × 2 tables are used.

Full query Rate

　
The m documents should be a + B, W documents should be A, and X documents should be A + C. The full query rate can be expressed as a/(A + C), in formula
The value can be determined after a retrieval. The C value can be determined by the following four methods: ① if n
The value is not big. You can identify each document one by one to determine the value of C. ② If the N value is large, the undetected documents can be randomly sampled. For example, if the sampling is 1/100, where R documents are related, c = 100r is estimated. ③
If an experienced user identifies a detected document
C = 1-2/3 = 1/3. ④ Search for the documents of the same subject through different channels, add the documents detected together, remove duplicates, and form a complete list of relevant documents,
By comparing the relevant literature detected each time, you can know the corresponding C value. With values A and C, enter the formula/
(A + C), you can find the query full rate. Generally, the greater the indexing depth of a search tool, the higher the query completeness rate. The better the indexing process is, the higher the query rate. C. W. Clever Deng

In 1963, a test was conducted to reveal the recall and precision rates.

Generally, there is an inverse relationship between each other, that is, to improve the query rate, the precision is often reduced, and vice versa. 1982 in 1983, some people in China used probability theory and calculus to prove this empirical law, and made a further scientific explanation of the physical meaning of the matrix edge and corner of precision ratio.

 

2
TSAP (TREC-style
Average precision)

 

When document I is a relevant document, rI = 1/I; otherwise, I = 0. N indicates that the first n documents are used as measurement indicators. Generally, relevant documents should be listed first. The more relevant documents there are in the first n documents, the higher the positions of relevant documents, and the greater the TSAP @ n, the more accurate the retrieval system is, the better.

 

3map (mean average precision)

MAP is used to measure the accuracy of sorting results. Assume that there are only two types of documents: Correlation and non-correlation. P (n) indicates the accuracy of the First n results.

P (n) = (number of positive instances within N)/N;

The average query accuracy AP Based on P (n) is:

 

Where n indicates the total number of retrieved documents, and POs (n) indicates whether the N document is related. If yes, it is 1; otherwise, it is 0. Map is the average value of AP for all queries in the test set.

 

4 ndcg (normalized discount cumulative gain)

Ndcg is used to measure the accuracy of sorting results, and document relevance is used to evaluate documents rather than binarization documents.

R (j) indicates the related score of Article J. Zn is a standardized factor for N. ndcg only calculates the detected document.