Learning to rank evaluation index map ndcg

Map (mean average precision): an indicator for evaluating the performance of a rank method. Now there are many queries. First, let's take a look at how the rank method performs for each query (that is, AP ), on average, map is used. How does AP calculate it? For example, there is a query. The related documents include 4--d1, D2, D3, and D4. After the rank method is used, the order of D2, D3, and D4 is 1, 3, 5; D1 is not found. Then ap = (0 + 1/1 + 2/3 + 3/5)/4. Do you find out? First, the related information is not retrieved, and the AP contribution is 0. In the denominator, 1, 3, and 5 are the results after rank. And then average, that is, divide by 4.

This method has a problem that doesn't take into account the sorting between D1--D4 and they, just look at relevant, or not.

Ndcg (normalized discounted cumulative gain): for example, there are four documents: D1, D2, D3, D4, and a query. In map, the four documents are either related or irrelevant to the query, that is, the relevance is not 0 or 1. Here we have improved the degree of correlation from 0 to R, where R is set manually. Here we take 3. For this query, the relevance of the four documents is, respectively. Ideally, the order should be D2, D1, D3, D4 ), the rank method is sorted as D1, D2, D4, D3 (2, 3, 1, 1 ). In this case, the DCG value is: rel1/log2 (1 + 1) +
Rel2/log2 (2 + 1) + rel3/log2 (3 + 1)/rel4/log2 (4 + 1) = 2/log2 (2) + 3/log2 (3) + 0/log2 (4) + 1/log2 (5 ). Log2 is the weight, because the more you think about it, the less valuable the document will be. If you rank the most relevant item behind it, the worse it will be.

Finally, the DCG obtained by the DCG value/rank algorithm when ndcg = ideal sorting is calculated = [3/log2 (2) + 2/log2 (3) + 1/log2 (4) + 0/log2 (5)]/[2/log2 (2) + 3/log2 (3) + 0/log2 (4) + 1/log2 (5 )]

Evaluation indicators such as ndcg map are usually non-smooth (continuous.