Sesame HTTP: Learning to Rank overview, learningrank

Sesame HTTP: Learning to Rank overview, learningrank

Learning to Rank (L2R for short) is a machine Learning method for building a ranking model. It plays an important role in information retrieval, natural language processing, data mining, and other scenarios. The result is that, given a group of documents, documents that reflect the relevance of any query request are sorted. This article briefly introduces the basic algorithms and Evaluation Indicators of L2R.

Background

With the rapid development of the Internet, L2R technology has become increasingly popular, which is one of the common tasks of machine learning. For information retrieval, given a query target, we need to calculate the most suitable results and return them. This involves some feature calculation, matching, and other algorithms. For massive data, if we only rely on manual intervention to sort some of the parameters, it is far from meeting the requirements, and the L2R algorithm is used to solve this problem, l2R applies Machine Learning Technology to sorting, and puts forward some new theories and methods to effectively solve the Sorting Problem, in addition, the efficiency has also increased by several orders of magnitude compared with manual intervention.

L2R Algorithm

The L2R algorithm consists of Pointwise, Pairwise, and Listwise algorithms.

1. Pointwise

Pointwise converts a problem into a multiclass classification or regression problem. If it comes to a multiclass classification problem, for a Query, tag the degree of relevance between the document and the Query into a limited category, so that the problem is converted into a multiclass classification problem; if it comes down to a regression problem, for a Query, the Score is calculated based on the degree of relevance between the document and the Query, which leads to the regression problem.

Model

The Pointwise application model includes Subset Ranking, oc svm, McRank, and Prank.

Input

Specific Query, feature vector of the document.

Output

Tag category or correlation score of the document and Query.

Loss Function

Loss regression, Loss classification, and ordered regression.

Advantages and disadvantages

The Pointwise algorithm is simple to implement and easy to understand. However, it only models the relevance of a given Query single document and only considers the Absolute Relevance of a single document, pointwise only learns the global correlation between documents and queries, which has a certain impact on the sorting order. In some scenarios, the first few documents have an important impact on sorting results. For example, the first page of the search engine is very important, and Pointwise does not consider the impact, do not punish the merits or demerits of sorting.

2. Pairwise

The Pointwise method mentioned above only takes into account the Absolute Relevance between a single document and a Query. Pairwise considers the relative relevance between two documents and compares the order of different documents. Pairwise is a popular method that converts the entire sorting problem into a binary classification problem, that is, a binary classifier is built to perform binary classification on <Doc1, Doc2> in a document, one is that Doc1 is sorted before Doc2, and the other is opposite. Through comparison between two documents, the model can learn the order of different documents.

Model

Pairwise models include Ranking SVM, RankBoost, RankNet, GBRank, and ir svm.

Input

Specific Query, document pair <Doc1, Doc2>.

Output

Document bias score, {-1, 1 }.

Loss Function

Pairwise classification Loss.

Advantages and disadvantages

The Pairwise method sorts documents by considering the relevance between two documents, which makes some progress. However, Pairwise uses the loss function of relevance between the two documents, which is very different from the indicator that really measures the sorting effect, and may even be negatively correlated, if Pairwise Loss is lower and lower, the NDCG score is also lower. In addition, this method only considers the order of the two documents, and does not consider the position of the document in the search list, resulting in unsatisfactory final sorting effect.

3. Listwise

Compared with the Pointwise and Pairwise methods, the Listwise algorithm does not convert the sorting problem into a classification or regression problem, but directly optimizes the sorting result of the document based on the evaluation indicators, such as commonly used MAP and NDCG.

Model

Listwise models include ListNet, ListMLE, svm map, AdaRank, SoftRank, LambdaRank, and LambdaMART. LambdaMART (improvements to RankNet and LambdaRank) demonstrates the best performance in Yahoo Learning to Rank Challenge.

Input

Specific Query and document set

Output

Scoring or sorting of all documents

Loss Function

Evaluation indicators such as NDCG and MAP.

Advantages and disadvantages

Because this method is directly optimized for evaluation indicators, it often shows good results.

Evaluation Indicators

L2R evaluation indicators include NDCG, MAP, WTA, and MRR.

1. NDCG

NDCG, fully called Normalized Discounted Cumulative Gain, is an indicator that comprehensively considers the relationship between the model sorting result and the real sequence, and is also the most commonly used indicator to measure the sorting result, the formula is as follows:

$ \ Mathrm {NDCG @ K} = \ frac {DCG} {iDCG} $

NDCG is actually calculated by the value of DCG. the numerator calculates the value of DCG for the model, and the denominator is the ideal value of DCG. The calculation formula of DCG is as follows:

$ \ Mathrm {DCG @ K} = \ sum _ {I = 1} ^ {k} {\ frac {2 ^ {r (I )} -1 }}{\ log _ {2 }{( I + 1) }}$ $

In the DCG expression, $ \ sum _ {I = 1} ^ {k} $ indicates sum accumulation, and $ {r (I)} $ indicates in the order given by the model, the actual score of the element ranked as I. Here, the difference in the score is amplified through the ${2 ^ {r (I)}-1} $ operation, $ \ log _ {2} {(I + 1)} $ is the discount for each element. Because the elements in the top ranking are more likely to be selected, so here we can make the elements before the ranking have a greater influence on the weight.

2. MAP

MAP, all called Mean Average Precision, that is, Average accuracy. For each truly relevant document, consider its position P in the model sorting result, calculate the classification accuracy of the document set before the position, and take the average of all these accuracy values.

For a Query, there are originally four related results. During the Query, all four results are queried, and their rank is 1, 2, 4, 7, respectively, MAP is (1/1 + 2/2 + 3/4 + 4/7)/4 = 0.83. For another Query, there are originally five related results. The Query has only three related results, and their rank is 1, 3, 5, respectively, MAP is (1/1 + 2/3 + 3/5 + 0 + 0)/5 = 0.45. MAP = (0.83 + 0.45)/2 = 0.64.

3. WTA

WTA, full name: Winners Take All. For a given Query, if the first document in the result list returned by the model is related, WTA = 1; otherwise, it is 0.

For example, a Query has five related results. If the first result is related in the Query results, WTA = 1. If the first result is irrelevant, then WTA = 0.

4. MRR

MRR, short for Mean Reciprocal Rank, uses the sorting Reciprocal of relevant documents in the results as the accuracy, and then obtains the average value.

For the first Query, the rank of the correct result is 3, and the Reciprocal Rank is 1/3. For the second Query, the rank of the correct result is 2, the Reciprocal Rank is 1/2. For the third Query, the rank of the correct result is 1, and the Reciprocal Rank is 1, then MRR = (1/3 + 1/2 + 1) /3 = 11/18 = 0.61.

References

• Learning to Rank, Hang Li
• Learning to Rank for Information Retrieval, Tie-Yan Liu
• Summary of basic Learning to rank Algorithms
• Introduction to Learning to Rank
• Introduction to Smart Search and conversational OS
• Introduction to Learning to Rank
• NDCG evaluation criteria