Baidu search R & D department: synonym Feedback Mechanism

1. Introduction

Due to the limitations of the search algorithm itself, the user's semantic and intent understanding is not enough, and the click adjustment based on user behavior is used as a supplement to the traditional search algorithm, it plays an important role in search. Although user behavior has been proven to be effective in the search, it remains at the query-URL level or the Ngram-URL level [1] without in-depth feedback on the basic policies in the search algorithm, for example, synonyms, closeness, and omission affect the relationship between URLs and queries. This paper uses synonym feedback as an example to propose a general basic policy feedback framework based on user behavior.

Due to the limitations of the synonym dictionary and online application algorithms, some synonyms with poor quality in the retrieval system or which are of good quality but are incorrectly used reduce the weight value. After a synonym is recalled, it is displayed in front of the user. user behavior data can help us identify the synonyms. After calculating the synonym quality, it can be directly applied to the exit of the synonym or adjust the weight of the application.

 

2. Feedback framework

In the mining of feedback mechanisms, there are three main parts:

1) log records. This module records user behaviors in basic policies and collects statistics on user behavior data using query-URL. It solves the problem of how to use user behavior to measure query-URL escape. This part also records the policies that affect the specific query-URL. For example, the URL is recalled by a synonym or the term is omitted.

2) Feedback Mechanism mining. Statistics on basic policies are performed based on the user behavior data of the basic policies in query-URL. Different Basic Policies in this region can be measured in the same way, but the information extracted by basic policies is different. For example, the synonym is a replacement pair, and the omission refers to the omitted term.

3) online feedback applications. Apply the dictionary mined in step 2 to specific queries, such as context matching and some application policies.

The above framework is general. The following is a detailed discussion of synonym feedback.

3. log record and statistics

In this section, you must first record specific policies. For example, in this query, the basic policy that affects each URL is more specific. For example, synonyms need to be recorded by specific synonyms. Because a query usually has many synonyms, but the actual number of each URL is only affected by the number of synonyms between 1 and 2.

It is a key step to determine whether a query-URL is escaped. This article focuses on this. The measurement method requires the use of user behavior. In the log system of the search engine, the query-URL has the following user behavior statistics: (in the following discussion, URL statistics are related to queries and are not described in detail)

Display times: the number of times that the search engine returns the first K URLs displayed after a user's search (Display)

Clicks: the number of clicks on a URL)

Satisfied clicks: Determine whether the click meets the user's needs (relative stay time, whether it is the last click) (satisfy)

Therefore, we can use click/disply, satisfy/display to measure the URL quality. But there are the following problems:

1. location offset problem: the number of clicks is very sensitive to the location. In the search results, the number of clicks of a URL increases with the ranking of the URL. The more clicks, the faster the number of clicks decreases. Therefore, although the URL at the beginning is escaped, many users click it. On the contrary, the URL at the end meets user requirements, but few users click it. This easily invalidates our feedback system.

2. In the search engine, users' satisfaction with the search results can be roughly divided into two levels: 1) whether the title and abstract of the retrieved URL are consistent with the intent of the user's query. 2) whether the quality of the URL content meets the user's needs, such as whether the URL is dead-chain, no answer to the page, and cheating pages. Our goal is to identify escape substitution word pairs, which are only related to 1st levels of satisfaction. We can assume that since the user clicks this URL, the title abstract of this URL is not escaped. The quality of the webpage is not affected by the quality of the synonym.

In order to solve problem 1, we can consider it from this perspective. The reason for the low number of URL clicks at the end is that the number of user views is small. Therefore, the ratio between the display and click is not allowed. You can use some methods to estimate the number of user views, this is called check ). Here are some simple methods. For example, for each user's search, if the last URL clicked by the user is P, the URL check times before P are 1, the check times of the URL after P are one probability in turn. These probabilities can be learned using Bayesian methods. [2]

The number of checks can partially solve the location bias problem, but the learned attenuation parameter is for all query-URLs, but there is a big difference between different query-URLs, this is also the shortcoming of this method.

4. Feedback mining and application feedback mining 4.1

Based on the log record operation in chapter 1, the number of clicks can be used to indicate the number of times the URL meets the query, while check-click indicates the number of times the URL does not meet the query. In this way, the value of click/(check-click) is used to indicate that the URL meets the query level. For specific synonym feedback tasks, you can replace the same synonyms recorded in multiple query-URL results to calculate the number of clicks and checks (that is, the key is the Binary Group of the original word replacement word ), use the last click/(check-click) as the similarity to measure the synonym replacement, that is, the feedback replacement similarity of the synonym:

Another major problem in this area is that many synonyms are contextual, for example, considering a pair of synonyms-> treatment, in some contexts, such:Which of the following is better?Is synonymous. In some contexts, for example:Where to watch the broadcast?. Therefore, for more intelligent synonym feedback in different contexts, you need to consider the context During statistics, that is, the key of the statistics is: the original word context replaces the word triple.

However, the entire query cannot be used as the context, so the statistics will have a large degree of data sparsity. However, if a single word is used as the context, there will be a high accuracy problem. For exampleWherePairView->TreatmentAndView->WatchAll are supported. Therefore, a context selection algorithm is required to ensure the sparsity and accuracy of Context data. In natural language processing, the likelihood ratio (LLR, likelihood ratio) [3] is usually used to measure the matching strength of orig and context, so that the matching intensity is stronger, the context word can be considered as the replacement context of the orig word. The calculation method is as follows:

Here, a indicates the number of orig and context co-occurrence; B indicates the number of orig S and context s; C indicates the number of orig S and context S; d indicates the number of times the oirg and context do not appear. N = A + B + C + D indicates the total number of samples. Then, the LLR formula is as follows:

4.2 feedback Application

When applying a feedback mechanism, the system makes an independent judgment on each replacement, that is, the known replacement pair (orig sub). Context Selection is required first. In essence, a word to be replaced is a synonym. For most queries, only one context word can limit the meaning of a word to be replaced. Therefore, from a simple perspective, as well as the noise caused by context fusion of multiple words and the fusion mode, the feedback mechanism only selects one word in a certain context window.

Finally, the selected context is calculated, and the training data in section 4.1 is used as the feedback similarity of replacement, that is, the SIM (orig, contex, sub ). Use this value as the Synonym's confidence level to apply it to online: Back, drop, or escalate.

5. Summary and prospects

In the retrieval system, user behavior-based feedback on basic policies is a new direction and is of great significance for improving basic data. Based on the in-depth research on user behavior, this article discusses some methods and indicators.

In general, the framework is equivalent to two assumptions: the relationship between user behavior and relevance is positively correlated, And the URL relevance is positively correlated with the correctness of basic policies.

The first hypothesis involves the research and consideration of basic statistical features. Click Check is one of the characteristics of these relationships. You can also consider more features, such as satisfied clicks and URL entries. There are also statistics on the impact of blushing on clicks, users' cheating identification, and other basic interference features. Different Basic Policies can be unified.

The second hypothesis involves the form in which basic policies represent these basic statistical features. This is closely related to basic policies. For example, you can use a synonym to select the context method, the context location, multiple contexts, or do not need to replace the context for recognition. In addition, you also need to pay attention to the application of basic policies, such as synonym escaping and URL escaping, which may mislead the identification of basic policies.

From the perspective of machine learning, this method mainly starts from the perspective of model generation. Therefore, the steps of the model are strongly explanatory, but more features cannot be used, more features can be mined and used by machine learning.

6. References

[1] huihsin T, longbin C, Fan Li etc. 2009. Mining search engine clickthrough log for matching n-gram features. Proceedings of the 2009 Conference on emnlp, 524-533.

[2] Ricardo Baeza-Yates, Carlos Hurtado, etc. Modeling user search behavior. In La-web 05

[3] Christopher D. Manning, Hinrich schutze. Foundations of statistical natural language processing. The MIT Press. 172-175

Baidu search R & D department: synonym Feedback Mechanism