[Elasticsearch] adjacent match (3)-performance, associated word query and Shingles

[Elasticsearch] adjacent match (3)-performance, associated word query and Shingles
Improve Performance

Phrase and closeness queries are more expensive than simple match queries. The match query only checks whether the entry exists in the Inverted Index, while the match_phrase Query Needs to calculate and compare Multiple possible duplicate entries (Multiple possibly repeated.

In Lucene Nightly Benchmarks, a simple term query is about 10 times faster than a phrase query, and about 20 times faster than a neighboring query (a phrase query with slop. Of course, this price is paid during the search period rather than during the index period.

TIP

Generally, the extra cost of phrase query is not as scary as the numbers. In fact, the performance difference only shows how fast a simple term query is. Phrase queries on standard full-text data can generally be completed within milliseconds, so they are fully usable in the actual production environment, even in a busy cluster.

In some specific scenarios, phrase query may consume a lot of resources, but this is not often the case. A typical example is the DNA sequence, where there will be a lot of identical duplicate entries in many locations. Using a High slop value will significantly increase the location calculation.

Therefore, how can we limit the performance consumption of phrase and proximity queries? A useful method is to reduce the total number of documents that require phrase queries for inspection.

Rescoring Results)

In the previous section, we discussed how to adjust the relevance by using the degree of proximity query, instead of using it to add or exclude documents from the result list. A query may match millions of results, but our users may only be interested in the results of the previous pages.

A simple match query has sorted documents containing all search terms before the result list. However, we only want to re-sort the preceding results to give additional relevance to documents that match the phrase query at the same time.

The search API supports this row through Rescoring. In the score re-calculation stage, you can use a more expensive score calculation algorithm-for example, a phrase query-to re-calculate the score for the first K results of each shard. Then these results will be re-ordered based on their new values.

The request is as follows:

GET /my_index/my_type/_search{    "query": {        "match": {              "title": {                "query":                "quick brown fox",                "minimum_should_match": "30%"            }        }    },    "rescore": {        "window_size": 50,         "query": {                     "rescore_query": {                "match_phrase": {                    "title": {                        "query": "quick brown fox",                        "slop":  50                    }                }            }        }    }}

Match query is used to determine which documents will be included in the final result set, and the results are sorted by TF/IDF. Window_size is the number of scores that need to be recalculated on each shard.

Finding Associated Words)

Although phrase and closeness queries are useful, they still have a disadvantage. They are too strict: all entries in the phrase query must appear in the document, even if slop is used.

The flexibility obtained through slop to adjust the word order is also costly because you lose the association between words. Although you can identify sue, alligator, and ate in the document, you cannot determine whether it is Sue ate or alligator ate.

When words are used together, they represent more meanings than when used separately. "I'm not happy I'm working" and "I'm happy I'm not working" contain the same words and have similar closeness, but they have different meanings.

If we index word pairs instead of independent words, we can retain them.