The intersection of Lucene merge inverted Table Algorithms

As you may know, Lucene uses the inverted table data structure in traditional search engines. when searching, suppose we want to query "+ (A: Test) + (B: test1)", first we need to query the inverted table containing the test keyword in field, then we can query the inverted table that contains the test1 keyword in field B, and then perform the merge operation on the two inverted table structures. The intersection of the two is the result of our query.

Of course, this is just one example. In actual situations, we may encounter more
Multiple operations on the inverted table: intersection, union, and difference set. This article describes how Lucene processes the intersection: Merge the inverted table to generate the sumscorer result.

Step 1: Filter and filter:

Check each inverted table first: Each inverted table is a docidsetiterator. If the list in one of the inverted tables is empty, it indicates that the intersection must be empty and you do not need to perform the following work:

 

For (INT I = 0; I <scorers. length; I ++) {<br/> If (scorers [I]. nextdoc () = no_more_docs) {<br/> // if even one of the sub-scorers does not have any documents ENTs, this <br/> // scorer shocould not attempt to do any more work. <br/> lastdoc = no_more_docs; <br/> return; <br/>}< br/>}The time complexity is at the O (n) constant level.

Step 2: sort the inverted table array. The result is that the inverted table array is sorted by the first docid of each inverted table:

 

Arrays. sort (scorers, new comparator () {// sort the array <br/> Public int compare (Object O1, object O2) {<br/> return (scorer) o12.16.doc ID ()-(scorer) o22.16.doc ID (); <br/>}< br/> }); 

Step 3: delete useless docid: because it is an intersection of multiple inverted tables, you need to filter out the smaller docid In the inverted table first:

If (donext () = no_more_docs) {<br/> // The scorers did not agree on any document. <br/> lastdoc = no_more_docs; <br/> return; <br/>}< br/> donext (): What this method does is: for example, in the inverted table array, the first docid of each inverted table is 1, 3, 4, 5, 6, and 7, respectively. Because each inverted table iterator is in ascending order, 6. It is not in the last inverted table. Therefore, each inverted table should start from 7 instead of 1: <br/> int first = 0; <br/> int Doc = scorers [scorers. length-12.16.doc ID (); <br/> scorer firstscorer; <br/> while (firstscorer = scorers [first]) . Docid () <DOC) {<br/> Doc = firstscorer. Advance (DOC); <br/> first = scorers. Length-1? 0: First + 1; <br/>}< br/> return Doc; <br/> Advance Method: <br/> If (lastdoc = no_more_docs) {<br/> return lastdoc; <br/>} else if (scorers [(scorers. length-12.16).doc ID () <target) {<br/> scorers [(scorers. length-1)]. advance (target); <br/>}< br/> return lastdoc = donext ();