Inverted index optimization-Table jumping

In the previous article about inverted indexes, we know that the complexity of merging two keywords is θ (N). If the merge operation encounters the most extreme situation, the number of scans and comparisons is the sum of the numbers of all elements in the two list sets, that is, linear growth, which is very inefficient when the data volume is very large. let's take a look at the sample code for merging two sets:

A = [1, 2, 3, 6, 9, 11, 45, 67] B = [4, 6, 13, 45, 69, 98] I = J = 0 result = [] While I <Len (A) and j <Len (B): If a [I] = B [J]: result. append (A [I]) I = I + 1 J = J + 1 Elif A [I] <B [J]: I = I + 1 else: j = J + 1 print result # output [6, 45]

What happens if the two inverted tables to be merged have a large amount of data but few intersections?

[1, 2, 3, 4, 5, ... 10001, 10005][1, 10001, 10008]

If the two elements are merged, the result of the last intersection is only two elements: [1, 10001 ].10001So is there any way to optimize this. you may have already thought that we have done so many useless comparisons because every time we move the Pointer Forward, the step is too small. If we move a little more forward after each comparison, you can ignore useless operations. this is the idea of table jumping.

Let's look at the first inverted table. If it moves forward with a step of 5000, we only need to look forward to two to find the elements we need:10001. Here is the merge algorithm code for the table jumping function:

a = range(10008)b = [1, 10001, 10008]i = j = 0result = []step = 100count = 0while i < len(a) and j < len(b):    if a[i] == b[j]:        result.append(a[i])        i = i +1        j = j + 1        count = count + 1    elif a[i] < b[j]:        while (i + step < len(a)) and a[i+step] <= b[j]:            i = i + step            count = count + 1        else:            i = i + 1            count = count + 1    else:        while (j + step < len(b)) and b[j+step] <= a[i]:            j = j + 5000            count = count + 1        else:            j = j + 1            count = count + 1print resultprint counta = range(10008)b = [1, 10001, 10008]count = 0i = j = 0result = []while i < len(a) and j < len(b):    if a[i] == b[j]:        result.append(a[i])        i = i + 1        j = j + 1        count = count + 1    elif a[i] < b[j]:        i = i + 1        count = count + 1    else:        j = j + 1        count = count + 1print resultprint count

The above code intentionally constructs a large set [0... 10007], and then use the variable count as the counter to analyze the number of operations performed by the two algorithms separately. We can see that when the table jumping algorithm is used (we simulate step = 100), the number of computations is 207, the previous calculation times are 10008, which means the performance has been improved many times.

Here are some notes:

1. Here, we use arrays to represent the inverted table. In fact, the actual data structure should be the linked list structure, which is consistent with the disk storage structure.

2. the original structure algorithm of the table to be jumped is more complex than this algorithm. based on different scenarios, the table to be jumped has different implementations. the Skip table's Quick query function is not used here, so there is no multi-level pointer index concept. The detailed skip Table Implementation check: Skip List

Inverted index optimization-Table jumping