Realization of full-text retrieval of static structured data

The static structured data here refers to extracting data stored in NoSQL, text files, or other format files from a relational database, and the data no longer changes.

Full-Text search refers to the exact query, fuzzy query & range query of any field. Letters or numbers fuzzy queries support only leading queries, and Chinese fuzzy queries are not restricted.

First, when extracting the data, the data is very large data table, according to the partition by the rowID shard sub-file storage.

Second, the implementation of data query.

Idea One:

The most common way to create indexes on all fields is to create a B + Tree index because it is unclear how the data is distributed beforehand.

The implementation of Chinese fuzzy query is to use Chinese word segmentation to combine all possible Chinese word segmentation to create an index.

Idea two:

field-insensitive, field-insensitive fields create a bitmap inverted index for all values of the data table, key is the data value, index value is the file location where the value is located, and is sorted by the key value.

An index (that is, the metadata of the index) is also created for the inverted index, which includes the maximum and minimum data values for the index file and the data values for every n rows to speed up the positioning of the data.

For example, table T, the contents of a data file are as follows:

1, 430111780101777, hello, test, 2016010101

2, 430111780101778, hello, test, 2016010101

3,430111780101779, hello, test, 2016010101

Assuming that the first row has an offset of 0, the second row has an offset of 1, and the third offset is 3, then the corresponding inverted index is:

1:0

2:1

3:2

2016010101:0,1,2

430,111,780,101,777:0

430,111,780,101,778:1

430,111,780,101,779:2

Test: 0,1,2

Hello: 0,1,2

The corresponding inverted index meta-data is as follows:

[Data]

Min:1

Max: Hello.

[Index]

1:1

Realization of full-text retrieval of static structured data