original, reproduced please be sure to keep the author and the source of this article (connection form).
2 years ago the content of the work, has become ephemeral, share, welcome to exchange.
1. Previous optimization effects
Before the Sphinx threading, network IO did a lot of optimization (Netty, Epoll, libevent), the index part is also optimized, from the current test results, performance has reached the bottleneck, especially all search, thread scheduling, network IO optimization for this kind of query help is very small. For a single-node performance improvement, you also need to optimize the query algorithm from the index itself. Sphinx attribute filtering has a significant impact on performance itself, but the optimization space is not very large, the properties read are discrete, and unlike indexes that can be read continuously, this operation is limited by physical hardware performance. Currently can be optimized mainly the index query part, these two days by writing some demo, simulation index and query algorithm for quantitative analysis of the search process.
2. Index Analysis
At present, only two kinds of algorithms and indexes are considered for performance analysis, the first is the Sphinx of the current query and index structure, the second is a brother earlier said a search company's bitmap index.
Now the query way performance statistics:
11,000 data, through the way of incremental compression (in order to facilitate the comparison, the increment is 1 bytes), the compressed read index time is 2597us.
After reading the buffer more than 8K, basically change little. After loading into memory, loop one read byte, do not unzip, take time 65645us,
Loop 10 million times, if the edge of the reading edge decompression, need 142438us, equivalent to 77ms of time to understand the pressure operation.
21,000 data, do not compress, in the way of int directly storage (currently sphinx with a long, for us, only save Memory property offset address,
int is enough), the storage file is 4 times times the original, read time 10470us, more than 8ms read time, loop read int, no decompression operation,
Used 86493us.
3. Two 10 million data lists do and operations, are stored in an uncompressed form, the entire operation time is 482461us, average down to 300ms of time
It's used in the comparison operation.
Based on bitmap operations:
1.20 million-bit two bitmap list, performing an and operation on the bitmap, time 1855us, basically does not consume much. To the bitmap result, traverse the query results,
This time is limited by the size of the result set. When the result is empty, the shortest time, less than 2ms, if the result is many, need 27ms.
2. Bitmap compression operation using zlib, this time-consuming, 20 million bitmap compression needs 300-600ms, compressed size and bitmap-related, instant slots
Figure, the compression also has 2 K, the maximum will not exceed the original size.
3. Bit graphic pressure operation, time-consuming needs 10-30ms, instant vacancy map, the minimum time will be 10ms.
Read the file-by-read decompression and operation extract result
Sphinx Compressed Index <3MS <65ms <77ms not tested without this action
Sphinx do not compress the index <12ms <86ms no need to extract 300ms This action is not required
Bitmap <1ms does not require this operation 10~30ms <2ms <27ms
Comparison results:
Sphinx Query mode query time in 1ms~500ms
Bitmap Query mode query time in 30ms~90ms
Bitmap and operation is fast, because the use of 64-bit and operations, equivalent to a 64 data comparison, in the inverted table is very large, the performance advantage is very prominent, if the inverted table is very small, most of the operation is useless.
New test results
It is not appropriate to store data directly using bitmaps, only when the inverted list is more than 2 million, the bitmap storage has the advantage that the decompression time is better than the storage uncompressed inverted
List time. In practice, 20 million of the data in an inverted list of more than 2 million of the data is a minority. So it's better to use an uncompressed int store,
When you read the bitmap, and then use bitmaps for subsequent and/or operations, so that you can be compatible with the previous index structure, but also to improve the performance of the inverted table merge.
3. Conclusion
Overall, Sphinx query mode performance fluctuations, in the case of large data volume, the drop is obvious, the bitmap compared to the performance fluctuations more smoothly. The current compression method used by Sphinx, in fact, does not make much sense, but increases CPU overhead. For normal searches (query results, the number of results returned), the use of bitmap advantages of this query method is more obvious. For queries that do not require a result count and return only the first n results, Sphinx queries may be faster, and bit plot compression has a minimum consumption time.
Sphinx Search Engine Optimization and testing
