When using Lucene, the first thing to do is to create an index file, which is a very time-consuming task, especially when Indexing large data volumes.
Lucene provides several optimization parameters
Mergefactor, maxmergedocs, and minmergedocs are described as mergefactor. The default value is 10, which controls the merging frequency and size of the index segment. That is, when 10 document objects are added to the index segment, lucene will create a new segment on the disk. After 10 such segments are created, the 10 segments will be merged into one segment, and so on, when the number of document objects in this segment does not exceed the value of maxmergedocs, the document objects are merged according to this rule, and the number of index segments in the disk directory is limited to 10.
When a small mergefactor is set, a large number of disk operations will be performed, but the advantage is that a small number of index files will be generated, which leads to conflicts.
If mergefactor is small, the index file creation is very slow, but the search is relatively fast.
How can we solve such conflicts? I have proposed the following solution and used this method in our project.
Let me explain our requirements first.
We make the content in the database into an index file and provide search. The database contains millions of data and it takes about one and a half hours to create an index.
Program On a machine)
Okay, no nonsense. Let's talk about our approach.
Lucene provides two methods: fsdirectory and ramdirectory. Here we first use ramdirectory, that is, we create an index file in the memory. The number of stored documents depends on your memory, in this way, no matter how much mergefactor is specified and how many times the index segments are merged, they are all operated in the memory, reducing Io operations, when the number of documents is greater than 5000, for example, the contents in ramdirectory are merged into fsdirectory through fsdirectory. addindexes (directory [] {ramdirectory}); implementation.
This means that I/O operations are performed every time you add 5000 documents. If you set mergefactor to 5000, the problem is that if your data volume is large enough, this allows you to generate more index segments.
The above is my approach and I hope there will be a better way to communicate.
