Factors Affecting indexing speed by Lucene-mergefactor, maxmergedocs, and rambuffersizemb

When the indexing algorithm determines the index speed of Lucene, three parameters are most affected: mergefactor, maxmergedocs, and rambuffersizemb in indexwriter. These parameters are nothing more than controlling the internal and external memory exchanges and the frequency of index merging to increase the index speed. Of course, these parameters must be set flexibly according to hardware conditions.

Maxmergedocs

This parameter determines the number of documents written into the memory index. After this number is reached, the memory index is written to the hard disk to generate a new index segment file.
Therefore, this parameter is also a memory buffer. In general, the larger the index, the faster the speed.
The maxbuffereddocs parameter is disabled by default, because Lucene uses another parameter (rambuffersizemb) to control the number of index documents of this bufffer.
In fact, the maxbuffereddocs and rambuffersizemb parameters can be used together. If one of the trigger conditions is met, the data is written to the hard disk and a new index segment file is generated.

Rambuffersizemb

Controls the memory upper limit for Buffer Index documents. If the number of buffer index documents reaches the upper limit, the files will be written to the hard disk. Of course, in general, the larger the index, the faster the speed.
When we are not sure about the document size, this parameter is quite useful and will not cause outofmemory error.

Mergefactor

This parameter is used for merging sub-indexes (segment.
In Lucene, indexes are written to the memory. After certain restrictions are triggered, the indexes are written to the hard disk to generate an independent sub-index called segment in Lucene. In general, these sub-indexes need to be merged into an index, that is, optimize (). Otherwise, the retrieval speed may be affected and the open too program files may also be caused.
The mergefactor parameter is used to control the number of sub-indexes in the hard disk, so we need to merge these indexes into a slightly larger index.
Mergefactor cannot be set too large, especially when maxbuffereddocs is relatively small (more segment), otherwise it may cause open too program files errors, or even virtual machine external errors.

Note: The default index merge mechanism in Lucene is not a merge of two shards. It seems that multiple segments are merged into the final large index. Therefore, the larger the mergefactor is, the more memory is consumed, and the faster the index is, but I feel too big, for example, 300. The final merge is still full. Batch indexing should be mergefactor> 10