Introduction to Lucene dynamic shards

In the recently developed search engine, the index needs to be fragmented. Depending on the needs of the project, we offer two ways of partitioning. Process blog record.

Hash algorithm

The principle is very simple, by the hash value of the row key (_id) to determine where the Shard, and then the operation.

Take a chestnut (example), now has an index, initialize 5 shards, respectively shard0, Shard1, Shard2, Shard3, Shard4.
Now you need to save a row of data, _id for 0001000000123,_id of the hashcode is 1571574097, 5 for the remainder (1571574097 5) is 2, so that the data should be saved in the Shard2. Here is a simple diagram:

Hash Algorithm shard Implementation is very simple, the calculation process only need to know the number of shards to complete the positioning. But also because the number of shards is part of the algorithm, the cost of modifying the number of shards is also very expensive.

One solution is to rearrange, for example, to increase from M shard to n shards, cut each shard into n small shards, and then merge all small shards into large shards. A graphic description was copied from the network,

The advantage of this approach is that you can arbitrarily set the number of new shards. The disadvantage is that all data needs to be rearranged, which can be time-consuming if the amount of data is large.

Of course, because the growth of the project data is unpredictable, we did not choose the above method of adding slices, but chose another way to increase the film.

Dynamic shards

Combined with the hash algorithm and the principle of two-fork tree, the dynamic increase of shards.

First, the hash algorithm is the same as before, when the search is created, you can set an initialized number of shards, such as initializing 5 shards, Shard_0, Shard_1, Shard_2, Shard_3, Shard_4, respectively. When adding data, the hash value of _id determines which shard the data needs to be saved to. The difference is that we set the maximum number of rows per shard, and when the number of a shard reaches the maximum row count, the Shard splits into two small shards and acts as a child shard of the current shard.

For example, set the maximum number of rows for a shard to 10 million, and when Shard_2 exceeds 10 million, split to two sub-shards shard_2_2 and shard_2_7. If the shard_2_2 data continues to grow to 10 million, split sub-shards shard_2_2_2 and Shard_2_2_12.

As can be seen from the example, fragmentation is not a rule, assuming that the initial number of shards is m,k for the binary tree depth, then the split rule for Shard N is

Shard_n split into Shard_n_n and Shard_n_ (n + M * 1)

Shard_n_n split into Shard_n_n_n and Shard_n_n_ (n + M * 2)

Shard_n_ (n + M * 1) Split to Shard_n_ (n + M * 1) _ (n + M * 1) and Shard_n_ (n + M * 1) _ (n + M * 1 + M * 2)

...

The formula above looks very complex, and we use plots to illustrate the splitting process.

If you do not understand, we can find the corresponding shards by _id to comb the idea, or the above example,

A row of data needs to be saved, the hashcode of _id to 0001000000123,_id is 1571574097, and the remainder (1571574097 5) for 5 is 2, thus determining that the data should be stored in the shard_2.

Shard_2 has been divided into shard_2_2 and shard_2_7 two sub-shards, the base of this layer is 10 (cardinality = Initialize the number of shards * layer), we will 1571574097 to 10 redundancy (1571574097 10) 7, the data is stored in the Shard _2_7.

The shard_2_7 has no sub-shards, which means that the Shard is not split and can be stored directly in the Shard.

Analyze The Shard Search principle:

1. According to the hash algorithm to find the Shard;
   

2. If the Shard has child shards, it is looked up from the sub-shards;

3. If the Shard has no child shards, the data is saved in the Shard;
   

Then analyze the Shard splitting rule, why Shard_1 split into Shard_1_1 and shard_1_6?

The reason is simple, shard_1 that the ID of the hash value of 5 to take the remainder value of 1, if shard_1 split into 2 parts, then the 2nd level of the base 10 = The previous layer of the radix * *, that is, 5 * 2. For 5, the residual value is 1, then the 10 results will only be 1 and 6, so

Shard_1 split into Shard_1_1 and shard_1_6.

Data consistency

Dynamic shards are automatically fragmented during use, and the Shard process can be very long, tested, and divided into two sub-shards in 5 million rows of index 32, taking 245 seconds. Splitting process These modifications may be lost if the original data has been modified. Therefore, it is necessary to ensure the security of data in the process of splitting.

Method One, use pessimistic locks.

Before splitting, the locked shards cannot be modified until the split is complete.

Advantages: Simple and rough logic, low development difficulty.

Disadvantage: Too long a lock can cause a large number of exception requests to be generated by the calling service.

Method Two, use the transaction log.

Create transaction log before splitting, current Shard all new, modified, and deleted operations are written to the transaction log. After the split is complete, lock the shards and sub-shards, recover the data from the transaction log to the child shards, and then unlock.

Advantage: The Shard is locked only when the transaction log is created and the data is recovered, and the lockout time is short, affecting the service callers almost unaffected.

Disadvantage: The development is very difficult, need to develop a set of transaction log and log recovery operation interface. But the underlying Lucene storage already has a set of transaction log interfaces and implementations, which can be almost negligible.

Row key Increment Shard

If the row key that holds the data is incremented on the whole, for example, the row key is 000000001,000000002,000000003,... In this format, you can slice by row key. This method of partitioning is relatively simple,

1. Set an initialization shard when the index is created;

2. Add data process, and record the Shard row key minimum value MiniD and maximum value maxid;

3. When the amount of shard data exceeds the maximum setting, a new Shard is created, and the added data is saved in the Shard;

4. When updating data, identify the Shard by comparing MiniD and Maxid with each shard.

Row-key increment shard and hash algorithm shard comparison:

1. The line key Increment Shard Method realizes simpler, the development cost is lower;

2. The row key increment shard locates the shards through MiniD and Maxid, if the MiniD and maxid of the shards need to be recorded in each shard information;

3. The row key increment the Shard to deposit the data, need to deposit in certain order, otherwise may cause the data to tilt;

4. Row-Key increment shard to add shards on demand, only need to set the maximum number of rows per shard, no splitting process;

5. Row key Increment shard A large amount of pressure is concentrated on the latest shards, hash algorithm fragmentation pressure dispersed to each shard, theoretically hash algorithm shard can support higher throughput.

Introduction to Lucene dynamic shards