Remember the resolution of Shard unassigned issues after Elasticsearch restart

Remember the resolution environment for Shard unassigned issues after a elasticsearch restart

• ElasticSearch6.3.2, three node cluster
• Ubuntu16.04
• An index named user, with the index configured as: 3 primary shard, each primary shard 2 replica

Under normal circumstances, the distribution of each shard is as follows:

As you can see, the three shards of the user index are evenly distributed across the machines, with the ability to tolerate a single machine outage without losing any data.

The node-151 node was down due to a single failure (a word breaker was modified, but the plugin failed to load correctly). After fixing the problem, it performs ./bin/elasticsearch -d a normal startup, but discovers that there are three unassigned shards in the cluster. It was thought that these unassigned shards could be automatically assigned after node-151 normal startup, but found that it was not automatically assigned.

Workaround

First: GET user/_recovery?active_only=true discover that the cluster is not recovering from a replica.

Perform GET _cluster/allocation/explain?pretty discovery:

"Explanation": "Shard have exceeded the maximum number of retries [5] on failed allocation attempts-manually call [/_clus Ter/reroute?retry_failed=true] To retry, [unassigned_info[[reason=allocation_failed], at[2018-09-29t08:02:03.794z] , Failed_attempts[5], Delayed=false, details[failed Shard on Node [mkkj4112t7alec2onouorg]: Failed to update mapping for I Ndex, failure mapperparsingexception[failed to parse mapping [profiles]: Analyzer [Hanlp_standard] not found for field [det Ails]]; Nested:mapperparsingexception[analyzer [Hanlp_standard] not found for field [details]; ]

The original is a word plug-in error caused. Look closely at the log, one line:

Allocation_status: "No_attempt"

The reason is that the Shard automatic allocation has reached the maximum number of retries 5 times, still failed, resulting in the "Shard distribution status is: No_attempt". At this point in Kibana Dev Tools, execute the command: POST /_cluster/reroute?retry_failed=true ready. The index.allocation.max_retries maximum number of retries is controlled by the parameter.

The cluster would attempt to allocate a shard a maximum of index.allocation.max_retries times in a row (defaults to 5), BEF Ore giving up and leaving the Shard unallocated.

When the reroute command is re-routed to the Shard, the elasticsearch is automatically load balanced, and the load balancer parameter cluster.routing.rebalance.enable defaults to True.

It is important to note the after processing any reroute commands Elasticsearch would perform rebalancing as normal (Respe Cting the values of settings such as cluster.routing.rebalance.enable) in order to remain in a balanced state.

After a period of time: execution GET /_cat/shards?index=user can see that all the Shard allocations in the user index are in normal condition.

User 1 P STARTED 13610428 2.6GB node-248
User 1 R STARTED 13610428 2.5GB node-151
User 1 R STARTED 13610428 2.8GB node-140
User 2 P STARTED 13606674 2.8GB node-248
User 2 R STARTED 13606674 2.7GB node-151
User 2 R STARTED 13606684 3.8GB node-140
User 0 P STARTED 13603429 2.6gb node-248
User 0 R STARTED 13603429 2.6GB node-151
User 0 R STARTED 13603429 2.7GB node-140

First column: index name; the second column identifies whether Shard is primary (p) or replica (R), the state of the third column Shard, the fourth column: the number of documents on the Shard, and the last column node name.

Summarize

In general, Elasticsearch automatically assigns those unassigned shards, and when some shards are found to be unassigned for a long time, first look at whether it is because they specify too many primary shard and replica quantities for the index. Then the number of machines in the cluster is not enough. Another reason is mentioned in this article: As a result of the failure, shard automatic allocation reached the maximum number of retries, then the execution of reroute can be.

Resources

/_cat/shards command: https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-shards.html

2018.9.30
Original: https://www.cnblogs.com/hapjin/p/9726469.html

Troubleshoot Shard unassigned issues after Elasticsearch restart