How to prevent the Elasticsearch cluster from having a brain fissure phenomenon

What is a "brain fissure" phenomenon?

Due to the failure of some nodes, the network connection of some nodes will be broken, and a cluster with the same name as the original cluster is formed, which is known as the Cluster Brain fissure (split-brain) phenomenon. This is a very dangerous issue because two newly formed clusters will index and modify the cluster's data at the same time.

How to avoid brain fissure problem?

One of the parameters used to avoid brain fissures is: discovery.zen.minimum_master_nodes. This parameter determines how many nodes (the minimum number of candidate nodes) will be required to elect a master. The default value is 1. According to general experience this is generally set to N/2 + 1,n is the number of nodes in the cluster, such as a cluster with 3 nodes, Minimum_master_nodes should be set to 3/2 + 1 = 2 (rounded down).

Another parameter used is: Discovery.zen.ping.timeout, waiting for the ping response timeout, the default value is 3 seconds. If the network is slow or congested, it is recommended to slightly increase this value. This parameter is not only suitable for higher network latencies, but also for a node that is slow to respond due to overload.

If you are just starting out with Elasticsearch, it is recommended to build a cluster of 3 nodes, which can set Discovery.zen.minimum_master_nodes to 2, which limits the possibility of brain splitting. and maintains a high degree of availability: if you set up a replica, the cluster can still run if you lose a node.

Do you really have peace of mind?

In fact, the problem still exists, ES of the issue space is also discussing a special case of "#2488": Even if minimum_master_nodes set a correct value, brain fissure can occur.

How do I identify this problem?

It is important to identify the problem as quickly as possible within your cluster. An easy way is to periodically get each node/_nodes response, it returns the status of all nodes in the cluster report, if the two nodes returned by the cluster state is not the same, is a brain fissure situation occurred warning signal.

New Solutions

For a full-featured ES node, you must have an active master node. After Es1.4.0.beta1, there is a new block cluster operation setting without master: Discovery.zen.no_master_block.

When there is no active master node in the cluster, the setting specifies which actions (read, write) need to be rejected (that is, blocking execution). There are two setting values: All and write, and the default is Wirte.

This configuration does not affect basic APIs such as cluster status, node information, and status APIs, and these nodes are not blocked from being executed on any node.

Summarize

The problem of brain fissure is still a difficult problem to solve, the final solution is also the result of compromise. This problem is also the problem that the distributed system will face. Suddenly think of the last few days to see the CAP theory, is it only CP or AP?
The overall feeling ES is still very young, but because of its out-of-the-box, natural clustering, automatic fault tolerance, strong extensibility and other advantages, or choose it to do full-text search.

Resources

• How to AVOID the split-brain problem in ELASTICSEARCH
• Minimum_master_nodes does not prevent split-brain if splits is intersecting #2488
• Zen Discovery
• "Mastering Elasticsearch (Chinese version)"

http://xingxiudong.com/2015/01/05/resolve-elasticsearch-split-brain/

How to prevent the Elasticsearch cluster from having a brain fissure phenomenon