about network partition
Network fragmentation caused by network device failure. For example, there are a\b\c\d\e five nodes, a\b in the same subnet, b\c\d in another subnet, in the middle through the switch connected. If the switch between the two subnets fails, the network partition occurs, and a\b and c\d\e cannot communicate.
Some systems are partition-tolerant, that is, even if the network partition system is split for multiple subsystems, the whole system can still work properly.
RabbitMQ cluster cannot handle network Partition very well. RABBITMQ stores information such as queue, exchange, bindings, and so on in the distributed database Mnesia of Erlang. So the RABBITMQ behavior of the network partition is closely related to Mnesia behavior.
detection of Network partition
If a node cannot get in touch with another node for a period of time (depending on the settings of the Net_ticktime), then Mnesia thinks that the node that has not been contacted is down. If two node resumes contact with each other, but all thought that the other side has been down, then Manesia decided that the network partition occurred.
behavior of RABBITMQ after network partition occurs
If the two parties (or multiple) in the network partition,cluster will exist independently, each party will assume that the other party has collapsed. Queues, bindings, exchanges can be individually created, deleted. For mirrored queues, each party in a different network partition will have its own master and read and write independently. (Other bizarre behavior may also occur). If the network partition is restored, the cluster status does not automatically revert to the state before the network partition occurs until the action is taken to fix it.
the partitions caused by Suspend/resume
As long as the different node in the cluster itself does not fail, but the communication between the interruption can be considered to have occurred partitions. For example, the suspension of the entire OS will cause the cluster of the nodes to hang, but these nodes do not think they are dead or stopped, and the other nodes in cluster cannot get in touch with them, and the nodes is considered down. For example, if a node in cluster is running on a laptop computer, closing the computer screen may cause node to hang. In addition, if node in cluster is running in a virtual machine, the hypervisor may cause the virtual machine to hang, causing node to hang.
How to recover from network partition
First select the most trusted Partition,mnesia to use the state in the partition, and the changes that occur in the other partitions will be lost.
Stop all nodes in the other partitions, and then restart these nodes. When these nodes rejoin cluster, the state will be restored from the trusted partition.
Finally, you need to restart all the nodes in the trusted partition to clear the network partition warning message
RABBITMQ Automatic Processing partitions
RABBITMQ provides two ways to automatically process network partitions: pause-minority mode and Autoheal mode (the default is Ignore mode, which requires manual processing)
In pause-minority mode, when the other nodes is detected as down, RABBITMQ will automatically pause to consider himself a minority nodes (e.g. less than or equal to half of the total nodes), and once the network partition occurs, the "minority" Nodes will be immediately suspended until the end of partition is resumed. This guarantees that at most one nodes in the partition will continue to run when the network partition occurs. (Sacrificing availability to ensure consistency)
In Autoheal mode, the PARTITION,RABBITMQ will automatically determine a winning partition once it occurs, and then restart all nodes that are not in the winning partition. The winning partition is the partition with the most client connections (the partition with the most nodes if the connection is the same). The settings for automatic processing of partitions are in the cluster_partition_handling parameter of the configuration file.
two scenarios for automatic processing of partitions modes
Network partitions automatic processing does not guarantee that the cluster will not have any problems. Generally, the following options are available:
Ignore: If the network is very reliable. All nodes are in the same rack, connected via a switch, which is also an outlet to the external network. You do not want the remainder to be affected when some part of the cluster fails. or cluster only two node.
Pause_minority: The network is less reliable. The cluster is in the 3 AZ of EC2, assuming that at most one AZ failure at a time, the nodes in AZ that wants the remaining AZ to continue to service and fail is automatically added to the cluster after AZ resumes.
Autoheal: The network is unreliable. Pay more attention to the continuity of service than data integrity. Cluster has only two node.
about pause-minority Mode
A paused nodes on an Erlang VM will continue to run without listening to any ports or doing other work. They will detect if the other nodes in the cluster are visible once per second and wake from the pause state if they are visible.
Attention:
Nodes will not enter the paused state at startup, even in "minority";
RABBITMQ may suspend nodes in the "minority" in the non-strict sense. For example, contains more than half the total nodes total of nodes. It is therefore not a good idea to use pause-minority mode in a cluster that contains only two nodes, because it is possible that two node will pause when network partition occurs or node fails. However, pause_minority patterns in cluster with more than two nodes are safer than ignore;
The partitions pause_minority pattern caused by cluster nodes hangs is powerless. Because the suspended node will not be able to see if the remaining node is "visible", it cannot trigger a break from cluster.
RabbitMQ cluster and network partition
