[Erlang_question14] How do I simulate various failures after node interconnection?

Scenario:

When a node group is connected, the heartbeat packet is used to check whether the connected node is connected normally. The default heartbeat time is 60 s.

net_kernel:set_net_ticktime(600).

To reset the time value, how to test?

Each time I kill one of the nodes, the node connected to the node will immediately receive the nodedown message, and it is impossible to test whether the ticktime takes effect.

Cause:

In ERL Doc, there is a clip about the heartbeat check between nodes during node Interconnection:

Http://www.erlang.org/doc/man/kernel_app.html

Net_ticktime = ticktime

Specifies the net_kernel tick time. ticktime is given in seconds. once every ticktime/4 second, all connected nodes are ticked (if anything else has been written to a node) And if nothing has been inclued from another node within the last four (4) tick times that node is considered to be down. this ensures that nodes which are not responding, for reasons such as hardware errors, are considered to be down.

The time t, in which a node that is not responding is detected, is calculated as: Mint <t <maxt where:

         MinT = TickTime - TickTime / 4         MaxT = TickTime + TickTime / 4

Ticktime is by default 60 (seconds). Thus, 45 <t <75 seconds.

Note:All communicating nodes shoshould have the same ticktime value specified.

Note:Normally, a terminating node is detected immediately.

It can be seen from the document that the node checks every ticktime/4 seconds whether the connected node is normal. If the node does not receive the Heartbeat message four times in a row (within the ticktime, this node will be considered to have crashed,

However, if the node is terminated, other nodes will immediately receive the down notification.

To test the heartbeat time, you must understand the basic EPMD and net_kernel operating principles of the Erlang node connection. [Google is a panacea !].

 

The Erlang node uses TCP for communication. When a new node is enabled, it randomly selects a port (about 52300) and registers it in EPMD (Erlang Port Mapper daemo, by default, EPMD uses port 4369 for external connection,

 

Conclusion:

1. EPMD is the port er in Erlang. Each computer starts a process that records/exchanges the port information of each process in the Cluster:

2. When each node starts, an available port is registered with the local EPMD for receiving information;

3. When node A tries to establish two-way communication with Node B, it first queries the information of Node B from the local EPMD process (the first connection cannot be found );

4. If no message is found, the system queries the portb of the received message on Node B's EPMD and carries the Porta on node A to Node B;

5. node A random port porta1 ----> portb; Node B random port B1 -----> Porta;

6. The two-way communication connection is established successfully.

 

The following figure shows three NodesN1,N2, AndN3With their incoming connection TCP port 52383,522 36, 52275. Communication is ongoingN1AndN2,N1AndN2Where the nodes have picked random portsR1,R2,R3, AndR4.

Let's take a look at how the three nodes are connected?

Nodes N1, N2, N3, register 52383,522 36, 52275 respectively to establish the TCP connection for receiving. R1, R2, R3, and R4 are the TCP connections sent by randomly generated ports.

Erlang has many methods to detect whether a node is connectable. For example, the heartbeat net-ticks set automatically when the node is started (links is used in two directions, and monitors is used for one-way monitoring ),

1> net_kernel:monitor_nodes(true, [{node_type, visible}, nodedown_reason]).

Possible causes of failure:

%%connection_setup_failed%%no_network%%net_kernel_terminated%%shutdown%%connection_closed%%disconnect%%net_tick_timeout%%send_net_tick_fail%%edget_status_failed

How can we make these mistakes?

1. ineffective methods:

1.1. Change the node cookie.

As we all know, to connect two nodes together, you must know the cookie of the other node. After the connection is established, manually change the cookie:

1> erlang:set_cookie(node(), zhongwencool).

The results are very interesting: they have no impact on the connected nodes in the operation.

1.2. Block or kill the EPMD process.

The result is the same as that of 1, which has no effect on connected nodes. This is because EPMD is only used for connecting to the established node, and there is no EPMD after the established node.

2. destructive methods:

2.1 remove VM crash:

Directly in the operating system

$ kill -9 $PID

Or use:

1> os:cmd("kill -9 " ++ os:getpid()).

The result is that the peer node immediately receives the message.

{connection_closed}

2.2 normal kill or stop VM: If you are in Erlang shell, you can use c-c to end the VM. You can also call:

1> erlang:halt().

Or more commonly:

1> init:stop().

But the two results are: immediately receive:

{connection_closed}

3. Temporary methods:

3.1 you can use C-Z on the terminal, or:

1> os:cmd("kill -STOP " ++ os:getpid()).

Hold the VM halt.

After 60 s, we finally received:

{net_tick_timeout}

Finally, we saw a desired result, but it seems that this node cannot be used any more ......

3.2 kill the net_kernel process:

1> timer:kill_after(0, whereis(net_kernel)).

The result is received immediately:

{connection_closed}

4. Blocked Port:

Use firewall rules to block ports:

$ erl -name ‘[email protected]‘$ erl -name ‘[email protected]‘

The preceding two nodes use the network interface that is not used.

$ sudo iptables -I INPUT --destination 127.0.1.1 -j DROP$ sudo iptables -I INPUT --source 127.0.1.1 -j DROP$ sudo iptables -S-P INPUT ACCEPT-P FORWARD ACCEPT-P OUTPUT ACCEPT-A INPUT -s 127.0.1.1/32 -j DROP-A INPUT -d 127.0.1.1/32 -j DROP::$ sudo iptables -F

 

After 60 s, the following message is received:

｛net_tick_timeout｝

Final result:

If you want to test the nodedown message, it is best to use the firewall to set the port for simulation.

 

Refer:

1. http://blog.yufeng.info/archives/2779

2. http://www.cnblogs.com/me-sa/p/erlang-epmd.html

10 thousand horses in the heart are galloping .....................................