JGroups Setting and Tuning

UDP configuration:

UDP protocol stack would bundle message into large ones and send them together to reduce network overhead. Relevant configurations is:

• Bundler_type:old, Defaultbundler use a timerscheduler to send the message. New->transferqueuebundler use a internal linkedblockingqueue to hold the message ADD message to queue unless total Message size reaches Max_bundle_size or queue is 90% full (bundler_capacity)

• Max_bundle_timeout useful only for the Defaultbundler, this is the scheduling delay

• Mcast_send_buf_size, Mcast_recv_buf_size, ucast_send_buf_size, ucast_recv_buf_size unicast or multicast socket sender/ Receiver buffer size.

• Timer_type, Timer_min_threads, Timer_max_threads, timer.keep_alive_time, timer.queue_max_size timer thread pool is used To perform scheduled task, like bundling

• Thread_pool.min_threads, Thread_pool.max_threads, Thread_pool.keep_alive_time, thread_pool.queue_enabled, Thread_ Pool.queue_max_size, Thread_pool.rejection_policy thread pool is used to handle received batch of message and execute B Atchhandler

• enable_batching:directly pass a batch of messages up, instead of processing them one by one, default to True

Other protocol stacks:

• Fd:failure detection

• Pbcast. NAKACK2, unicast:message reliability, use a internal sliding window to ensure that messages is delivered in order

• Stable:message stability, ensures that message is seen to every member in the cluster, periodically broadcast latest STA BLE message

Monitoring:

JAVA-CP Jgroups-xxx.jar org.jgroups.tests.Probe jmx=[protocol name]

You can view num_msgs_sent, num_msgs_received, Num_bytes_sent, num_bytes_received, Num_rejected_msgs, Num_oob_msgs_ Received etc

JGroups optimization for invalidation usage:

• The JGroups config shipped by default was simply taken directly from the JGroups documentation. It does not take into account our requirements, or improvements in JGroups and provides functionality we do not need.

  
  

• Flow control:we use MFC and UFC and by default allow them 4M credits. This seems to adversely affect the performance of import and catalog sync, as the system soon blocks waiting for more cred Its. Disabling Flow Control floods the network and quickly causes problems. Testing with 200M found that CPU utilisation is able to climb to 100%, but then timeouts on cluster sync and heartbeat me Ssages caused the importing node to being evicted from the cluster, also causing problems. Testing with *40m* produced much increased performance over 4M and no errors reported. Recommendation:
  
  <UFC max_credits= "40M" min_threshold= "0.4"/><mfc max_credits= "40M" min_threshold= "0.4"/>

  
  

• BARRIER and Pbcast. State_transfer:not Useful for invalidation usage (order isn't important) so they can removed

  .

• Merge:we use MERGE2, more recent MERGE3 should is used as it uses a more efficient algorithm.

  
  

• Frag2:we set 60K by default, but this can is optimised by using the network ' s max frame size.

  
  

• Udp:
  Set your send and receive buffer sizes to match your operating system settings.
  Set your timer type to "NEW3"
  Ensure thread_pool.enabled= "true" and thread_pool.queue_enabled= "true" are set. Make your queue massive, 1000000 are fine, or more. Increase number of threads to 2xCores on box.
  Bundling. This was very important and max_bundle_timeout= "5" or less should be set, as-seconds is a very long time between invalid ation messages. Max_bundle_size should be slightly bigger than your FRAG2 frag_size setting.
  
  

To simulate 2 network interfaces in my CentOS virtual server I have defined a alias for my eth0 interface. In a real customer environment skip this step assuming you already has the dedicated network interface for cluster MESSAG Es configured.

sudo vi/etc/sysconfig/network-scripts/ifcfg-eth0:0

#I have pasted the lines below. Make sure-these settings according to your local network settings:

device=eth0:0
network=10.5.0.0
netmask=255.255.255.0
ipaddr=10.5.0.246

#end of ifcfg-etg0:0 settings

sudo ifup eth0:0

Ifconfig #verify you had an eth0:0 interface


JGROUPS UDP Setting:
Mcast_addr= "230.0.0.1"
Bind_addr= "10.5.0.246"

Sometimes multicast traffic choses to use IPV6 network interfaces even if we specifically bind by an IPV4 address. To make sure the JVM uses IPv4 add the following settings
-djava.net.preferipv4stack=true-djava.net.preferipv4addresses=true


Verification:

Netstat-ng
Ipv6/ipv4 Group Memberships
Interface refcnt Group
--------------- ------ ---------------------
Lo 1 224.0.75.75
Lo 1 224.0.0.1
Eth0 1 224.0.75.75
Eth0 1 230.0.0.1
Eth0 1 224.0.0.1
Lo 1 ff02::1
Eth0 1 ff02::202
Eth0 1 ff02::1:ffa4:7c61
Eth0 1 Ff02::1
Eth1 1 Ff02::1



NETSTAT-ANP | grep 230.0.0.1
(Not all processes could is identified, non-owned process info is not being shown, you would has to be root to see it all.) )
UDP 0 0 230.0.0.1:9997 0.0.0.0:* 3761/java


Server Console log:
Gms:address=node-1, Cluster=broadcast, physical address=10.5.0.246:38815
Or
Netstat-an | grep udp| grep 10.5.0.246
UDP 0 0 10.5.0.246:38815 0.0.0.0:*



sudo tcpdump-ni eth0:0 UDP port 9997

Tcpdump:verbose output suppressed, use-v OR-VV for full protocol decode
Listening on eth0:0, Link-type EN10MB (Ethernet), capture size 65535 bytes
16:05:11.457605 IP 10.5.0.246.palace-6 > 230.0.0.1.palace-6: UDP, length 78
16:05:11.963756 IP 10.5.0.246.palace-6 > 230.0.0.1.palace-6: UDP, length 78
16:05:13.184506 IP 10.5.0.246.palace-6 > 230.0.0.1.palace-6: UDP, Length 91



In case of any issues, would also want to make sure that the jgroups settings:receive_on_all_interfaces and Send_on_a  Ll_intefaces is set to false. By default they should is, at least the receive_on_all_interfaces based on JGroups documentation.
Other jgroups settings This could be explored is receive_interfaces and send_interfaces. Not sure if-a large environment, for performance reasons one would want to further separate the receive and send Traff IC on their own interfaces.
For more strict control monitoring or troubleshooting:bind_port and Port_range could also is explored.

Besides the multicast traffic jgroups also uses unicast sockets for node to node communication. The physical address of the node is the unicast socket. If a network filter blocks the communication on these sockets, the cluster cannot form and your ' d get error in the log Simi Lar to these.
[OOB-434,BROADCAST,NODE-17] [UDP] Node-17:no Physical address for 8a0e9532-54df-d91e-79af-185b2cadaf1f, dropping message. By default the unicast socket uses a random port number, which is fine in the most environments and for environments where IPTA Bles or other local firewalls is enabled, you'll need to set the Bind_port and Port_range, otherwise the cluster would Not being able to form. Define your port range using these 2 values and add rules to your iptables to allow communication on these ports.


The most of the servers out of the box sysctl settings don ' t seem to be is optimal for Jgroups. Check for warns in the console, log right after Jgroups started. Ask your admin to adjust these settings accordingly.
WARN [main] [UDP] [JGRP00014] The send buffer of socket datagramsocket is set to 640KB, and the OS only allocated 229.38 Kb. This might leads to performance problems. Please set your max send buffer in the OS correctly (e.g. Net.core.wmem_max on Linux)
WARN [main] [UDP] [JGRP00014] The receive buffer of socket Datagramsocket is set to 20MB, but the OS is only allocated 229. 38KB. This might leads to performance problems. Please set your max receive buffer in the OS correctly (e.g. Net.core.rmem_max on Linux)
[Java] WARN [main] [UDP] [JGRP00014] The send buffer of socket MulticastSocket is set to 640KB, and the OS only allocated 229.3 8KB. This might leads to performance problems. Please set your max send buffer in the OS correctly (e.g. Net.core.wmem_max on Linux)
[Java] WARN [main] [UDP] [JGRP00014] The receive buffer of socket MulticastSocket is set to 25MB, and the OS only allocated 229 .38KB. This might leads to performance problems. Please set your max receive buffer in the OS correctly (e.g. Net.core.rmem_max on Linux)



JGroups with large cluster:
Newer version of JGroups 2.2.9 and higher could leverage TCP NIO for TCP clusters:http://www.jgroups.org/perftest.html. While in small cluster there does not seem to be a difference, actually the opposite, NIO seems to being slower, it is expect Ed to yield better performance in larger clusters.
http://sourceforge.net/p/javagroups/mailman/message/31439524/

This article from "Shadowisper" blog, declined reprint!

JGroups Setting and Tuning