CDH Cluster frequent alarm (host frequent swapping)

Recently CDH cluster frequent alarm, because some host frequent swapping, greatly affected the performance of the cluster.

Later found a setting (/proc/sys/vm/swappiness) needs to be modified, the default value of 60

Setting the vm.swappiness Linux Kernel Parameter

vm.swappinessis a Linux Kernel Parameter This controls how aggressively memory pages are swapped to disk. It can set to a value between 0-100; The higher the value, the more aggressive the kernel are in seeking out inactive memory pages and swapping them to disk.

You can see if value is currently set-to- vm.swappiness looking at /proc/sys/vm ; Example:

Cat/proc/sys/vm/swappiness

On the most systems, it's set to $ by default. This isn't suitable for Hadoop clusters nodes, because it can cause processes to get swapped off even when there was free Memory available. This can affect stability and performance, and could cause problems such as lengthy garbage collection pauses for important System daemons. Cloudera recommends that's set this parameter to 0; For example:

CDH Cluster frequent alarm (host frequent swapping)