Linux Cluster setup__Linux

http://oboguev.net/kernel-etc/linux-cluster-setup.html

Helpful reading:

https://alteeve.ca/w/AN!Cluster_Tutorial_2
https://alteeve.ca/w/2-Node_Red_Hat_KVM_Cluster_Tutorial_-_Archive

RedHat 7 documentation

RHEL7 High Availability Add-On Administration

HighAvailability Add-On Reference Reference

GlobalFile System 2

LoadBalancer Administration

http://clusterlabs.org
http://clusterlabs.org/quickstart-redhat.html
http://clusterlabs.org/quickstart-ubuntu.html
http://clusterlabs.org/quickstart-suse.html
http://clusterlabs.org/doc
http://clusterlabs.org/faq.html

Clustersfrom Scratch
PacemakerExplained (Reference)
SUSEdocumentation

"ProLinux High Availability Clustering" (Kindle)

"CentOSHigh Availability" (Kindle)

http://corosync.org
https://alteeve.ca/w/Corosync
google: corosynctotem
google: OpenAIS


Older (CMAN-based) clusters included:

/etc/cluster/cluster.conf=> corosync.conf + cib.xml
system-config-cluster or conga (luci + ricci) configuration UI=> replaced by (still deficient) pcs-gui on port 2224
rgmanager => pacemaker
ccs => pcs




Setup Corosync/Pacemaker cluster named vc composed of three nodes ( vc1 , vc2 , vc3 )

Based on Fedora Server 22.

Warning: a bug in virt-manager Clone command may destroy AppArmorprofile both on source and target virtual machines.
Replicate virtual machines manually, or at least backup sourcemachine profile (located in /etc/apparmor.d/libvirt).


Networkset-up:

It is desirable to set up separate network cards for general internettraffic, SAN traffic and cluster backchannel traffic.
Ideally,interfaces should be link-aggregated (bonded or teamed) pairs, witheachlink in a pair connected to separate  stacked switches.
backchannel/cluster network can be two sub-nets (on separate interfaces) withcorosync redundant ring configured through them however bonded interface is easier to set up, moreresilient to failures, and allows traffic for other components befail-safe too it is also possible to bind multiple addresses to thebonded interface and set up corosync redundant ring amont them - but itdoes not make sense
SAN network can be two sub-nets (on separate interfaces), with iSCSImulti-pathing configured between them however can also be bonded: either utilizing one sub-netfor all SAN traffic (with disks dual-ported between iSCSI portalswithin the same sub-net, but different addresses), or binding muiltiplesub-nets to the bonded interface (with disks dual-ported between iSCSIportals located on different sub-nets)
general network better be bonded, so each node can be convenientlyaccessed by a single IP address however load balancer can instead be configured to usemultiple addresses for a node Bonded interfaces are slightly preferable to teamed interfaces forclustering, as all link management for bonded interfaces happens inthe kernel and does not involve user-land proccesses (unlike in the teamedinterfaces set-up).

It makes sense to use dual-port network cards and scattergeneral/SAN/cluster traffic ports between them, so a card failure doesnot bring down the whole network category.

If interfaces are bonded or teamed (rather than configured for separatesub-nets), switches should allow cross-traffic, i.e. be eitherstackable(preferably) or have ISL/IST (inter-switch link/trunking, akaSMLT/DSMLT/R-SMLT). 802.1aq (Shortest Path Bridging) support may bedesirable. See here.

Notethat IPMI (AMT/SOL) interface cannot be included in the bond or teamwithout loosing its IPMI capabillity, since it ceases to be indviduallyaddressable (having own P address).
Thus if IPMI is to be used for fencing or remote management, IPMI port is to be left alone.

For a real physical NIC, can identify port with

ethtool --identify ethX [10] => flashes LED 10 times

When hosting cluster nodes in KVM, create KVM macvtap interfaces (virtio/Bridge).

Bond interfaces:

About bonding
RHEL7 documentation
more about bonding
Notethat bonded/teamed interfaces in most setups do not provide increaseddata speed or increased bandwidth from one node to another. Theyprovide a failover and may provide an increased aggregate bandwidth for concurrent connections to multiple target hosts (but not to the same target host). However, see further down below.
Use network manager GUI:

"+" -> select Bond
Add->Create->Ethernet->select eth0
Add->Create->Ethernet->select eth1
Link Monitoring: MII => check media state
                ARP => use ARP to "ping" specified IP addresses(comma-separated),
                       at least one responds -> link ok (canalso configure to require all to respond)
Mode = 802.3ad => if linked to a real switch (802.3ad-compliant peer)
                  Adaptive load balancing => otherwise (if connected directly or via a hub, not a switch)
Monitoring frequency = 100 ms      

Or create files:

/etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
NAME=bond0
TYPE=Bond
ONBOOT=yes
BONDING_MASTER=yes
BOOTPROTO=none
#DEFROUTE=yes
#IPV4_FAILURE_FATAL=no
#UUID=9d1c6d47-2246-4c74-9c62-adf260d3fcfc
#BONDING_OPTS="miimon=100 updelay=0 downdelay=0 mode=balance-rr"
BONDING_OPTS="miimon=100 updelay=0 downdelay=0 mode=balance-alb"
IPADDR=223.100.0.10
PREFIX=24
#IPV6INIT=yes
#IPV6_AUTOCONF=yes
#IPV6_DEFROUTE=yes
#IPV6_FAILURE_FATAL=no
#IPV6_PEERDNS=yes
#IPV6_PEERROUTES=yes
#IPV6_PRIVACY=no

/etc/sysconfig/network-scripts/ifcfg-bond0_slave_1

HWADDR=52:54:00:9C:32:50
TYPE=Ethernet
NAME="bond0 slave 1"
#UUID=97b83c1b-de26-43f0-91e7-885ef758d0ec
ONBOOT=yes
MASTER=bond0
#MASTER=9d1c6d47-2246-4c74-9c62-adf260d3fcfc
SLAVE=yes

/etc/sysconfig/network-scripts/ifcfg-bond0_slave_2

HWADDR=52:54:00:CE:B6:91
TYPE=Ethernet
NAME="bond0 slave 2"
#UUID=2bf74af0-191a-4bf3-b9df-36b930e2cc2f
ONBOOT=yes
MASTER=bond0
#MASTER=9d1c6d47-2246-4c74-9c62-adf260d3fcfc
SLAVE=yes

nmcli device disconntctifname
nmcli connection reload [ifname]
nmcli connecton up ifname

route -n => must go to bond, not slaves

also make sure default route is present
if not, add to /etc/sysconfig/network:  GATEWAY=xx.xx.xx.xx

To team interfaces:

dnf install -y teamd NetworkManager-team

then configure team interface with NetworkManager GIU

Bonded/teamed interfaces in most setups do not provide increaseddata speed or increased bandwidth from one node to another. Theyprovide a failover and may provide an increased aggregate bandwidth for concurrent connections to multiple target hosts (but not to the same target host). However, there is a couple of workarounds:

Option 1:

Use bonding mode=4 (802.3ad)
lacp_rate=0
xmit_hash_policy=layer3+4

The latter hashes using src-(ip,port) and dst-(ip,port).
Still not good for a single connection. Option 2: Create separate VLAN for each port (on each of the nodes) and use bonding mode = Adaptive load balancing.

ThenLACP-compliant bridge will consider links separate and won't try tocorrelate the traffic and direct it via a single link according toxmit_hash_policy.
However this will reduce somewhat failover capacity: for example if Node1.LinkVLAN1 and Node2.LinkVLAN2 both fail.
Italso requires that all peer systems (such as iSCSI servers, iSNS, etc.)have their interfaces configured accordingly to the sameVLAN scheme.
Remember to enable jumbo frames: ifconfig ethX mtu 9000.


Prepare:

Names vc1, vc2 and vc3 below are forcluster backchannel.

On each node:

# set node name
hostnamectlset-hostname vcx

# disable "captive portal"detection in Fedora
dnfinstall -y crudini
crudini --set/etc/NetworkManager/conf.d/21-connectivity-local.conf connectivityinterval 0
systemctl restartNetworkManager


Clustershells

Install

dnf install-y pdsh clustershell

To use pdsh:

#non-interactive:
pdsh -R exec -f 1 -w vc1,vc2,vc3 cmd | dshbak
pdsh -R exec -f 1 -w vc[1-3]  cmd | dshbak

#interactive:
pdsh -R exec -f 1 -w vc1,vc2,vc3
pdsh -R exec -f 1 -w vc[1-3]

cmd substitution:

%h  => remote host name
%u  => remote user name
%n  => 0, 1, 2, 3 ...
%%  => %
To set up for clush, first enable password-less ssh.
Clumsy way:

ssh vc1
ssh-keygen -t rsa

ssh vc1 mkdir -p .ssh
ssh vc2 mkdir -p .ssh
ssh vc3 mkdir -p .ssh

ssh vc1 chmod 700 .ssh
ssh vc2 chmod 700 .ssh
ssh vc3 chmod 700 .ssh

cat .ssh/id_rsa.pub | ssh vc1 'cat>> .ssh/authorized_keys'
cat .ssh/id_rsa.pub | ssh vc2 'cat >>.ssh/authorized_keys'
cat .ssh/id_rsa.pub | ssh vc3 'cat >>.ssh/authorized_keys'
Ctrl-D

ssh vc2
ssh-keygen -t rsa
cat .ssh/id_rsa.pub | ssh vc1 'cat >>.ssh/authorized_keys'
cat .ssh/id_rsa.pub | ssh vc2 'cat >>.ssh/authorized_keys'
cat .ssh/id_rsa.pub | ssh vc3 'cat >>.ssh/authorized_keys'
Ctrl-D

ssh vc3
ssh-keygen -t rsa
cat .ssh/id_rsa.pub | ssh vc1 'cat >>.ssh/authorized_keys'
cat .ssh/id_rsa.pub | ssh vc2 'cat >>.ssh/authorized_keys'
cat .ssh/id_rsa.pub | ssh vc3 'cat >>.ssh/authorized_keys'
Ctrl-D
Cleaner way:

Create id_rsa.pub, id_rsaand authorized_keys on one node,
then replicate them to other nodes in the cluster.
To use clush:

clush -w vc1,vc2,vc3 -b  [cmd]
clush -w vc[1-3] -b  [cmd]
Basiccluster install:

On each node:

dnf install -y pcsfence-agents-all fence-agents-virsh resource-agents pacemaker

Optional: dnfinstall -y dlm lvm2-cluster gfs2-utils iscsi-initiator-utils lsscsihttpd wget

systemctl startfirewalld.service
firewall-cmd--permanent --add-service=high-availability
firewall-cmd--add-service=high-availability
systemctl stopfirewalld.service
iptables --flush

## optionally disable SELinux:
#setenforce 0
#edit /etc/selinux/config and change SELINUX=enforcing=> SELINUX=permissive

passwd hacluster

systemctl startpcsd.service
systemctl enablepcsd.service

# make sure no http_proxyexported
pcscluster auth vc1.example.com vc2.example.com vc3.example.com -uhacluster -pxxxxx--force
e.g. pcs cluster auth vc1 vc2 vc3-u hacluster -p abc123 --force

# created auth data isstored in /var/lib/pcsd

On one node:

pcs cluster setup [--force] --name vcvc1.example.com vc2.example.com vc3.example.com

pcs cluster start --all

to stop:   pcscluster stop --all

On each node:

# to auto-start cluster onreboot
# alternatively can manually do "pcs cluster start" on each reboot
pcscluster enable --all

to disable: pcs cluster disable --all

View status:

pcs status
pcs cluster status
pcs clusterpcsd-status
systemctl statuscorosync.service
journalctl -xe
cibadmin--query
pcsproperty list [--all] [--defaults]
corosync-quorumtool-oi [-i]
corosync-cpgtool
corosync-cmapctl  [  |  grepmembers]
corosync-cfgtool -s
pcs cluster cib

Verify current configuration

crm_verify --live --verbose

Start/stop node

pcs cluster stop vc2
pcs status
pcs cluster start vc2

Disable/enable hosting resources on the node (standby state)

pcs cluster standby vc2
pcs status
pcs cluster unstandby vc2

"Transactional"configuration:

pcs clustercib my.xml                   # get a copy ofCIB to my.xml
pcs -f my.xml  ... change command ...     #make changes of config in my.xml
crm_verify --verbose --xml-file=q.xml     # verifyconfig
pcs cluster cib-push my.xml              # push config from my.xml to CIB

ConfigureSTONITH

All agents: https://github.com/ClusterLabs/fence-agents/tree/master/fence/agents
fence_virsh -fences machine via ssh to vm host and execuiting sudo virsh destroy<vmid> or sudo virsh reboot<vmid>
Alternative to virsh:fence_virt/fence_xvm

dnf install -y fence-virt
STONITH is needed:
In resource (non-quorum) basedclusters, for obvious reasons

In two-node clusters withoutquorum disk (a special case of the above), for obvious reasons

Inquorum-based clusters, because Linux clustering solutions includingCorosync and CMAN run as user-level processes and are unable tointerdict user-level and kernel-level activity on the node when clusternode losesconnection to majority-votes partition. By comparison, in VMSCNXMAN is a kernel component which makes all CPUs to spin inIOPOST by requeueing the req