Installation and configuration of MapR distributed cluster and shell automation script

Installation and configuration of MapR distributed cluster and shell automation script

The distributed cluster Installation Process of MapR is still very difficult, far from being planned. I have summarized the installation configuration. Because there are many machines in the cluster, It is very tiring to manually configure each server. I have compiled an automatic configuration script. The following describes the script as the main line (the script is not complete, ).

Writing scripts requires shell basics, and it is better to master statistical methods such as awk and sed. The operating system is CentOS7.

Prepare: Install the CentOS operating system and partition it.

Partition:

/50 GB

/Home 50G

/Boot 500 M

/Var 50G

/Swap 8G

/Disk 305 GB (all remaining, used by the MapR file management system. Pay special attention to the need to remove the Mount later. After reboot, it will enter the emergency mode. You need to delete the disk mount in the configuration)

1. Configure IP

Configure the Ip address based on the company's network proxy. Pay attention to awk here, use = to divide each line, and output the variable = variable before = to the backup file; then replace the original Ip configuration file and restart the NIC. The ip address must be pinged before it can be used. Otherwise, there will be ip conflict.

function config_ip(){NETWORK_PATH=/etc/sysconfig/network-scripts/ENFILE=`ls $NETWORK_PATH/ifcfg-en*`IPADDR="10.4.45.211"NETMASK="255.255.255.0"GATEWAY="10.4.45.1"DNS1="210.83.210.155"TEMP_FILE="ifcfg.temp"cat $ENFILE | awk -F "=" '{  if($1=="BOOTPROTO"){    print $1"=static"  }  else if($1=="ONBOOT"){    print $1"=yes"  }  else{    print $1"="$2  }}' > $TEMP_FILEecho "IPADDR="$IPADDR >> $TEMP_FILEecho "NETMASK="$NETMASK >> $TEMP_FILE echo "GATEWAY="$GATEWAY >> $TEMP_FILEecho "DNS1="$DNS1 >> $TEMP_FILEmv $TEMP_FILE $ENFILEservice network restart}

2. Configure the host file

This cluster currently has four hosts: apm1, apm2, apm3, and apm4. Replace the ip address with the corresponding name to simplify inter-machine communication.

function config_hosts(){echo "10.4.45.210    apm1" >> /etc/hostsecho "10.4.45.211    apm2" >> /etc/hostsecho "10.4.45.212    apm3" >> /etc/hostsecho "10.4.45.213    apm4" >> /etc/hosts}

3. configure a network proxy

Add the proxy configuration to the/etc/profile file, but append it to the end of the file. Replace it with the top of the file. Configure the account and password based on your proxy.

function config_proxy(){echo "# add by user" >> /etc/profileecho "setterm -blength 0" >> /etc/profileecho "export http_proxy=http://sunyandong:Ulic2016@dl-proxy.neusoft.com:8080/" >> /etc/profileecho "export https_proxy=http://sunyandong:Ulic2016@dl-proxy.neusoft.com:8080/" >> /etc/profileecho "export ftp_proxy=http://sunyandong:Ulic2016@dl-proxy.neusoft.com:8080/" >> /etc/profilesource /etc/profile#curl "www.baidu.com"}

4. Configure local repo

Configuring repo as a domestic image can accelerate the download speed.

function config_repo(){WY_REPO=CentOS-163.com.repomv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backupcp /mnt/usb/$WY_REPO /etc/yum.repos.d/CentOS-Base.repoyum clean allyum makecache}

5. Download and install JDK

After configuring the proxy in the previous step, you can download and install JDK, vim, and createrepo in yum.

function config_software(){yum install vimyum install java-1.8.0-openjdk-devel.x86_64yum install createrepo}

6. Install MapR

Use the MapR repo to configure and install the corresponding MapR services, including fileserver, nfs, nodemanager, zookeeper, resourcemanager, and cldb.

function config_mapr(){MAPR_REPO=CentOS-mapr.repocp /mnt/usb/$MAPR_REPO /etc/yum.repos.d/maprtech.repoyum install mapr-fileserveryum install mapr-nfsyum install mapr-nodemanageryum install mapr-zookeeperyum install mapr-resourcemanager#yum install mapr-cldb}

Configure env. sh

function config_env(){bash /opt/mapr/conf/env.sh}

The above is the script content, which can complete most of the automated configuration and installation of MapR.

8. format the disk and configure the node.

Subsequent improvements .......

The script source code is as follows:

#!/bin/bash# config ip address function config_ip(){NETWORK_PATH=/etc/sysconfig/network-scripts/ENFILE=`ls $NETWORK_PATH/ifcfg-en*`IPADDR="10.4.45.211"NETMASK="255.255.255.0"GATEWAY="10.4.45.1"DNS1="210.83.210.155"TEMP_FILE="ifcfg.temp"cat $ENFILE | awk -F "=" '{  if($1=="BOOTPROTO"){    print $1"=static"  }  else if($1=="ONBOOT"){    print $1"=yes"  }  else{    print $1"="$2  }}' > $TEMP_FILEecho "IPADDR="$IPADDR >> $TEMP_FILEecho "NETMASK="$NETMASK >> $TEMP_FILE echo "GATEWAY="$GATEWAY >> $TEMP_FILEecho "DNS1="$DNS1 >> $TEMP_FILEmv $TEMP_FILE $ENFILEservice network restart}# config hostsfunction config_hosts(){echo "10.4.45.210    apm1" >> /etc/hostsecho "10.4.45.211    apm2" >> /etc/hostsecho "10.4.45.212    apm3" >> /etc/hostsecho "10.4.45.213    apm4" >> /etc/hosts}# config proxyfunction config_proxy(){echo "# add by user" >> /etc/profileecho "setterm -blength 0" >> /etc/profileecho "export http_proxy=http://sunyandong:Ulic2016@dl-proxy.neusoft.com:8080/" >> /etc/profileecho "export https_proxy=http://sunyandong:Ulic2016@dl-proxy.neusoft.com:8080/" >> /etc/profileecho "export ftp_proxy=http://sunyandong:Ulic2016@dl-proxy.neusoft.com:8080/" >> /etc/profilesource /etc/profile#curl "www.baidu.com"}function config_repo(){#cp -r /mnt/usb/mapr /home/# config repoWY_REPO=CentOS-163.com.repomv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backupcp /mnt/usb/$WY_REPO /etc/yum.repos.d/CentOS-Base.repoyum clean allyum makecache}# config softwarefunction config_software(){yum install vimyum install java-1.8.0-openjdk-devel.x86_64yum install createrepo}# mapr repofunction config_mapr(){#cp /mnt/usb/mapr/* /home/apm2/mapr#createrepo /mnt/usb/maprMAPR_REPO=CentOS-mapr.repocp /mnt/usb/$MAPR_REPO /etc/yum.repos.d/maprtech.repo#rpm --import http://package.mapr.com/releases/pub/maprgpg.keyyum install mapr-fileserveryum install mapr-nfsyum install mapr-nodemanageryum install mapr-zookeeperyum install mapr-resourcemanager#yum install mapr-cldb}function config_env(){bash /opt/mapr/conf/env.sh}# mainconfig_repoconfig_softwareconfig_mapr#config_ip#config_hosts#config_proxy

This article permanently updates the link address: