MAPR distributed cluster installation configuration and shell automation scripts

MAPR distributed cluster installation process is still very difficult, far from the simple Plan. I summarize the installation configuration, because the cluster has a lot of machines, manual each configuration is very tired, wrote an automated configuration script, the following script as the main line narrative (the script is not perfect, follow-up continues to improve).

Scripting requires shell-based, and it's better to master the statistical tools of awk and sed. the operating system is CentOS7.

　　Preparation: Install the CentOS operating system and partition

Partition:

/50G

/Home 50G

/boot 500M

/var 50G

/swap 8G

/disk 305G (all remaining, used for MAPR file management system. Special attention , need to remove the mount after reboot, will enter the emergency mode, need to delete the disk in the configuration of the Mount)

　　One configuration IP

Configure the IP according to the company network Proxy, note here that awk is used to divide each line by =, the variables that are modified by = before the variable =, output to the backup file, and then replace the original IP profile, and finally restart the NIC. IP before setting must ping, not generalists can use, otherwise there will be IP conflicts.

function Config_ip () {Network_path=/etc/sysconfig/network-scripts/Enfile= ' ls $NETWORK _path/ifcfg-en*' IPAddr="10.4.45.211"NETMASK="255.255.255.0"GATEWAY="10.4.45.1"DNS1="210.83.210.155"Temp_file="ifcfg.temp"Cat $ENFILE| Awk-f"=" '{  if($1=="Bootproto") {print $1"=static"  }  Else if($1=="Onboot") {print $1"=yes"  }  Else{print $1"="$2  }}'> $TEMP _fileEcho"ipaddr="$IPADDR >>$TEMP _fileecho"netmask="$NETMASK >>$TEMP _file Echo"gateway="$GATEWAY >>$TEMP _fileecho"dns1="$DNS 1 >>$TEMP _filemv $TEMP _file $ENFILEservice Network Restart}

　　Two configuration host files

This cluster temporarily has four hosts, respectively APM1,APM2,APM3 and APM4. It simplifies communication between machines by replacing IP with the corresponding name.

" 10.4.45.210    apm1" >>/etc/"10.4.45.211    apm2" >>/etc/"10.4.45.212    apm3" >>/etc/"  10.4.45.213    apm4" >>/etc/hosts}

　　Three Configuration network proxy

The proxy configuration is added to the/etc/profile file, but this is appended to the end of the file and needs to be replaced with the top of the file. Configure the account and password according to your agent.

function Config_proxy () {echo"# Add by User">>/etc/Profileecho"Setterm-blength 0">>/etc/Profileecho"Export Http_proxy=http://sunyandong:[email protected]:8080/">>/etc/Profileecho"Export Https_proxy=http://sunyandong:[email protected]:8080/">>/etc/Profileecho"Export Ftp_proxy=http://sunyandong:[email protected]:8080/">>/etc/Profilesource/etc/Profile#curl"www.baidu.com"}

　　Quad Config local repo

Repo is configured as a domestic mirror, which speeds up the download speed.

function Config_repo () {wy_repo=centos-163/etc/yum.repos.d/centos-base.repo/ etc/yum.repos.d/centos-/mnt/usb/$WY _repo/etc/yum.repos.d/centos-base.repoyum clean Allyum Makecache} 

　　Five Downloads Install JDK

After configuring the agent in the previous step, yum can download and install Jdk,vim, and Createrepobelow.

function Config_software () {Yum install Vimyum install Java-1.8. 0-openjdk-devel.x86_64yum install Createrepo}

　　Six-Mount MapR

With MAPR repo, configure the installation MAPR the appropriate services, including Fileserver,nfs,nodemanager,zookeeper,resourcemanager and CLDB

function Config_mapr () {mapr_repo=centos-mapr.repocp/mnt/usb/$MAPR _repo/etc/yum.repos.d/ Maprtech.repoYum install mapr-fileserveryum install Mapr-nfsyum install Mapr-nodemanageryum Install Mapr-zookeeperyum install Mapr-resourcemanager#yum install MAPR-CLDB}

　　Seven Configuration env.sh

/opt/mapr/conf/env.sh}

The above is the content of the script, you can complete the most automated configuration installation of MAPR.

　　Eight format disk, configure nodes

Follow-up improvement .....

Script source for the following:

#!/bin/bash# Config IP address function config_ip () {Network_path=/etc/sysconfig/network-scripts/Enfile= ' ls $NETWORK _path/ifcfg-en*' IPAddr="10.4.45.211"NETMASK="255.255.255.0"GATEWAY="10.4.45.1"DNS1="210.83.210.155"Temp_file="ifcfg.temp"Cat $ENFILE| Awk-f"=" '{  if($1=="Bootproto") {print $1"=static"  }  Else if($1=="Onboot") {print $1"=yes"  }  Else{print $1"="$2  }}'> $TEMP _fileEcho"ipaddr="$IPADDR >>$TEMP _fileecho"netmask="$NETMASK >>$TEMP _file Echo"gateway="$GATEWAY >>$TEMP _fileecho"dns1="$DNS 1 >>$TEMP _filemv $TEMP _file $ENFILEservice network restart}# config hostsfunction config_hosts () {echo"10.4.45.210 apm1">>/etc/Hostsecho"10.4.45.211 apm2">>/etc/Hostsecho"10.4.45.212 apm3">>/etc/Hostsecho"10.4.45.213 apm4">>/etc/hosts}# config proxyfunction config_proxy () {echo"# Add by User">>/etc/Profileecho"Setterm-blength 0">>/etc/Profileecho"Export Http_proxy=http://sunyandong:[email protected]:8080/">>/etc/Profileecho"Export Https_proxy=http://sunyandong:[email protected]:8080/">>/etc/Profileecho"Export Ftp_proxy=http://sunyandong:[email protected]:8080/">>/etc/Profilesource/etc/Profile#curl"www.baidu.com"}function Config_repo () {#cp-r/mnt/usb/mapr/home/# config Repowy_repo=centos-163. COM.REPOMV/etc/yum.repos.d/centos-base.repo/etc/yum.repos.d/centos-BASE.REPO.BACKUPCP/mnt/usb/$WY _repo/etc/yum.repos.d/centos-base.repoyum clean Allyum makecache}# config softwarefunction config_software () {Yum install Vimyum install Java-1.8.0-openjdk-Devel.x86_64yum Install createrepo}# mapr repofunction config_mapr () {#cp/mnt/usb/mapr/*/home/apm2/mapr#createrepo/mnt/usb/maprmapr_repo=centos-mapr.repocp/mnt/usb/$MAPR _repo/etc/yum.repos.d/ maprtech.repo#rpm--importHttp://package.mapr.com/releases/pub/maprgpg.keyYum Install Mapr-fileserveryum install Mapr-nfsyum install Mapr-nodemanageryum install Mapr-zookeeperyum Install Mapr-resourcemanager#yum install Mapr-cldb}function config_env () {bash/opt/mapr/conf/env.sh}# mainconfig_repoconfig _softwareconfig_mapr#config_ip#config_hosts#config_proxy

MAPR distributed cluster installation configuration and shell automation scripts