Hadoop CDH5.0.0 upgrade to CDH5.3.0 steps

1. Stop Monit on all Hadoop servers (we use Monit on line to monitor processes)

Login Idc2-admin1 (we use idc2-admin1 as a management machine and Yum Repo server on line)
# mkdir/root/cdh530_upgrade_from_500
# cd/root/cdh530_upgrade_from_500
# pssh-i-H idc2-hnn-rm-hive ' Service Monit stop '
# pssh-i-H idc2-hmr.active ' Service Monit stop '

2. Confirm that the local CDH5.3.0 yum repo server is ready

http://idc2-admin1/repo/cdh/5.3.0/
http://idc2-admin1/repo/cloudera-gplextras5.3.0/

3. Update the corresponding repo template in ansible (we use Ansible as configuration Management tool on line)

{% if ' idc2 ' in group_names%}

...

{% if ' cdh5-all ' in group_names%}
[heylinux.el6.cloudera-cdh5.3.0]
Name= el6 Yum Cloudera cdh5.3.0
baseurl=http://idc2-admin1/repo/cdh/5.3.0
Enabled=1
Gpgcheck=0

[heylinux.el6.cloudera-gplextras5.3.0]
Name= el6 Yum Cloudera gplextras5.3.0
baseurl=http://idc2-admin1/repo/cloudera-gplextras5.3.0
Enabled=1
Gpgcheck=0
{% Else%}

...

{% ENDIF%}

4. Update repo files for all Hadoop servers (/etc/yum.repos.d/heylinux.repo)

# Ansible-playbook--private-key/path/to/key_root-u Root--vault-password-file=/path/to/vault_passwd.file base.yml- I hosts.idc2--tags localrepos--limit cdh5-all

5. Upgrade HDFs related content

5.1. Get the current Activie Namenode (a CNAME that always checks and points to active Namenode is created in our DNS server on the line)
# host ACTIVE-IDC2-HNN
Active-idc2-hnn.heylinux.com is a alias for idc2-hnn2.heylinux.com
Idc2-hnn2.heylinux.com has address 172.16.2.12

5.2. Enter Safe mode on active Namenode and generate a new fsimage and wait for the whole process to end.

# sudo-u HDFs HDFs Dfsadmin-safemode Enter
# sudo-u HDFs HDFs dfsadmin-savenamespace

5.3 Turn off all the Hadoop services

Back to the working directory on Idc2-admin1
# cd/root/cdh530_upgrade_from_500

First by Pssh the Namenode,resourcemanager and the Hadoop-related processes on the Hive server (writes the corresponding server address or host name list to idc2-hnn-rm-hive and idc2-hmr.active)
# pssh-i-H idc2-hnn-rm-hive ' for x in ' Cd/etc/init.d; LS hadoop-* '; Do sudo service $x status; Done
# pssh-i-H idc2-hmr.active ' for x in ' Cd/etc/init.d; LS hadoop-* '; Do sudo service $x status; Done

# pssh-i-H idc2-hnn-rm-hive ' for x in ' Cd/etc/init.d; LS hadoop-* '; Do sudo service $x stop; Done
# pssh-i-H idc2-hmr.active ' for x in ' Cd/etc/init.d; LS hadoop-* '; Do sudo service $x stop; Done

# Check if there is a libhadoop.so file that conflicts with the new version, delete if present (we have snappy installed on the line, it will generate a file that conflicts with CDH5.3.0 's libhadoop.so and place it under the current JDK Lib directory).
# pssh-i-H idc2-hnn-rm-hive ' rm-f/usr/java/jdk1.7.0_45/jre/lib/amd64/libhadoop.so '
# pssh-i-H idc2-hmr.active ' rm-f/usr/java/jdk1.7.0_45/jre/lib/amd64/libhadoop.so '
Backup the HDFS metadata on the Namenodes

Back up the metadata files on the namenodes (we have two namenode on the line, Idc2-hnn1 and IDC2-HNN2 respectively:
# Mkdir/root/cdh530upgrade
# Cd/root/cdh530upgrade
# tar-cf/root/nn_backup_data.data1. ' Date +%y%m%d '. Tar/data1/dfs/nn
# tar-cf/root/nn_backup_data.data2. ' Date +%y%m%d '. Tar/data2/dfs/nn

6. Upgrade Hadoop Related Packages

Log in and upgrade the hive server idc2-hive1
# yum Clean all; Yum Upgrade Hadoop

Log in and upgrade ResourceManager server Idc2-rm1 and Idc2-rm2
# yum Clean all; Yum Upgrade Hadoop

Go back to Idc2-admin1 and upgrade all the Datanode servers idc2-hmr*
# pssh-i-H idc2-hmr.active ' yum clean all; Yum upgrade Hadoop hadoop-lzo-y '

Log in and upgrade Idc2-hnn1 (Standby Namenode, as judged by the previous host ACTIVE-IDC2-HNN command)
# yum Clean all; Yum Upgrade Hadoop Hadoop-lzo

Log in and upgrade Idc2-hnn2 (Active Namenode, as judged by the previous host ACTIVE-IDC2-HNN command)
# yum Clean all; Yum Upgrade Hadoop Hadoop-lzo

Go back to Idc2-admin1 and upgrade all the Hadoop Clients
# pssh-i-H idc2-client ' yum clean all; Yum Upgrade Hadoop-y '

7. Start related services

Login and start Journal nodes Service (we are on line for IDC2-HNN1, IDC2-HNN2, idc2-rm1 three servers)
# service Hadoop-hdfs-journalnode Start

Log on to active Namenode and update HDFs Metadata
# Service Hadoop-hdfs-namenode Upgrade
# tailf/var/log/hadoop/hadoop-hdfs-namenode-' hostname-s '. Heylinux.com.log

Wait until the end of the process, for example, in the log, like the following:
/var/lib/hadoop-hdfs/cache/hadoop/dfs/<name> is complete.

Log in to all Datanode and start the service (we are on line for idc2-hmr* server)
# service Hadoop-hdfs-datanode Start

Wait until Namenode quits safe Mode and then restarts standby Namenode

Login standby Namenode and restart service
# sudo-u HDFs HDFs Namenode-bootstrapstandby
# service Hadoop-hdfs-namenode Start

Log in to all ResourceManager and start the service
# service Hadoop-yarn-resourcemanager Start

Log in to all NodeManager and start the service (we are on line for idc2-hmr* server)
# service Hadoop-yarn-nodemanager Start

Start Historyserver on active ResourceManager (we are on line for Idc2-rm1 server)
# service Hadoop-mapreduce-historyserver Start

At this point, the entire Hadoop-related upgrade is over, and below, the Hive,oozie and pig upgrades will be introduced accordingly.

8. Upgrade hive and Oozie Server (we have unified installation to a single machine idc2-hive1 on line)
8.1 Upgrade Hive Server

Backing Up the Metastore database

# mkdir-p/root/backupfiles/hive
# cd/root/backupfiles/hive
# Mysqldump-uoozie-pheylinux Metastore > Metastore.sql.bak. ' Date +%y%m%d '

Update Hive-site.xml

Confirm the following settings are present in Hive-site.xml
<property>
<name>datanucleus.autoCreateSchema</name>
<value>false</value>
</property>
<property>
<name>datanucleus.fixedDatastore</name>
<value>true</value>
</property>
Stop Hive related services
# Service Hive-server2 Stop
# Service Hive-metastore Stop

Upgrade Hive Related Packages
# Yum Upgrade Hive Hive-metastore Hive-server2 HIVE-JDBC
# yum Install hive-hbase Hive-hcatalog hive-webhcat

Upgrade Hive's Metastore
# sudo-u Oozie/usr/lib/hive/bin/schematool-dbtype Mysql-upgradeschemafrom 0.12.0

Start the Hive service
# service Hive-metastore Start
# service Hive-server2 Start

8.2 Upgrade Oozie Server
Backing Up the Oozie database
# mkdir-p/root/backupfiles/hive
# cd/root/backupfiles/hive
# Mysqldump-uoozie-pheylinux Oozie > Oozie.sql.bak. ' Date +%y%m%d '

Backing up Oozie configuration files
# tar CF oozie.conf.bak. ' Date +%y%m%d '/etc/oozie/conf

Stop Oozie
# Service Oozie Stop

Upgrade Oozie Packages
# Yum Upgrade Oozie oozie-client

Carefully proofread the parameters in the new configuration file with the original configuration file and update the parameters in the original configuration file to the new profile

Backup Oozie lib Directory
# tar CF oozie.lib.bak. ' Date +%y%m%d '/var/lib/oozie

Upgrading the Oozie Database
# sudo-u oozie/usr/lib/oozie/bin/ooziedb.sh Upgrade-run

Upgrade Oozie Shared Library
# sudo-u Oozie Hadoop fs-mv/user/oozie/share/user/oozie/share.orig. ' Date +%y%m%d '
# sudo oozie-setup sharelib create-fs hdfs://idc1-hnn2:8020-locallib/usr/lib/oozie/oozie-sharelib-yarn.tar.gz

Move all the library from the directory/user/oozie/share/lib/lib_<new_date_string> to the/user/oozie/share/lib (<new_date_string > The timestamp after the catalog is generated
# sudo-u Oozie mv/user/oozie/share/lib/lib_<new_date_string>/*/user/oozie/share/lib/

Check all files in the/user/oozie/share directory in the HDFs and compare them to the files in the Share.orig. ' Date +%y%m%d ' after the backup, except that the package with the "CDH5" version only retains the update, The rest is copied to the new Lib directory.

Start Oozie Server
# service Oozie Start

9. Upgrade Pig

Kill all the running pig processes
# Pkill-kill-f Pig

Update Pig Packages
# Yum Upgrade Pig

10. Execute the finalizeupgrade command for the final finish when all packages are upgraded and HDFS is working properly
Log on to active Namenode and execute the following command
# sudo-u HDFs HDFs Dfsadmin-finalizeupgrade