Hadoop2.7.3 compilation supports 5 types of compression

Guide:
After you follow the steps in the blog, you can support 5 types of local compression after Hadoop compiles: zlib, snappy, lz4, bzip2, OpenSSL

The author's environment is configured as: JDK1.8, hadoop2.7.3,hive2.3.0, you can also use other versions, note that version compatibility can be 1. Installing dependent tools 1. Install related tools

As the author is the smallest installation, so many tools are not installed, specifically need to install which tools, also not very clear, so installed a lot. It is recommended that you use the root user installation.

# yum-y  Install make gcc gcc-c++ gcc-g77 Flex Bison file Libtool libtool-libs autoconf kernel-devel libjpeg libjpeg-d  Evel libpng libpng-devel libpng10 libpng10-devel gd gd-devel freetype freetype-devel libxml2 libxml2-devel zlib zlib-devel Glib2 glib2-devel bzip2 bzip2-devel libevent libevent-devel ncurses ncurses-devel Curl curl-devel e2fsprogs E2fsprogs-dev El krb5 krb5-devel libidn libidn-devel OpenSSL openssl-devel gettext gettext-devel ncurses-devel gmp-devel pspell-devel un Zip libcap lsof build-essential cmake 1g-dev pkg-config libssl-dev lzo-devel fuse fuse-devel zlib1g-dev libprotobuf-dev PR Otobuf-compiler Snappy Libbz2-dev Libjansson-dev Libfuse-dev

If you are still missing after installing these tools, you can use the following command to detect the installation

# yum-y Groupinstall "Development Tools"

2. Installing Protobuf

Unzip the build installation

# TAR-ZXVF protobuf-2.5.0.tar.gz
# cd/home/hadoop/protobuf-2.5.0
#/configure--prefix=/home/hadoop/ protobuf/  
# make && make install

Environment variables

# vim ~/.BASHRC
# source ~/.BASHRC

Export Path=/home/hadoop/protobuf/bin: $PATH

If the following error is reported when compiling the installation
Libtool:install:error:cannot install ' libaprutil-1.la ' to a directory
Cause: It may have been previously installed./configure, causing the installation file to be less "clean"
Workaround:
(1). Execute command Make clean
(2). Remove the directory that you just compiled the installation protobuf, re-unzip the installation to compile 3. Install snappy

Unzip the build installation

# TAR-ZXVF snappy-1.1.1.tar.gz
# cd snappy-1.1.1
#/configure
# make && make install

Check to see if the snappy installation is complete

ll/usr/local/lib/| grep Snappy

2.hadoop Compilation 1. Configure Maven

Unzip, rename, configure environment variables

# tar-zvxf apache-maven-3.3.9-bin.tar.gz
# mv apache-maven-3.3.9-bin.tar.gz Maven3
# vim ~/.BASHRC

Export Maven_home=/home/hadoop/maven3
export m2_home= $MAVEN _home

Reload the file to make the environment variable effective immediately

# source ~/.BASHRC

Create the. m2 file, copy the setting to the directory

# cd/home/hadoop
# mkdir. m2

2. Compiling Hadoop

Adding Hadoop local libraries to environment variables before Hadoop is compiled

Export hadoop_opts= "-djava.library.path=${hadoop_home}/lib/native"

Download the Hadoop source package from the website and unzip

$ TAR-ZXVF hadoop-2.7.3-src.tar.gz
$ cd hadoop-2.7.3-src/

Enter the following command to start compiling, the compilation process is long, wait patiently

$ mvn PACKAGE-DSKIPTESTS-PDIST,NATIVE-DTAR-DREQUIRE.SNAPPY-E-X

If you see build SUCCESS, and there is no exception information, the Hadoop has been compiled successfully

After compiling, you can see the compressed package of Hadoop

$ cd/home/hadoop/hadoop-2.7.3-src/hadoop-dist/target

The author configured the pseudo-distributed Hadoop, unzip the compiled Hadoop compression package, replace the inside of the configuration file can be started to enter the following command

$ HDFs Namenode-format    #只在第一次启动时执行
$ start-dfs.sh
$ start-yarn.sh
$ mr-jobhistory-daemon.sh Start Historyserver

During startup, always prompt for password, the solution is to configure SSH login, enter the following command

$ su Root
# ssh-keygen-t RSA
# CP id_rsa.pub Authorized_keys
# exit

Check the local library for Hadoop

$ Hadoop checknative