ubuntu14.04 installation hadoop2.7.1 pseudo-distributed and error resolution

ubuntu14.04 installation hadoop2.7.1 pseudo-distributed and error resolution
Need to explain is that I downloaded the source code, by compiling the source code and install
First, the software needs to be prepared:
1.JDK and GCC
Set Java_home:
Vim ~/.BASHRC
Added at last: Export java_home=/usr/lib/jvm/jdk1.8.0_31, save exit.
Make the setting effective immediately: source ~/.BASHRC
Test whether the Java_home is set successfully, output the path set above to indicate success:
Echo $JAVA _home
2. Install Maven
Decompression TAR-ZXVF apache-maven-3.2.5-bin.tar.gz
Configure environment variables, edit/etc/profile or ~/.BASHRC files
Export Maven_home=/opt/maven/current/bin
Export path= $PATH: $MAVEN _home/bin
Last to Source/etc/profile.
Check for normal installation by MVN--version
3. Installing PROTOBUF
Decompression TAR-ZXVF protobuf-2.5.0.tar.gz
Enter the protobuf-2.5.0 directory and execute the following command
./configure
Make
Make check
sudo make install
Protoc--version can be used to see if the installation is OK
If it fails, then:
1. Put all the users need to use the library to/usr/loca/lib;
2. Create a new file usr-libs.conf by VI in the/etc/ld.so.conf.d/directory, the content is:/usr/local/lib
3. #sudo Ldconfig
4. Installing CMake
Decompression TAR-ZXVF cmake-3.4.0.tar.gz
Enter the cmake-3.4.0 directory and execute the following command
./bootstrap
Make
sudo make install
CMake--version can be used to see if the installation is OK
5. Installing Autotool
sudo apt-get install autoconf automake libtool
6. Configure SSH password-free login
To install SSH server:
sudo apt-get install Openssh-server
CD ~/.ssh/# If you do not have this directory, first execute SSH localhost once
SSH-KEYGEN-T RSA # will be prompted, press ENTER to
Cat id_rsa.pub >> Authorized_keys # Add to license
Use ssh localhost to try to log in directly

Second, compile Haodoop:
1. Download the source package hadoop-2.7.1-src.tar.gz and Unzip
TAR-ZXVF hadoop-2.7.1-src.tar.gz
2. Enter the HADOOP-2.7.1-SRC directory and execute
MVN Clean Package-pdist,native-dskiptests-dtar
3. A missing library error was encountered during compilation, so you will need to install the missing library first
such as error messages:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin
Apt-get Install Libglib2.0-dev
Apt-get Install Libssl-dev
If you also reported a similar error during the compilation process, you can find it by yourself in a similar way.
Then there is a long wait, and when you see a series of success, you succeed.
4. The compiled hadoop-2.7.1.tar.gz is in the hadoop-dist/target/directory under the Hadoop-2.7.1-src wood directory, which can then be installed.

5. Installing Hadoop2.7.1
Copy the fourth-step compiled target file to/opt/hadoop/hadoop_2_7_1_64
Command:
sudo cp-r hadoop-dist/target/hadoop-2.7.1/opt/hadoop/hadoop_2_7_1_64
Then create a soft link
sudo ln-s/opt/hadoop/hadoop_2_7_1_64 current

6. Configure Environment variables
Vim ~/.BASHRC
#HADOOP VARIABLES START
Export Hadoop_install=/opt/hadoop/current
Export Hadoop_mapred_home= $HADOOP _install
Export Hadoop_common_home= $HADOOP _install
Export Hadoop_hdfs_home= $HADOOP _install
Export Yarn_home= $HADOOP _install
Export hadoop_common_lib_native_dir= $HADOOP _install/lib/native
Export hadoop_opts= "-djava.library.path= $HADOOP _install/lib/native"
#HADOOP VARIABLES END

Export java_home=/usr/lib/jvm/jdk1.8.0_31
Export JRE_HOME=${JAVA_HOME}/JRE
Export Maven_home=/opt/maven/current/bin
Export classpath=.:${java_home}/lib:${jre_home}/lib: $HIVE _home/lib: $CLASSPATH
Export Path=${path}:${java_hove}/bin: $JRE _home/bin: $MAVEN _home: $HADOOP _install/bin
Make the setting effective immediately: source ~/.BASHRC
Checks whether the installation succeeds, executes the command, and indicates success if the command help appears: HDFs

7. Configuring Pseudo-distributed

Switch to profile directory: Cd/opt/hadoop/current/etc/hadoop
Configure Core-site.xml, execute command: Vim core-site.xml. Replace <configuration></configuration> in file with:
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/opt/hadoop/current/tmp</value>
<description>abase for other temporary directories.</description>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
Configure Hdfs-site.xml, execute command: Vim Hdfs-site.xml, modified to:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/opt/hadoop/current/tmp/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/hadoop/current/tmp/dfs/data</value>
</property>
</configuration>

A little explanation of Hadoop configuration items:
Although you only need to configure FS.DEFAULTFS and dfs.replication to run (this is the official tutorial), if you do not configure the Hadoop.tmp.dir parameter, the default temporary directory is/tmp/hadoo-hadoop. This directory may be removed by the system when it restarts, causing the format to be re-executed. So we set it up, and we also specify Dfs.namenode.name.dir and Dfs.datanode.data.dir, otherwise you might get an error in the next step.

Configure Yarn-site.xml, execute command: Vim Yarn-site.xml, modified to:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
Configure Mapred-site.xml
Copy an XML from the template file, execute the command: MV Mapred-site.xml.template mapred-site.xml
Execute command: Vim mapred-site.xml
Change the document to

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

</configuration>

After the configuration is complete, perform the formatting: HDFs namenode-format. The fifth line appears exitting with status 0 indicates success, and if exitting with status 1 is an error.
If something goes wrong here, there is a good chance that the library is missing.
8. Running
Run the command first before running all services:
HDFs Namenode-format Error
WARN Namenode. Namenode:encountered exception during format:
Java.io.IOException:Cannot Create Directory/usr/local/hd/dfs/name/current
Java.io.IOException:Cannot Create Directory/usr/local/hd/dfs/name/current
This error indicates that the Hadoop file user rights are insufficient and needs to be modified:
sudo chown coco:coco-r Hadoop
Finally, start all of the Hadoop processes:
SH sbin/start-all.sh
To see if each process starts normally, execute: JPS. If everything is OK, you will see the following results:
11010 Secondarynamenode
10596 NameNode
11349 NodeManager
10777 DataNode
10442 Jps

PS: If you use the OH-MY-ZSH command in this machine, you may encounter the following problem
Question one:
? ~ Source ~/.BASHRC
/etc/bashrc:37:command not found:shopt
/etc/bashrc:fc:38:event not found:-A
/etc/bashrc:40:command not found:shopt
/etc/bashrc:51:command not found:shopt

Because the ZSH environment variable needs to be configured in ~/.ZSHRC, it is no longer ~/.BASHRC, so you can copy the same environment variable configuration to the ~/.ZSHRC file,
SOURCE ~/.ZSHRC File
------------------------------------------------------
Question two:
15/11/27 10:55:34 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable
Unable to load local library
You can first view the status with a command:
Hadoop checknative–a

The results are all false, then please check the environment variable configuration file in the following two configuration, whether add native:
Export hadoop_common_lib_native_dir= $HADOOP _install/lib/native
Export hadoop_opts= "-djava.library.path= $HADOOP _install/lib/native"
If not, add on, and then run the Hadoop checknative-a command:
The results will be as follows:
% Hadoop checknative-a
15/11/30 11:24:38 WARN bzip2. bzip2factory:failed to Load/initialize NATIVE-BZIP2 library system-native, would use Pure-java version
15/11/30 11:24:38 INFO zlib. Zlibfactory:successfully Loaded & initialized Native-zlib Library
Native Library checking:
hadoop:true/opt/hadoop/hadoop_2_7_1_64/lib/native/libhadoop.so.1.0.0
Zlib:true/lib/x86_64-linux-gnu/libz.so.1
Snappy:false
Lz4:true revision:99
Bzip2:false
Openssl:true/usr/lib/x86_64-linux-gnu/libcrypto.so
15/11/30 11:24:38 INFO util. Exitutil:exiting with status 1
There are still two false, in fact, snappy,bzip2,lz4,zlib are compressed library, so there are two is true, the other two is false also has no too much influence.
Question three:
Be aware of the issues with your native permissions for the users and user groups to which Hadoop is installed.
You can modify it to a native common user and user group, command:
sudo chown coco:coco-r Hadoop
Question four:
Prompt when running Hadoop fs-ls
LS: '. \\\ ': No such file or directory
At this point, you need to create the directory
Hadoop Fs-mkdir/user
Then run Hadoop fs-ls/
% Hadoop Fs-ls/
Found 1 Items
Drwxr-xr-x-Coco supergroup 0 2015-11-30 10:38/user

This is the end of the installation process.

ubuntu14.04 installation hadoop2.7.1 pseudo-distributed and error resolution