Hadoop 2.5.2 installation and deployment

Hadoop 2.5.2 installation and deployment

Hadoop 2.5.2 installation and deployment

Software and version:

Hadoop version	Hadoop-2.5.2
Operating System	Red Hat Linux 6.4	32bit
JDK version	Jdk-8u25-linux-i586.rpm

Software preparation:

Download hadoop-2.5.2:

Wget http://mirror.bit.edu.cn/apache/hadoop/common/stable2/hadoop-2.5.2.tar.gz

Download JDK: jdk-8u25-linux-i586.rpm
Http://www.Oracle.com/technetwork/java/javase/index.html

The overall process is as follows:
1. Implement ssh password-less authentication Configuration
2. Install jdk and configure Environment Variables
3. install and configure Hadoop
4. Format and start
5. Verify that the instance is started.

New Features of Hadoop2.5.2

Install and configure Hadoop2.2.0 on CentOS

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition

Build a Hadoop environment (using virtual machines to build two Ubuntu systems in a Winodws environment)

I. No password verification for SSH Between Hosts
Use the ssh-kengen-t rsa command to generate a public key and copy the public keys between hosts to the authorized_keys file.

Ii. Install JDK
After installation, use java-version to verify

Configure environment variables:

### Set java_env

Export JAVA_HOME =/usr/java/jdk1.8.0 _ 25/
Export JRE_HOME =/usr/java/jdk1.8.0 _ 25/jre
Export CLASS_PATH =.: $ CLASS_PATH: $ JAVA_HOME/lib: $ JRE_HOME/lib
Export PATH = $ PATH: $ JAVA_HOME/bin: $ JRE_HOME/bin

### Set hadoop_env
Export HADOOP_HOME =/home/bkjia/hadoop-2.5.2
 
Export HADOOP_COMMON_HOME = $ HADOOP_HOME
Export HADOOP_HDFS_HOME = $ HADOOP_HOME
Export HADOOP_MAPRED_HOME = $ HADOOP_HOME
Export HADOOP_YARN_HOME = $ HADOOP_HOME
Export HADOOP_CONF_DIR = $ HADOOP_HOME/etc/hadoop
Export PATH = $ PATH: $ HADOOP_HOME/bin: $ HADOOP_HOME/sbin: $ HADOOP_HOME/lib
 
Export HADOOP_COMMON_LIB_NATIVE_DIR = $ HADOOP_HOME/lib/native
Export HADOOP_OPTS = \ "-Djava. library. path = $ HADOOP_HOME/lib \"

3. Deploy and configure Hadoop
Decompress Hadoop to your hadoop directory.

Configuration-related configuration files
The 2.5.x configuration file is in the $ Hadoop_Home/etc/hadoop directory.
2. version X is later than Version 1. the version X has been greatly changed. It mainly uses Hadoop MapReduceV2 (Yarn) framework to replace a generation of architecture. JobTracker and TaskTracker are missing and replaced by ResourceManager, ApplicationMaster, and NodeManager, the location and content of the configuration file have also changed accordingly. For more information, see:

(1) hadoop/etc/hadoop/hadoop-env.sh and hadoop/etc/hadoop/yarn-env.sh to configure JAVA_HOME in two files

(2) etc/hadoop/core-site.xml, configured:

<Configuration>

 

<Property>

<Name> hadoop. tmp. dir </name>

<Value>/home/bkjia/hadoop-2.5.2/tmp </value>

<Description> A base for other temporary directories. </description>

</Property>

 

<Property>

<Name> fs. default. name </name>

<Value> hdfs :/// namenode: 9000 </value>

</Property>

 

<Property>

<Name> io. file. buffer. size </name>

<Value> 131072 </value>

</Property>

 

<Property>

<Name> hadoop. proxyuser. root. hosts </name>

<Value> namenode </value>

</Property>

<Property>

<Name> hadoop. proxyuser. root. groups </name>

<Value> * </value>

</Property>

</Configuration>

(3) etc/hadoop/hdfs-site.xml, configured as: (Note: here you need to manually create the name and data folder with mkdir, the specific location can also be selected, where dfs. we recommend that you set the replication value to the actual number of DataNode hosts in the distributed cluster .)

<Configuration>

<Property>

<Name> dfs. namenode. name. dir </name>

<Value>/home/bkjia/hadoop-2.5.2/hdfs/name </value>

<Final> true </final>

</Property>

 

<Property>

<Name> dfs. datanode. data. dir </name>

<Value>/home/bkjia/hadoop-2.5.2/hdfs/data </value>

<Final> true </final>

</Property>

 

<Property>

<Name> dfs. replication </name>

<Value> 2 </value>

</Property>

 

<Property>

<Name> dfs. permissions </name>

<Value> false </value>

</Property>

 

</Configuration>

For more details, please continue to read the highlights on the next page:

1 2 Next Page