Hadoop introduction and latest stable version hadoop 2.4.1 download address and single-node Installation

Hadoop Introduction

Hadoop is a software framework that can process large amounts of data in a distributed manner. Its basic components include the HDFS Distributed File System and the mapreduce programming model that can run on the HDFS file system, as well as a series of upper-layer applications developed based on HDFS and mapreduce.

HDFS is a distributed file system that stores large files in a network in streaming data access mode across multiple computers. The range of ultra-large files currently supported is from MB to Pb.

Mapreduce is a programming model that can be used for data processing. Programs Based on the mapreduce model are essentially running in parallel. Programs Based on the mapreduce programming model include map functions for data extraction, merge functions for processing intermediate results (merge functions are generally optional), and reduce functions for generating final processing results. The data processed by the map function and Merge function is the key-value of the sorted and grouped data. After the intermediate results are processed by reduce, the final calculation result is generated. The map function runs in parallel, and each map function processes a file block of a large file. Therefore, for large files based on HDFS file systems, the map function can take full advantage of the processing capabilities of multiple computers to quickly calculate and generate intermediate results.

 

The Apache™Hadoop®Project develops open-source software for reliable, scalable, distributed computing.

The Apache hadoop software library is a framework that allows for the Distributed Processing of large data sets feature SS clusters of computers using simple programming models. it is designed to scale up from single servers to thousands of machines, each offering local computation and storage. rather than rely on hardware to deliver high-availability, the Library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

The project schemdes these modules:

Hadoop common: the common utilities that support the other hadoop modules.

Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data.

Hadoop yarn: A Framework for Job Scheduling and cluster resource management.

Hadoop mapreduce: a yarn-based system for parallel processing of large data sets. www.169it.com

 

Latest stable version of hadoop 2.4.1

Hadoop 2.4.1 (2.x series stable version ):Hadoop 2.4.1 download

 

 

Hadoop 2.4.1 Virtual Machine installation-single-node Installation Steps

1. Setup of Java and Java environment variables

2. Set the account, Hostname/etc/hosts of the host.

Add the following content to your. bash_profile:

12345

export JAVA_HOME=/usr/java/jdk1.7.0_60export HADOOP_PREFIX=/home/hadoop/hadoop-2.4.1export CLASSPATH=".:$JAVA_HOME/lib:$CLASSPATH"export PATH="$JAVA_HOME/:$HADOOP_PREFIX/bin:$PATH"export HADOOP_PREFIX PATH CLASSPATH

3. Set login without a password

First, make sure that the firewall of all hosts is disabled.

1234	$cd ~/.ssh$ Ssh-keygen-t rsa # press the Enter key until the generated key is saved in the. Ssh/id_rsa file according to the default options.$cp id_rsa.pub authorized_keyssudo service sshd restart

4 configure hadoop2.4.1

Go to the hadoop2.4.1 folder and configure files in ETC/hadoop.

12	hadoop-env.shexport JAVA_HOME=/usr/java/jdk1.7.0_60

In addition, you can add the following options:

12	export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/nativeexport HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib"

 

5 configure core-site.xml for hadoop2.4.1

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455

<configuration>       <property>                <name>fs.default.name</name>                <value>hdfs://localhost:9000</value>       </property>       <property>                <name>io.file.buffer.size</name>                <value>131072</value>        </property>       <property>               <name>hadoop.tmp.dir</name>               <value>file:/home/hadoop/tmp</value>       </property>        <property>                <name>dfs.namenode.name.dir</name>               <value>file:/home/hadoop/hadoop-2.4.1/dfs/name</value>       </property>       <property>               <name>dfs.datanode.data.dir</name>               <value>file:/home/hadoop/hadoop-2.4.1/dfs/data</value>       </property></configuration>hdfs-site.xml <configuration>        <property>                <name>dfs.namenode.name.dir</name>               <value>file:/home/hadoop/hadoop-2.4.1/dfs/name</value>       </property>       <property>               <name>dfs.datanode.data.dir</name>               <value>file:/home/hadoop/hadoop-2.4.1/dfs/data</value>       </property>       <property>               <name>dfs.replication</name>               <value>1</value>       </property></configuration>mapred-site.xml<configuration>       <property>               <name>mapreduce.jobtracker.address</name>               <value>hdfs://localhost:9001</value>       </property></configuration>yarn-site.xml<configuration>       <property>               <name>mapreduce.framework.name</name>               <value>yarn</value>       </property>       <property>               <name>yarn.nodemanager.aux-services</name>               <value>mapreduce_shuffle</value>       </property></configuration>

 

After the preceding five steps, the configuration of the single-host environment of hadoop2.4.1 has been completed. Start the following:

./Bin/hadoop namenode-format node Information

Bin/start-all.sh. The new version of hadoop is not recommended so direct start-all, it is recommended to step by step, first start-DFS, and then in start-Map

./Bin/hadoop dfsadmin-Report

Http: // localhost: 50070

 

Source: hadoop introduction and latest stable version hadoop 2.4.1 and single-node Installation

Hadoop introduction and latest stable version hadoop 2.4.1 and single-node Installation