Hadoop 2.6 pseudo-distributed installation

"Pseudo" fractional installation of Hadoop 2.6 compared to "full" fractional installation, 99% of the operation is the same, the only difference is not to configure the slaves file, here are a few key configurations:

(Install JDK, create user, set SSH password, set environment variable these preparations, you can refer to the HADOOP 2.6 full distribution installation article, all the following configuration files, all in the $hadoop_home/etc/hadoop directory)

First, modify hadoop-env.sh, yarn-env.sh

The main is to set the path of Java_home

Ii. Modification of Core-site.xml

1 <Configuration>2   < Property>3     <name>Fs.default.name</name>4     <value>hdfs://localhost:9000</value>5   </ Property>6   < Property>7     <name>Hadoop.tmp.dir</name>8     <value>/opt/app/hadoop-2.6.0/tmp</value>9   </ Property>Ten </Configuration>

Third, modify the Hdfs-site.xml

1 <Configuration>2   < Property>3     <name>Dfs.name.dir</name>4     <value>/opt/app/hadoop-2.6.0/hdfs/name</value>5     <Description>Namenode Storage Path</Description>6   </ Property>7   < Property>8     <name>Dfs.data.dir</name>9     <value>/opt/app/hadoop-2.6.0/hdfs/data</value>Ten     <Description>Datanode Storage Path</Description> One   </ Property> A   < Property> -     <name>Dfs.replication</name> -     <value>1</value> the   </ Property> - </Configuration>

Iv. Modification of Mapred-site.xml

1 <Configuration>2   < Property>3     <name>Mapred.job.tracker</name>4     <value>hdfs://localhost:9001</value>5   </ Property>6   < Property>7     <name>Mapred.local.dir</name>8     <value>/opt/app/hadoop-2.6.0/mapred/local</value>9     <Description>MapReduce Storage Path</Description>Ten   </ Property> One   < Property> A     <name>Mapred.system.dir</name> -     <value>/opt/app/hadoop-2.6.0/mapred/system</value> -     <Description>MapReduce System Level Storage path</Description> the   </ Property> - </Configuration>

V. Modification of Yarn-site.xml

1 <Configuration>2   < Property>3     <name>Yarn.nodemanager.aux-services</name>4     <value>Mapreduce_shuffle</value>5   </ Property>6 </Configuration>

Note: The above configuration in the directory, to do exist, if not, please create manually, refer to the command mkdir-p mapred/system

You can start the test:

1. Format first

Bin/hdfs Namenode–format

2. Start DFS, yarn

sbin/start-dfs.sh

sbin/start-yarn.sh

Then using JPS to view the Java process, you should see several processes:

25361 NodeManager
24931 DataNode
25258 ResourceManager
24797 NameNode
25098 Secondarynamenode

You can also view the reports for HDFs with the following command:

Bin/hdfs Dfsadmin-report Under normal circumstances, you can see the following

configured capacity:48228589568 (44.92 GB)
Present capacity:36589916160 (34.08 GB)
DFS remaining:36589867008 (34.08 GB)
DFS used:49152 (KB)
DFS used%: 0.00%
under replicated blocks:0
Blocks with corrupt replicas:0
Missing blocks:0

-------------------------------------------------
Live datanodes (1):

name:127.0.0.1:50010 (localhost)
hostname:dc191
Decommission Status:normal
configured capacity:48228589568 (44.92 GB)
DFS used:49152 (KB)
Non DFS used:11638673408 (10.84 GB)
DFS remaining:36589867008 (34.08 GB)
DFS used%: 0.00%
DFS remaining%: 75.87%
configured Cache capacity:0 (0 B)
Cache used:0 (0 B)
Cache remaining:0 (0 B)
Cache used%: 100.00%
Cache remaining%: 0.00%
xceivers:1
Last contact:tue 17:42:54 CST

3. Web Management Interface View

http://localhost:50070/

http://localhost:8088/

4. Create a directory in HDFs

Bin/hdfs dfs-mkdir/x

This creates a directory in HDFs x

5. Put the file into HDFs

Bin/hdfs Dfs-put readme.txt/x

The above command will put the README.TXT in the current directory into the/x directory of HDFs, which can also be seen in the Web management interface.

Hadoop 2.6 pseudo-distributed installation