Learning notes for the "DAY2" Hadoop fully distributed mode configuration

Hadoop Port

----------------

1.namenode 50070

http://namenode:50070/

2.resourcemanager:8088

http://localhost:8088/

3.historyServer

http://hs:19888/

4.name RPC (Remote procedure call, remoted procedure calls)

hdfs://namenode:8020/

SSH commands combined with operation command

---------------------

$>ssh s300 rm-rf/xx/x/x

Remote replication via SCP

--------------------

$>scp-r/xxx/x [Email Protected]:/path

Write scripts that implement files or folders that replicate remotely across all nodes.

xcopy.sh

--------------------

Scp-r Path [Email protected]:/path

Delete

------

Xrm.sh A.txt

SSH s200 RM-RF Path

Remotely copying files

[/usr/local/sbin/xcp.sh]#!/bin/bashif [$#-lt 1]; then echo no args exit;fi#get first argumentarg1=$1;cuser= ' WhoAmI ' fn Ame= ' basename $arg 1 ' dir= ' dirname $arg 1 ' If ["$dir" = "."];  Then dir= ' pwd ' fifor ((i=200;i<=500;i=i+100));d o echo-----coping $arg 1 to $i------; If [-D $arg 1]; then scp-r $arg 1 [email protected] $i: $dir Else SCP $arg 1 [email protected] $i: $dir fi echodone

Slaves

----------

Master

Masters

hadoop2.7.2 Source Code Processing

-----------------------

1. Download and compress the hadoop.2.7.2-tar.gz file

2. Classify jar packages According to Conf,lib,sources,tset, etc.

Extracting all configuration items from a jar package

------------------------

1.core-default.xml

D:\downloads\bigdata\hadoop-2.7.2\_libs\hadoop-common-2.7.2.jar

2.hdfs-default.xml

D:\downloads\bigdata\hadoop-2.7.2\_libs\hadoop-hdfs-2.7.2.jar

3.mapred-default.xml

D:\downloads\bigdata\hadoop-2.7.2\_libs\hadoop-mapreduce-client-core-2.7.2.jar

4.yarn-default.xml

D:\downloads\bigdata\hadoop-2.7.2\_libs\hadoop-yarn-common-2.7.2.jar

Master node = = NameNode

------------------------

{hadoop}/sbin/start-all.sh

--------------------------------------

1.{hadoop}\libexec\hadoop-config.sh

Hadoop_conf_dir= ... //--config Parameters

2./sbin/start-dfs.sh--config $HADOOP _conf_dir

3./sbin/start-yarn.sh--config $HADOOP _conf_dir

{hadoop_home}/sbin/start-dfs.sh

--------------------------------

1.{hadoop}\libexec\hadoop-config.sh

Hadoop_conf_dir= ... //--config Parameters

2.namenode={hadoop_home}/bin/hdfs getconf-namenodes//Extract the hostname of the name node

3.{hadoop_home}/sbin/hadoop-daemons.sh--config--hostnames ...--script "{HADOOP_HOME}/BIN/HDFS" start Namenode $ Datastartopt

4.{hadoop_home}/sbin/hadoop-daemons.sh--config--hostnames ...--script "{HADOOP_HOME}/BIN/HDFS" start Datanode $ Datastartopt

5.{hadoop_home}/sbin/hadoop-daemons.sh--config--hostnames ...--script "{Hadoop_home}/bin/hdfs" start Secondarynamenode

{hadoop_home}/sbin/hadoop-daemons.sh

---------------------------------------

1.{hadoop}\libexec\hadoop-config.sh

Hadoop_conf_dir= ... //--config Parameters

2.exec "$bin/slaves.sh"--config $HADOOP _conf_dir cd "$HADOOP _prefix" \; "$bin/hadoop-daemon.sh"--config $HADOOP _conf_dir "[Email protected]"

{hadoop_home}/sbin/slaves.sh

-----------------------------

1.{hadoop}\libexec\hadoop-config.sh

Hadoop_conf_dir= ... //--config Parameters

2. "${hadoop_conf_dir}/hadoop-env.sh"

3. Extract all host names for the slaves file-->slave_names

4.for slave_names---ssh @hostname ...

"$bin/hadoop-daemon.sh"

-----------------------------

1.{hadoop}\libexec\hadoop-config.sh

Hadoop_conf_dir= ... //--config Parameters

2.namenode|datanode|2namenode|.

Bin/hdfs/xxxx

2NN Configuring a standalone host

--------------------

<property>

<name>dfs.namenode.secondary.http-address</name>

<value>0.0.0.0:50090</value>

<description>

The secondary Namenode HTTP server address and port.

</description>

</property>

Modify the default Hadoop temp directory

-------------------------

[Core-site.xml]

hadoop.tmp.dir=/home/ubuntu/hadoop/

Modify BlockSize size, default is 128m

-----------------------------

[Hdfs-site.xml]

Dfs.blocksize=8m

1. Test method

Put file > 8m, view block size by WebUI

Learning notes for the "DAY2" Hadoop fully distributed mode configuration