Hadoop local deployment (single node)

Install a single Hadoop node to facilitate learning and debugging.

0. Install jdk, which can be easily found in Benbo.

(I used root to play the game directly. You can do this either)

Enter sudo-s. in the terminal window, enter the login password of a common user, and press enter to enter the root user permission mode.

 

Run vim/etc/lightdm. conf.

Add greeter-show-manual-login = true allow-guest = false. The modified configuration file is

 

[SeatDefaults]
Greeter-session = unity-greeter
User-session = Ubuntu
Greeter-show-manual-login = true # manually enter the user name and password for logging on to the system
Allow-guest = false # logon from guest is not allowed

 

1. Download hadoop-2.2.0.tar.gz from Apache)

We recommend that you install Hadoop under/opt for Hadoop path configuration, so change the/opt major to the actual user (this article takes kevin user as an example ):

1. $ Sudo chown kevin: kevin/opt

2. Tar xzvf hadoop-2.2.0.tar.gz

3. $ Mv hadoop-2.2.0/opt/hadoop.

 

Ssh local communication

Sudo apt-get install openssh-client
Sudo apt-get install openssh-server

Cd/root
Ssh-keygen-t rsa-P "" (Press ENTER)
$ Cat. ssh/id_rsa.pub>. ssh/authorized_keys run the command to verify whether the password-free connection to localhost is allowed.

Run the following command:

1. $ Ssh localhost

Global Variables

Add the following content to the end of/etc/profile:

Vim/etc/profile

1. Export HADOOP_HOME =/opt/hadoop

2. Export PATH = $ PATH: $ HADOOP_HOME/bin: $ HADOOP_HOME/sbin

 

Make sure that the hadoop command can be executed anywhere.

Enable the configuration file:

$ Source/etc/profile

 

Directory Configuration

Create a data storage directory for Hadoop and modify the owner permission (this is not required by the root user)

 

$ Sudo mkdir/hadoop
$ Sudo chown krvin: kevin/hadoop
$ Mkdir/hadoop/dfs
$ Mkdir/hadoop/tmp

Configure Hadoop configuration hadoop-env.sh modify hadoop-env.sh files:

$ Cd/opt/hadoop/etc/hadoop
$ Vim hadoop-env.sh.

Change the default JAVA_HOME to/opt/jdk1.6.0 _ 35.

 

Configure core-site.xml

Modify the core-site.xml file:

$ Cd/opt/hadoop/etc/hadoop
$ Vim core-site.xml.

Add the following content to the <configuration> tag (nested in the tag:

<Property>
<Name> hadoop. tmp. dir </name>
<Value>/hadoop/tmp </value>
<Description> temporary directories. </description>
</Property>
<Property>
<Name> fs. defaultFS </name>
<Value> hdfs: // 192.168.1.100: 9000 </value>
<Description> The name of the defaultfile system. Either the literal string "local" or a host: port forNDFS.
</Description>
<Final> true </final>
</Property>

For more configuration information, see:

Http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/core-default.xml

 

Configure hdfs-site.xml

Modify hdfs-site.xml

1. $ Vim hdfs-site.xml.

 

Add the following content to the <configuration> tag (nested in the tag:

<Property>
<Name> dfs. namenode. name. dir </name>
<Value> file:/hadoop/dfs/name </value>
<Description> Determineswhere on the local filesystem the DFS name node shocould store the name table. </description>
<Final> true </final>
</Property>

<Property>
<Name> dfs. datanode. data. dir </name>
<Value> file:/hadoop/dfs/data </value>
<Description> Determineswhere on the local filesystem an DFS data node shocould store its blocks.
</Description>
<Final> true </final>
</Property>

<Property>
<Name> dfs. replication </name>
<Value> 1 </value>
</Property>

<Property>
<Name> dfs. permissions </name>
<Value> false </value>
</Property>

For more hdfs-site.xml configuration information, see:

Http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

Copy local files to HDFS

Download files from HDFS to local

Upload local files to HDFS

Common commands for HDFS basic files

Introduction to HDFS and MapReduce nodes in Hadoop

Hadoop practice Chinese version + English version + Source Code [PDF]

Hadoop: The Definitive Guide (PDF]

1 2 Next Page