Configure the Hadoop environment in Ubuntu

Configure the Hadoop environment in Ubuntu

Configuring the Hadoop environment in Ubuntu to implement truly distributed Hadoop is not pseudo-distributed.

I. System and Configuration

We have prepared two machines to build a Hadoop cluster. Based on Ubuntu14.04, jdk1.6.0 _ 45, and Hadoop1.0.3, the VM uses VMware10.0

192.168.1.10 NameNode master (master)

192.168.1.20 DateNode slave1 (Slave)

My user name is hadoop

Next we need to install some common software: vim, ssh

Sudo apt-get update
Sudo apt-get install vim
Sudo apt-get install ssh

First modify the local IP Address

Run the following command to modify the hosts

Sudo vim/etc/hosts

Next, set ssh to achieve password-less login. Enter the following command to generate the key

Ssh-keygen-t rsa-P ""

Press enter all the time to generate the key, and then the id_rsa and id_rsa.pub files will be generated in the. ssh directory. These two files are the SSH private key and Public Key respectively.

Generate the authorized_keys file: cat id_rsa.pub> authorized_keys to log on to the local localhost ssh without a password,

Ii. Install hadoop and jdk

Decompress the hadoop compressed package. My installation directory is/home/hadoop (this is the user name)/hadoop (this is the folder )/

Tar-zxvf hadoop1.0.3.tar

Decompress the jdk compressed package. The command is similar to the above, but the file name is not listed here.

Next, modify the environment variable.

Sudo vim/etc/profile
Export JAVA_HOME =/home/hadoop/jdk1.6.0 _ 45
Export PATH = $ JAVA_HOME/bin: $ PATH
Export HADOOP_HOME =/home/hadoop/hadoop-1.0.3
Export PATH = $ JAVA_HOME/bin: $ HADOOP_HOME/bin: $ PATH

Do not forget to execute source/etc/profile to make the path take effect immediately

Finally, configure the files in the conf folder under hadoop.

Modify hadoop-env.sh

Modify core-site.xml

Modify hdfs-site.xml

Modify mapred-site.xml

Modify the masters and slaves files. In the masters, write only the master (192.168.1.0 mentioned above), and fill in the master and slave files in slaves.

Format namenode and write the following command in the hadoop-1.0.3 File

Bin/hadoop namenode-format

By the way, the slave configuration has not been mentioned here. It is actually very simple. Shut down the current virtual machine, copy a file of the virtual machine just now, rename it, and open it again, modify the user name and ip address. My Ubuntu name is the same, as long as it is not in a disk.

Finally, enter the following command in the master (user name, that is, the master node's ubuntu), also in the hadoop-1.0.3 File

Bin/start-all.sh

Enter JPS to view the java process. If the following five processes appear, the process is successful (excluding jps)

View webpages

There are already two nodes, and the entire hadoop distributed deployment is complete.

Install and configure Hadoop2.2.0 on CentOS

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition

Build a Hadoop environment (using virtual machines to build two Ubuntu systems in a Winodws environment)