Install Hadoop in Ubuntu system

1. Environment:ubuntu,hadoop2.7.3

2. Create hadoop user groups and users under Ubuntu

① Adding Hadoop users to system users

② now just added a user Hadoop, which does not have administrator privileges, we Add permissions to Hadoop users, open /etc/sudoers File

command:sudo vi/etc/sudoers

③ add Hadoop all= (all:all) all under root all= (All:all) all

This file is a read-only file and is forced to be saved after modification:: wq!

3. installing the SSH service

command:sudo apt-get install ssh openssh-server

4. Login without password authentication using ssh

① first into Hadoop users

command: su Hadoop

② as a Secure Communication protocol (SSH generated key has RSA and DSA two ways to generate, by default RSA mode), requires a password when used, so we want to set the password-free login to generate the private key and the public key:

(note: After carriage return , two files will be generated under ~/.ssh/:id_rsa and id_rsa.pub These two files are in pairs, the former is the private key, The latter is the public key)

Enter the ~/.ssh/directory to append the public key id_rsa.pub to the authorized_keys authorization file, starting with no authorized_keys file (authorized_keys is used to save all allowed logins to SSH as the current user Client user's public key content):

command:cat ~/.ssh/id_rsa.pub>> ~/.ssh/authorized_keys

③ can then login without password Authentication

command:ssh localhost

5. download the Hadoop installation package

Website address:

https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/stable/

6. Unzip the Hadoop installation package

to ensure that all operations are done under User Hadoop , set the owner of the Hadoop folder to Hadoop

command:sudo chown-r hadoop:hadoop Hadoop

7. Installing Java

8. Configure the appropriate files in Hadoop

the files that need to be configured are as follows,hadoop-env.sh,core-site.xml,mapred-site.xml.template , Hdfs-site.xml , all files are located under/usr/local/hadoop/etc/hadoop , and the following configuration is required:

The ①core-site.xml configuration is as follows:

<configuration>

<property>

<name>hadoop.tmp.dir</name>

<value>file:/usr/local/hadoop/tmp</value>

<description>abase for other temporary directories.</description>

</property>

<property>

<name>fs.defaultFS</name>

<value>hdfs://localhost:9000</value>

</property>

</configuration>

The ②mapred-site.xml.template configuration is as follows:

<configuration>

<property>

<name>mapred.job.tracker</name>

<value>localhost:9001</value>

</property>

</configuration>

The ③hdfs-site.xml configuration is as follows:

<configuration>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

<property>

<name>dfs.namenode.name.dir</name>

<value>file:/usr/local/hadoop/tmp/dfs/name</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>file:/usr/local/hadoop/tmp/dfs/data</value>

</property>

</configuration>

where the paths of Dfs.namenode.name.dir and dfs.datanode.data.dir can be set freely, preferably in Hadoop.tmp.dir the directory below.

Supplement, if run Hadoop found no jdk, you can directly put the JDK path in the hadoop-env.sh inside, Specific as follows:

　　　　Export Java_home= "/usr/local/jdk1.8.0_91"

9. Running Hadoop

① initializing HDFS system

command:bin/hdfs Namenode-format

② Open NameNode and DataNode Daemon

command:sbin/start-dfs.sh, Success is as follows:

③ Viewing process information

Command: JPS

④ viewing the Web UI

in the browser input http://192.168.86.126:50070/

running WordCount Demo

① Create a new file locally, in the/home/download/wangling directory to create a new words Document, the contents can be filled in freely. This words is written in the following:

Quux Labs Foo Bar Quux

② Create a new folder in HDFS to upload the local words Document and Enter the following command in the hadoop directory:

command:bin/hdfs dfs-mkdir/test, which establishes a test directory in the root directory of HDFs

use the following command to view directory structure under the HDFS root directory

command:bin/hdfs dfs-ls/

③ uploading the local words document to the test Directory

command:bin/hdfs dfs-put/home/download/wangling/words/test/

indicates that the local words document has been uploaded to the test directory.

④ running wordcount

command:bin/hadoop jar Share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount/test/words/test/ Out

after the run is complete, generate a file named out in the/test directory

view command:Bin/hdfs dfs-ls/test

⑤ View Run Results

command:bin/hdfs dfs-ls/test/out

Bin/hadoop fs-cat/test/out/part-r-00000

Install Hadoop in Ubuntu system