CentOS6.5 install Hadoop

Hadoop implements a Distributed File System (HDFS. HDFS features high fault tolerance and is designed to be deployed on low-cost hardware. It also provides high-throughput (highthroughput) to access application data, suitable for applications with large datasets. HDFS relaxed (relax) POSIX

Hadoop implements a Distributed File System (HDFS. HDFS features high fault tolerance and is designed to be deployed on low-cost hardware. It also provides high throughput to access application data, suitable for applications with large data sets. HDFS relaxed (relax) POSIX requirements and allows you to access data in a streaming access file system as a stream.

 

1. create a new Hadoop user configuration password-free login

 

[Root @ ipython ~] # Groupadd hadoop [root @ ipython ~] # Useradd hadoop-g hadoop [root @ ipython ~] # Passwd hadoop [root @ ipython ~] # Mkdir/tools [root @ ipython ~] # Chown hadoop: hadoop/tools/# Ssh # [root @ ipython ~] # Su-hadoop [hadoop @ ipython ~] $ Ssh-keygen-t dsa-p'-f ~ /. Ssh/id_dsa + -- [DSA 1024] ---- + | BE * |. *. = | + o .. |. o. o + | .. S o. | = o. | o |. | + --------------- + [hadoop @ ipython ~] $ Cat ~ /. Ssh/id_dsa.pub> ~ /. Ssh/authorized_keys [hadoop @ ipython ~] $ Chmod 0600 ~ /. Ssh/authorized_keys

2, JAVA. JDK has been installed (please refer to: [CentOS JDK-1.8 installation ])

 

[Hadoop @ ipython ~] $ Java-versionjava version "1.8.0 _ 25" Java (TM) SE Runtime Environment (build 1.8.0 _ 25-b17) Java HotSpot (TM) 64-Bit Server VM (build 25.25-b02, mixed mode)

3. download and unpackage Hadoop

[Hadoop @ ipython ~] $ Cd/tools/[hadoop @ ipython source] $ wget export source] $ tar zxf hadoop-2.2.0.tar.gz [hadoop @ ipython source] $ ln-s/tools/source/hadoop-2.2.0/tools/hadoop

4. add Hadoop environment variables

[Hadoop @ ipython source] $ cat >> ~ /. Bashrc <

5. change the configuration file

[Hadoop @ ipython source] $ cd $ HADOOP_PREFIX/etc/hadoop [hadoop @ ipython hadoop] $ vi core-site.xml #-------------------------------------------------------# Fs. default. name Hdfs: // 10.211.55.40: 9000 True # ----------------------------------------------------- # [Hadoop @ ipython hadoop] $ vi hdfs-site.xml #-------------------------------------------------------# Dfs. namenode. name. dir File:/hadoop/dfs/name True Dfs. datanode. data. dir File:/hadoop/dfs/data True Dfs. permissions False # Pipeline # [hadoop @ ipython hadoop] $ cp mapred-site.xml.template mapred-site.xml [hadoop @ ipython hadoop] $ vi mapred-site.xml #-------------------------------------------------------# Mapreduce. framework. name Yarn Mapred. system. dir File:/hadoop/mapred/system True Mapred. local. dir File:/hadoop/mapred/local True # ----------------------------------------------------- # [Hadoop @ ipython hadoop] $ vi yarn-site.xml #-------------------------------------------------------# Yarn. nodemanager. aux-services Mapreduce_shuffle Yarn. nodemanager. aux-services.mapreduce.shuffle.class Org. apache. hadoop. mapred. ShuffleHandler #-------------------------------------------------------#

6. start HDFS

 

[Hadoop @ ipython hadoop] $ hdfs namenode-format15/01/23 23:55:40 INFO namenode. FSImage: Saving image file/tools/hadoop/dfs/name/current/fsimage. ckpt_000000000000000 using no compression15/01/23 23:55:40 INFO namenode. FSImage: Image file/tools/hadoop/dfs/name/current/fsimage. ckpt_000000000000000 of size 198 bytes saved in 0 seconds.15/01/23 23:55:40 INFO namenode. NNStorageRetentionManager: Going to retain 1 images with txid> = 015/01/23 23:55:40 INFO util. exitUtil: Exiting with status 015/01/23 23:55:40 INFO namenode. nameNode: SHUTDOWN_MSG: /*************************************** * ******************** SHUTDOWN_MSG: shutting down NameNode at ipython. me/10.211.55.40 ************************************* * **********************/# Start All (namenode, datanode, yarn) ### [hadoop @ ipython hadoop] $ cd $ HADOOP_PREIFX/sbin [hadoop @ ipython sbin] $ start-all.sh # Jps # [hadoop @ ipython sbin] $ jps2656 Jps2000 DataNode2275 export NameNode2374 nodeManager2141 SecondaryNameNode

 

Access HDFS NameNode Web

Hadoop-cluster

 

Access NameNode web UI

Hadoop-namenode-info

 

Access the Resource Manager interface

Hadoop-node-manager

 

Test Hadoop

[Hadoop @ ipython hadoop] $ hdfs dfs-mkdir/user [hadoop @ ipython hadoop] $ hdfs dfs-put/tmp/test/logs

 

Hadoop_test