CentOS6.5 install Hadoop
Hadoop implements a Distributed File System (HDFS. HDFS features high fault tolerance and is designed to be deployed on low-cost hardware. It also provides high throughput to access application data, suitable for applications with large data sets. HDFS relaxed (relax) POSIX requirements and allows you to access data in a streaming access File System as a stream.
1. Create a New Hadoop user configuration password-free Login
[Root @ ipython ~] # Groupadd hadoop [root @ ipython ~] # Useradd hadoop-g hadoop [root @ ipython ~] # Passwd hadoop [root @ ipython ~] # Mkdir/tools [root @ ipython ~] # Chown hadoop: hadoop/tools/# Ssh # [root @ ipython ~] # Su-hadoop [hadoop @ ipython ~] $ Ssh-keygen-t dsa-p'-f ~ /. Ssh/id_dsa + -- [DSA 1024] ---- + | BE * |. *. = | + o .. |. o. o + | .. S o. | = o. | o |. | + --------------- + [hadoop @ ipython ~] $ Cat ~ /. Ssh/id_dsa.pub> ~ /. Ssh/authorized_keys [hadoop @ ipython ~] $ Chmod 0600 ~ /. Ssh/authorized_keys
2, JAVA. JDK has been installed (please refer to: [CentOS JDK-1.8 installation ])
[Hadoop @ ipython ~] $ Java-versionjava version "1.8.0 _ 25" Java (TM) SE Runtime Environment (build 1.8.0 _ 25-b17) Java HotSpot (TM) 64-Bit Server VM (build 25.25-b02, mixed mode)
3. Download and unpackage Hadoop
[Hadoop @ ipython ~] $ Cd/tools/[hadoop @ ipython source] $ wget export source] $ tar zxf hadoop-2.2.0.tar.gz [hadoop @ ipython source] $ ln-s/tools/source/hadoop-2.2.0/tools/hadoop
4. Add Hadoop Environment Variables
[Hadoop @ ipython source] $ cat >> ~ /. Bashrc <EOFexport HADOOP_PREFIX = "/tools/hadoop" export PATH =\$ PATH: \ $ HADOOP_PREFIX/binexport PATH =\$ PATH: \ $ export/sbinexport expiration =\$ {HADOOP_PREFIX} export YARN_HOME =\$ {HADOOP_PREFIX }#### hadoop-env #### export JAVA_HOME = "/tools/java" export HADOOP_COMMON_LIB_NATIVE_DIR =\\ {HADOOP_PREFIX}/lib/nativee Xport HADOOP_OPTS = "-Djava. library. path = \ $ HADOOP_PREFIX/lib "#### yarn-env #### export expiration =\$ {HADOOP_PREFIX}/lib/nativeexport HADOOP_OPTS ="-Djava. library. path = \ $ HADOOP_PREFIX/lib "EOF [hadoop @ ipython source] $ source ~ /. Bashrc
5. Change the configuration file
[Hadoop @ ipython source] $ cd $ HADOOP_PREFIX/etc/hadoop [hadoop @ ipython hadoop] $ vi core-site.xml # others # <configuration> <property> <name> fs. default. name </name> <value> hdfs: // 10.211.55.40: 9000 </value> <final> true </final> </property> </configuration> # ------------------------------------------------- # [hadoop @ ipython hadoop] $ vi hdfs-site.xml #------------ ----------------------------------------- # <Configuration> <property> <name> dfs. namenode. name. dir </name> <value> file:/hadoop/dfs/name </value> <final> true </final> </property> <name> dfs. datanode. data. dir </name> <value> file:/hadoop/dfs/data </value> <final> true </final> </property> <name> dfs. permissions </name> <value> false </value> </property> </configuration> #----------------------------------- -------------------- # [Hadoop @ ipython hadoop] $ cp mapred-site.xml.template mapred-site.xml [hadoop @ ipython hadoop] $ vi mapred-site.xml # others # <configuration> <property> <name> mapreduce. framework. name </name> <value> yarn </value> </property> <name> mapred. system. dir </name> <value> file:/hadoop/mapred/system </value> <final> true </final> </property> <proper Ty> <name> mapred. local. dir </name> <value> file: /hadoop/mapred/local </value> <final> true </final> </property> </configuration> # ----------------------------------------------------- # [hadoop @ ipython hadoop] $ vi yarn-site.xml # examples #------------------------------------------------------- # <configuration> <! -- Site specific YARN configuration properties --> <property> <name> yarn. nodemanager. aux-services </name> <value> mapreduce_shuffle </value> </property> <name> yarn. nodemanager. aux-services.mapreduce.shuffle.class </name> <value> org. apache. hadoop. mapred. shuffleHandler </value> </property> </configuration> #-------------------------------------------------------#
6. Start HDFS
[Hadoop @ ipython hadoop] $ hdfs namenode-format15/01/23 23:55:40 INFO namenode. FSImage: Saving image file/tools/hadoop/dfs/name/current/fsimage. ckpt_000000000000000 using no compression15/01/23 23:55:40 INFO namenode. FSImage: Image file/tools/hadoop/dfs/name/current/fsimage. ckpt_000000000000000 of size 198 bytes saved in 0 seconds.15/01/23 23:55:40 INFO namenode. NNStorageRetentionManager: Going to retain 1 images with txid> = 015/01/23 23:55:40 INFO util. exitUtil: Exiting with status 015/01/23 23:55:40 INFO namenode. nameNode: SHUTDOWN_MSG: /*************************************** * ******************** SHUTDOWN_MSG: shutting down NameNode at ipython. me/10.211.55.40 ************************************* * **********************/# Start All (namenode, datanode, yarn) ### [hadoop @ ipython hadoop] $ cd $ HADOOP_PREIFX/sbin [hadoop @ ipython sbin] $ start-all.sh # Jps # [hadoop @ ipython sbin] $ jps2656 Jps2000 DataNode2275 export NameNode2374 nodeManager2141 SecondaryNameNode
Access HDFS NameNode Web
Hadoop-cluster
Access NameNode web UI
Hadoop-namenode-info
Access the Resource Manager interface
Hadoop-node-manager
Test Hadoop
[Hadoop @ ipython hadoop] $ hdfs dfs-mkdir/user [hadoop @ ipython hadoop] $ hdfs dfs-put/tmp/test/logs
Hadoop_test