29.Hadoop of HDFs cluster build notes

Source: Internet
Author: User
Tags hdfs dfs hadoop fs

0. Modify IP, host name, Hosts fileSetupModify network card IPService Network RestartRestart the Network service for IP to take effect
Vim/etc/sysconfig/network Modify Host name vim/etc/hosts/Modify Hosts file hadoop001192.168.126.129 hadoop002192.168.126.130 hadoop003192.168.126.131 hadoop0041. Installing the JDK environment ALT + P         Open remote Connection SECURECRT Remote Transport tool SFTP   put    c:/........tar.gz         The win on the software to linux  & nbsp     Mkdir-p/java/jdk         Build multi-level folders         TAR-ZXVF jdk----. tar.gz &NB Sp;-c  /java/jdk         Unzip JDK to specified path         vim/etc/profile     &NBS P   Modify Global environment variables                 Export Java_home=/java/jdk/jdk1.7.0_65export path= $JAVA _home/bin: $PATHsource/etc/profile Reload profile echo$java_home print variable java_homejava-version see if environment variable configuration is installed successfully Protobuftar -ZXVF PROTOBUF-2.5.0.TAR.GZ&NBSP;CD protobuf-2.5.0./configure--prefix=/opt Configuring environment variables join/opt/binprotoc-–version  2. Installing HADOOPTAR-ZXVF hadoop-2.4.1.tar.gz-c/java/decompression hadoopls lib/native/See what files are in the extracted directory CD etc/hadoop/into the profile directory vim hadoop-env.sh Modify Profile environment variable (export java_home=/java/jdk/jdk1.7.0_65) *-site.xml*vim core-site.xml Modify configuration file (go to official website for parameter meaning) < configuration><property><name>fs.defaultfs</name><value>hdfs://hadoop001:9000< value></property><property><name>hadoop.tmp.dir</name><value>/java/ Hadoop-2.4.1/tmp<value></property></configuration> vim Hdfs-site.xml modifying HDFs Profiles < configuration><property><!--number of replicas, default 3--><name>dfs.replication</name><value>2 &LT;/VALUE&GT;&LT;/PROPERTY&GT;&LT;/CONFIGURATION&GT;&NBSP;CP mapred-site.xml.template Mapred-site.xml Copy a sample configuration file vim Mapred-site.xml modify MapReduce profile <configuration><property><name> Mapreduce.framework.name</name><value>yarn</value></property></configuration>  vim yarn-site.xml Configure the Yarn profile, or you can configure the hostname if you modify the HOSTs file mapping <configuration><!--Site specific YARN configuration Properties--><!--Master node--><property ><name>yarn.resourcemanager.hostname</name><value>hadoop001</value></property ><!--from Node--><property><name>yarn.nodemanager.aux-services</name><value> Mapreduce_shuffle</value></property></configuration> vim Slaves configuration from node (can be host name);SCP/JAVA/HADOOP-2.4.1/[email protected]:/java/ HADOOP-2.4.1/copied to other machines  cd/java/hadoop-2.4.1/bin./hadoop Namenode-format formatted HDFs file system (built files and folders) Success: INFO Common. Storage:storage Directory/java/hadoop-2.4.1/tmp/dfs/name has been successfully formatted 3. Start JPS look at the Java process, There's nothing at this time CD sbin/to Sbin to find the start command./start-dfs.sh start hdfs[[email protected] sbin]# jps2488 NameNode2746 SecondaryNameNode2872 Jps2601 datanode [[email protected] ~]# jps2422 DataNode2485 jps 4. Start HDFs Client CD/ Java/hadoop-2.4.1/bin into BIN./HDFs dfs-ls/view HSFS/lower content./hdfs dfs-put/root/install.log hdfs://hadoop001:9000/files to the HDFs file system (128M per block)./hdfs dfs-ls/View Hsfs /under Content ls/java/hadoop-2.4.1/tmp/dfs/View File save location (Namenode,datanode) ls/java/hadoop-2.4.1/tmp/dfs/(datanode) only Data folder CD /java/hadoop-2.4.1/tmp/dfs/data/current/bp-343820156- view files are divided into several pieces. HDFs dfs-get  hdfs://hadoop001:9000/install.log download files from HDFS (specify downloaded directories after get)   principle: Touch hadoop.tar.gz build an empty file cat Block1 >> hadoop.tar.gz add block to Hadoop.tar.gzcat Block2 >> hadoop.tar.gz Append Block2 to Hadoop.tar.gz ./hdfs dfshdfs other commands./hdfs dfs-chown tangwan:hadoop/install.log Change the HDFs file user owners and groups./hdfs Dfs-ls/install.log View the post-change status./hdfs dfs-chmod 600/install.log Change permissions./hdfs dfs-ls/View permissions Change permissions after users can still download but if you create a folder to give permissions, You can only specify permissions for the user to upload./hdfs DFS-DF/See how much space is left in HDFs Kb./hdfs dfs-df-h/See how much space is left in HDFs m,g./hdfs dfs-mkdir-p/aaa/bbb Create a folder in HDFs         ./Hadoop fs-du-s/#查看hdfs占用空间Stop Hdfs:/java/hadoop-2.4.1/sbin Enter sbin./stop-dfs.sh stop hdfshadoop001:50070 browser interface (can download not can upload)

From for notes (Wiz)

29.Hadoop of HDFs cluster build notes

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.