hadoop shell commands

Preface: Well, it's just a little bit more comfortable without writing code, but we can't slack off. The hive operation's file needs to be loaded from hereSimilar to the Linux commands, the command line begins with the Hadoop FS -(dash) LS / list file or directory cat Hadoop FS -cat ./hello.txt/opt/old/ Htt/hello.txt View files can dump directories or

1. View HelpHadoop fs-help 2. UploadPaths on files > such as: Hadoop fs-put test.log/3. View the contents of the filePaths on Hadoop fs-cat such as: Hadoop fs-cat/test.log4. View File listHadoop Fs-ls/5. Download the filePaths on Hadoop fs-get 6, execution jar: such as the implementation of the WordCount

1. Test the speed of Hadoop writesWrite data to the HDFs file system, 10 files, 10MB per file, files stored in/benchmarks/testdfsio/io_dataHadoop jar Share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.0-tests.jar Testdfsio-write-nrfiles 10- FileSize 10MB2. Test the speed of Hadoop read filesRead 10 files in t

Tags: linu lin Word use command dir OCA get ADO input Create a TXT file in the "/home/hadoop/" directory of your local Linux file system, where you can enter some words.mkdir hadoopcd Hadooptouch Test.txtgedit test.txt To view the file location locally (LS)Ls-al Displaying the contents of a file locallyCat Test.txt Use the command to upload "TXT" from the local file system to the input directory of the current user directory in HDFs.Cd/usr/l

> Major_compact ' R1 ', ' C1 '#Compact a single column family within a table:#hbase > major_compact ' t1 ', ' C1 'Configuration Management and Node restart1) Modify the HDFs configurationHDFs Configuration Location:/etc/hadoop/conf# Synchronous HDFS ConfigurationCat/home/hadoop/slaves|xargs-i-t scp/etc/hadoop/conf/hdfs-site.xml [email protected]{}:/etc/

Start the script under $hadoop_home/bin or $hadoop_home/sbin, the shell terminal will display the output information;Based on the output information, the information of Bash-x [Script-name] and the script itself can be used to locate where the error began.The following are the problems in these processes, and how they are addressed;1. No NodeManager to stop occurs when calling the Stop-yarn.sh script, closing ResourceManager and Nodemanger;But SSH to

entries for user, group, and other user and permission bitsComma-delimited list of -setfattr {-n name [-V value] |-x name} Sets the extended property name and value for a file or directory.-N Name Extended property name-V Value Extended property value-X Name Delete extended property-setrep [-R] [-W] To set the replication level for a file-W It requests the command to wait for replication to complete. This may take a long time.-R It is accepted for backward compatibility. It has no effect-stat [

1.hadoopView the directory on HDFs: hadoop fs-ls/ Create a directory on HDFs: -mkdir/jiatest upload the file to HDFs Specify directory: -put test.txt /Jiatest upload jar package to Hadoop run: hadoop jar maven_test-1.0-snapshot.jar org.jiahong.test.WordCount/ jiatest/jiatest/Output View result: -cat/jiatest/output/part-r-000002.linuxU

We take RHEL6.3 as an example to illustrate.There are command options at the end of the Linux command, and some options have option values. The options are preceded by a dash "-", with a space separating the commands, options, and option values. Some commands have no options and there are parameters. An option is a command built-in feature, which is a user-supplied content that conforms to the command forma

There are command options at the end of the Linux command, and some options have option values. The options are preceded by a dash "-", with a space separating the commands, options, and option values. Some commands have no options and there are parameters. An option is a command built-in feature, which is a user-supplied content that conforms to the command format.1.1.1. Command promptRight-click on the de

Hadoop Namenode-format formatted Distributed File systemstart-all.sh Start all Hadoop daemonsstop-all.sh Stop all Hadoop daemonsstart-mapred.sh Start the Map/reduce daemonstop-mapred.sh Stop Map/reduce DaemonStart-dfs.sh starting the HDFs daemonstop-mapred.sh Stop HDFs Daemonstart-balancer.sh HDFS data Block load BalancingFS in the following command can also be w

1.1)vim/etc/udev/rules.d/ --persistent-Net.rulesVI/etc/sysconfig/network-scripts/ifcfg-Eth0type=Ethernetuuid=57d4c2c9-9e9c-48f8-a654-8e5bdbadafb8onboot=yesnm_controlled=YesBootproto = staticDefroute=Yesipv4_failure_fatal=Yesipv6init=NoNAME="System eth0"HWADDR=xx: 0c: in: -: E6:ecipaddr =172.16.53.100PREFIX= -gateway=172.16.53.2Last_connect=1415175123dns1=172.16.53.2The virtual machine's network card is using the virtual network cardSave Exit X or Wq2)Vi/etc/sysconfig/networkNetworking=yesHostnam

From:http://www.2cto.com/database/201303/198460.htmlHadoop HDFs Common CommandsHadoop common commands:Hadoop FSView all commands supported by Hadoop HDFsHadoop fs–lslisting directory and file informationHadoop FS–LSRLoop lists directories, subdirectories, and file informationHadoop fs–put Test.txt/user/sunlightcsCopy the test.txt of the local file system to the/user/sunlightcs directory of the HDFs file sys

"---------------NOWFORMATHDFS------------" Hdfsnamenode -formatecho "---------------HDFSFORMATALREADY-------------" echo "----- ----------NOWSTARTHDFS--------------"start-dfs.shecho"---------------hdfs STARTALREADY--------------"echo"---------------NOWSTARTYARN system-------------"start-yarn.shecho"---------------YARNSYSTEMSTART already-------------"echo"---------------NOWCREATUSERDIRECTORY------ -------"hadoopfs-mkdir-p/user/yehom/dataecho"---------------USER diredctorycreatedalready----------

' People '8. Other operationsTo modify a table structure:First Deactivate user table (new version not available)Disable ' user 'Add two column families F1 and F2Alter ' people ', NAME = ' F1 'Alter ' user ', NAME = ' F2 'Enabling table enable ' user '# # #disable ' user ' (new version not available)Delete a column family:Alter ' user ', NAME = ' F1 ', METHOD = ' delete ' or alter ' user ', ' delete ' = ' F1 'Add Column Family F1 also delete column family F2Alter ' user ', {name = ' F1 '}, {name

THE-H option would format file sizes in a "human-readable" fashion (e.g 64.0m instead of 67108864)[Email protected] tmp]# hdfs dfs-du-h/user/root/zhu84.9 K /user/root/zhu/a1.png c12>54.5 K /user/root/zhu/a2.png[[email protected] tmp] -rm-r [-skiptrash] Uri [URI ...] -r:recursive version of Delete[Email protected] tmp]#HDFs Dfs-ls/user/root/zhufound1items-rw-r--r--3Root HDFs86908 .- the- Geneva to:Geneva/user/root/zhu/a1.png[[email protected] tmp]#HDF

Recently used a cluster HP1, because the maintenance of the cluster of people do not give force, the node always over a period of time dropped one or two. When Hadoop was restarted today, HDFs was in protected mode.decided to filter out all the inaccessible nodes in the slaves node, so I wrote a small script, which is recorded here, and it is convenient to use it directly.PS: Written in C shellThe code is as follows:#!/bin/cshif ($ #argv Incidentally,

, soHDFs has a high degree of fault tolerance.3. High data throughput HDFs uses a "one-time write, multiple read" This simple data consistency model, in HDFS , once a file has been created, written, closed, generally do not need to modify, such a simple consistency model, to improve throughput.4. Streaming data access HDFS has a large scale of data processing, applications need to access a large amount of information at a time, and these applications are generally batch processing, rather than

http://blog.csdn.net/pipisorry/article/details/44681081Little TricksDirect view of Linux shell commands and explanations in the command lineMans commandHelp commands for better use under Linux-cheat#pip Install cheat$cheat Tar[better use of Linux help command-cheat]Copy, paste on the command lineIn the Ubuntu Terminal window, the shortcut keys for copying and pas

‘ 4) manual Trigger major compaction #语法：#Compact all regions in a table:#hbase> major_compact ‘t1‘#Compact an entire region:#hbase> major_compact ‘r1‘#Compact a single column family within a region:#hbase> major_compact ‘r1‘, ‘c1‘#Compact a single column family within a table:#hbase> major_compact ‘t1‘, ‘c1‘ Configuration Management and node restart1) Modify the HDFs configurationHDFs Configuration Location:/etc/