Hadoop (v): Shell command

HDFs DFS

• -cat URI: Viewing the contents of a file

1. 1. HDFs dfs-cat HDFs Dfs-cat hdfs://mycluster/user/root/rcc1
   2. HDFs dfs-cat file:///usr/hdp/2.4.2.0-258/hadoop/bin/RCC

• -ls <args>: Check file list

1. return Format: Permissions number_of_replicas userid groupid filesize modification_ Date modification_time filename

   [Email protected] ~]#HDFs Dfs-lsfound4itemsdrwx-------Root HDFs0  .- the- on  +:xx . Trashdrwxr-xr-x-Root HDFs0  .- ,- A  A: in. hivejars-rw-r--r--3Root HDFs531  .- the- on  A: -mapred-rw-r--r--3Root HDFs1857  .- the- on  A: - RCC1

• -mkdir <paths>: Creating a Directory
1. The behavior is much like Unix mkdir-p Creating parent directories along the path

   [Email protected] ~]#HDFs Dfs-lsfound4itemsdrwx-------Root HDFs0  .- the- on  +:xx . Trashdrwxr-xr-x-Root HDFs0  .- ,- A  A: in. hivejars-rw-r--r--3Root HDFs531  .- the- on  A: -mapred-rw-r--r--3Root HDFs1857  .- the- on  A: -Rcc1[[email protected] ~]# HDFs Dfs-mkdir HDFs://Mycluster/user/root/zhu[Email protected] ~]#HDFs Dfs-lsfound5itemsdrwx-------Root HDFs0  .- the- on  +:xx . Trashdrwxr-xr-x-Root HDFs0  .- ,- A  A: in. hivejars-rw-r--r--3Root HDFs531  .- the- on  A: -mapred-rw-r--r--3Root HDFs1857  .- the- on  A: -rcc1drwxr-xr-x-Root HDFs0  .- the- Geneva Geneva: *Zhu

• -copyfromlocal <localsrc> URI: Upload file, similar to put
• -put <localsrc> ... <dst>

[Email protected] ~]#HDFs Dfs-ls/user/root/zhu[[email protected] ~]#HDFs Dfs-put/usr/zhu/a1.png/user/root/zhu[[email protected] ~]#HDFs dfs-copyformlocal/usr/zhu/a2.png/user/root/zhu-copyformlocal:unknown Command[[email protected] ~]#HDFs Dfs-copyfromlocal/usr/zhu/a2.png/user/root/zhu[[email protected] ~]#HDFs Dfs-ls/user/root/zhufound2items-rw-r--r--3Root HDFs86908  .- the- Geneva Geneva: -/user/root/zhu/a1.png-rw-r--r--3Root HDFs55823  .- the- Geneva Geneva: the/user/root/zhu/a2.png[[email protected] ~]#

• -copytolocal [-IGNORECRC] [-CRC] URI <localdst>: Download to Local
• -get [-IGNORECRC] [-CRC] <src> <localdst>

  [Email protected] ~]#Clear[[email protected] ~]#HDFs Dfs-ls/user/root/zhufound2items-rw-r--r--3Root HDFs86908  .- the- Geneva Geneva: -/user/root/zhu/a1.png-rw-r--r--3Root HDFs55823  .- the- Geneva Geneva: the/user/root/zhu/a2.png[[email protected] ~]# HDFs dfs-copytolocal HDFs://mycluster/user/root/zhu/a1.png/usr/zhu/tmp[Email protected] ~]#HDFs Dfs-get/usr/root/zhu/a2.png/usr/zhu/tmpget:'/usr/root/zhu/a2.png': No Such file or directory[Email protected] ~]#HDFs Dfs-get/user/root/zhu/a2.png/usr/zhu/tmp[[email protected] ~]#Cd/usr/zhu/tmp[[email protected] tmp]#Lltotal144-rw-r--r--1Root root86908Sep2 Geneva: Wua1.png-rw-r--r--1Root root55823Sep2 Geneva: -A2.png[[email protected] tmp]#

• -count [-Q] <paths>: Statistics
2. the output Columns With-count are:dir_count, file_ COUNT , content_size file_name
4. The output columns With-q Are:quota, Remaining_quata, Space_quota, Remaining_space_quota

   [[email protected] tmp]# hdfs dfs-count/user/root/zhu           1            2             142731 /user/root/zhu[[email protected] tmp]# HDFs dfs-count-q/user/root/zhu        none             inf            none< C14/>inf            1            2             142731  /user/root/zhu[[email protected] TMP]

• -du [-S] [-h] Uri [uri ...]: Statistic size

1. THE-S option would result in an aggregate summary of file lengths being displayed, rather than the individual file
2. THE-H option would format file sizes in a "human-readable" fashion (e.g 64.0m instead of 67108864)

   [Email protected] tmp]# hdfs dfs-du-h/user/root/zhu84.9  K  /user/root/zhu/a1.png c12>54.5  K  /user/root/zhu/a2.png[[email protected] tmp]

• -rm-r [-skiptrash] Uri [URI ...]

1. -r:recursive version of Delete

   [Email protected] tmp]#HDFs Dfs-ls/user/root/zhufound1items-rw-r--r--3Root HDFs86908  .- the- Geneva  to:Geneva/user/root/zhu/a1.png[[email protected] tmp]#HDFs dfs-rm-r/user/root/zhu -/ the/ Geneva  to:Geneva: inINFO FS. Trashpolicydefault:namenode Trash Configuration:deletion interval = theMinutes, emptier interval =0minutes. Moved:'Hdfs://mycluster/user/root/zhu'  toTrash At:hdfs://mycluster/user/root/. Trash/current[Email protected] tmp]#HDFs dfs-ls/user/root/zhuls:'/user/root/zhu': No Such file or directory[Email protected] tmp]#

• -touchz Uri [URI ...]

1. Create A file of zero length

   [Email protected] tmp]#HDFs Dfs-lsfound4itemsdrwx-------Root HDFs0  .- the- Geneva  to: Geneva . Trashdrwxr-xr-x-Root HDFs0  .- ,- A  A: in. hivejars-rw-r--r--3Root HDFs531  .- the- on  A: -mapred-rw-r--r--3Root HDFs1857  .- the- on  A: -Rcc1[[email protected] tmp]#HDFs dfs-touchz a.txt[[email protected] tmp]#HDFs Dfs-lsfound5itemsdrwx-------Root HDFs0  .- the- Geneva  to: Geneva . Trashdrwxr-xr-x-Root HDFs0  .- ,- A  A: in. hivejars-rw-r--r--3Root HDFs0  .- the- Geneva  to: ,a.txt-rw-r--r--3Root HDFs531  .- the- on  A: -mapred-rw-r--r--3Root HDFs1857  .- the- on  A: -Rcc1[[email protected] tmp]#HDFs Dfs-cat/user/root/a.txt[[email protected] tmp]#

Hadoop (v): Shell command