Hadoop File System Shell

Overview:

The file system (FS) shell contains commands for various classes of-shell, directly interacting with Hadoop Distributed File System (HDFS), and support for other file systems, such as: Local file system fs,hftp Fs,s3 FS, and others. Calls to the FS shell:

Bin/hadoop FS <args>

All FS shell commands have URI paths as parameters, and the URI format is: Scheme://authority/path. In the case of HDFs, scheme is HDFS, and if it is a local file system, scheme is file.

Scheme and Authorty are optional, and if not specified, the default scheme is specified in the configuration file. An HDFs file or directory such as:/parent/child in HDFs, it is: Hdfs://parent/child or simply write/parent/child (but the precondition is that you configure the file to point to thehdfs:// Namenodehost).

Most of the FS shell commands, in fact, are compatible with Linux. The difference is that each command has a specific description, the error message is stderr through the error output, and the other information is output through stdout.

If HDFs is used, then HDFs DFS and Hadoop FS are similar.

Appendtofile

Use:Hadoop fs-appendtofile <localsrc> ... <dst>

Appends a local file system source file or multiple source files to the target file system, or it can read standard input data and append to the target file system.

hadoop fs -appendToFile localfile / user / hadoop / hadoopfile append local file localfile to the target file system / user / hadoop / hadoopfile
     hadoop fs -appendToFile localfile1 localfile2 / user / hadoop / hadoopfile Append local files localfile1 and 2 to the target file system / user / hadoop / hadoopfile
     hadoop fs -appendToFile localfile hdfs: //nn.example.com/hadoop/hadoopfile Append local file localfile to / user / hadoop / hadoopfile of the specified target file system
     hadoop fs -appendToFile-hdfs: //nn.example.com/hadoop/hadoopfile read data from standard input

return value:

Returns 0 succeeds, and 1 fails.

Cat

Use:hadoop fs-cat uri [uri ...]

Outputs the contents of the specified URI with the standard output stdout.

Example:

hadoop fs -cat hdfs://nn1.example.com/file1 hdfs://nn2.example.com/file2
hadoop fs -cat file:///file3 /user/hadoop/file4

return value:

Returns 0 succeeds, and 1 fails.

Checksum using:Hadoop fs-checksum uri[uri ...]

Returns the checksum information for the file.

Example:

[[email protected] bin]# hadoop fs -checksum file:///usr/apache/hadoop-2.4.1/bin/1.txt file:///usr/apache/hadoop-2.4.1/bin/2.txt 
file:///usr/apache/hadoop-2.4.1/bin/1.txt	NONE	
file:///usr/apache/hadoop-2.4.1/bin/2.txt	NONE	

[[email protected] bin]# hadoop fs -checksum hdfs://10.0.1.78:8020/test1 /apps/hive/install/hive-exec-0.13.1.jar
hdfs://10.0.1.78:8020/test1	MD5-of-0MD5-of-512CRC32C	0000020000000000000000008f708c9e4a4ec19d554a96c72b655932
/apps/hive/install/hive-exec-0.13.1.jar	MD5-of-0MD5-of-512CRC32C	00000200000000000000000060814965c82060d526756df03bf81383

Chgrp

Use:Hadoop fs-chgrp [-R] GROUP uri [uri ...]

Change the filegroup, the user must be the owner of the file, or a super administrator.

Parameters

• -R If the file is a directory, the command will take effect recursively. Similar to many Linux commands.

chmod

Use:Hadoop fs-chmod [-r] <mode[,mode] ... | octalmode> uri [uri ...]

Change the permissions of the file,the user must be the owner of the file, or a super administrator.

Parameters

• -R If the file is a directory, the command will take effect recursively. Similar to many Linux commands.

Chown

Use:Hadoop Fs-chown [-R] [Owner][:[group]] uri [URI]

Change the owner of the file. This is a lake tour

Options

• -R If the file is a directory, the command will take effect recursively. Similar to many Linux commands.

Copyfromlocal

Use:Hadoop fs-copyfromlocal <localsrc> URIs

From the local file system to the cluster, the resource file must be outside the local file, similar to theput command.

Parameters:

• -F Overwrite if the file exists.

Example:

[[email protected] bin] # cat 1.txt
1111
[[email protected] bin] # cat 2.txt
22222222
[[email protected] bin] # hadoop fs -copyFromLocal 1.txt /fish/1.txt // Copy the local file to the HDFS file / fish / 1.txt
[[email protected] bin] # hadoop fs -cat /fish/1.txt // View
1111
[[email protected] bin] # hadoop fs -copyFromLocal -f 2.txt /fish/1.txt // Add the -f parameter to overwrite the HDFS file / fish / 1.txt
[[email protected] bin] # hadoop fs -cat /fish/1.txt // View, it is the content of file 2.txt
22222222

Copytolocal

Use:Hadoop fs-copytolocal [-IGNORECRC] [-CRC] URI <localdst>

Copy the cluster files to the local file system. In addition to the destination path is limited to local files, and the Get command is similar. The-IGNORECRC option replicates the file that the CRC check failed. Use the-CRC option to copy files and CRC information.

Count

Use:Hadoop fs-count [-Q] [-h] [-v] <paths>

calculate the number of files and directories under the directory, the size of the case, the output items using-count are:Dir_count, File_count, Content_size, PATHNAME

The output items using-count-q are: QUOTA, Remaining_quata, Space_quota, Remaining_space_quota, Dir_count, File_count, Content_size, PATHNAME

-h option shows sizes in human readable format.

-V multiple displays a header row (hadoop-2.4.1 no parameters )

[[email protected] bin]# hadoop fs -count -h
-count: Illegal option -h
Usage: hadoop fs [generic options] -count [-q] <path> ...

Example:

• Hadoop fs-count Hdfs://nn1.example.com/file1 Hdfs://nn2.example.com/file2
• Hadoop fs-count-q Hdfs://nn1.example.com/file1
• Hadoop fs-count-q-H hdfs://nn1.example.com/file1
• HDFs dfs-count-q-h-v Hdfs://nn1.example.com/file1

return value:

Returns 0 succeeds, and 1 fails.

Cp

Use:Hadoop FS-CP [-f] [-P |-p[topax]] URI [uri ...] <dest>

From the source file to the target file, this command allows multiple sources, and the destination path must be a directory.

Parameters:

• -F Overwrite if the target path already exists.
• -P This selection preserves the properties of the file [TopX] (Timestamps (timestamp), ownership (ownership), permission (permission), ACL, xattr). If-p does not specify a parameter, then save timestamps (timestamp), ownership (ownership), permission (permission). If-PA is specified, the permission information is also preserved. Because ACLs are more advanced than permission (permissions). The-p option determines whether the original namespace information is preserved.

Example:

// HDFS file is copied to the local file system
[[email protected] bin] # hadoop fs -cp /apps/tez-0.4.1/tez-api-0.4.1-incubating.jar file: /// home
[[email protected] bin] # ll / home /
total 748
drwx ------ 2 azkaban hadoop 4096 Aug 21 2014 azkaban
drwx ------. 11 hadoop hadoop 4096 Jul 30 09:48 hadoop
drwxr-xr-x 4 root root 4096 Jul 22 12:40 root
-rw-r--r-- 1 root root 752262 Aug 7 14:59 tez-api-0.4.1-incubating.jar
// HDFS file is copied to HDFS
[[email protected] bin] # hadoop fs -cp /apps/tez-0.4.1/tez-api-0.4.1-incubating.jar / fish /
[[email protected] bin] # hadoop fs -ls / fish /
Found 4 items
-rw-r--r-- 3 root supergroup 894 2015-08-07 12:31 /fish/3.txt
-rw-r--r-- 3 fish supergroup 193 2015-04-27 20:35 /fish/multi.txt
drwxr-xr-x-hadoop supergroup 0 2015-04-28 10:32 / fish / out
-rw-r--r-- 3 root supergroup 752262 2015-08-07 15:00 /fish/tez-api-0.4.1-incubating.jar
// Copy the local file to the local file system
[[email protected] bin] # hadoop fs -cp file: ///home/tez-api-0.4.1-incubating.jar file: /// usr /

return value:

Returns 0 succeeds, and 1 fails.

Df

Use:Hadoop fs-df [-h] uri [uri ...]

Show free space

Parameters:

• -H Displays the size of the file in a more humane way (for example, using 67108864 instead of 64M)

Example:

[[email protected] bin] # hadoop fs -df -h / fish
Filesystem Size Used Available Use%
hdfs: // nns 1.3 T 360.5 G 848.8 G 26%
// hdfs: // nns file system, there is a total of 1.3T space in the cluster, 360G is used, and 848.8G is available, which is 26% used

Du

Use:Hadoop fs-du [-S] [-h] uri [uri ...]

Displays the size of the file and directory, and if it is a directory, the size of all the files in that directory, and if it is a single file, displays the file's small

Parameters:

• -S displays the overall size of all files in the specified directory, rather than displaying the size of the file or directory individually
• -H Displays the size of the file in a more humane way (for example, using 67108864 instead of 64M)

Example:

• Hadoop fs-du/user/hadoop/dir1/user/hadoop/file1 Hdfs://nn.example.com/user/hadoop/dir1

Return value: 0 succeeds, 1 fails.

Dus

Use:Hadoop fs-dus <args>

Displays a summary of the length of a file.

Note: This command has expired and is replaced withHadoop fs-du-s.

Expunge

Use:Hadoop fs-expunge

Empty the trash bin.

Find

(Official documents have, Hadoop2.4.1 does not exist this command)

Use:Hadoop fs-find <path> ... <expression>...

Finds all files that match the specified expression and applies the selected action to them. If no path is specified, the current working directory is found by default, and if no expression is specified, the default is:-print

Example:

Hadoop fs-find/-name Test-print

Return value: 0 succeeds, 1 fails.

Get

Use:Hadoop fs-get [-IGNORECRC] [-CRC] <src> <localdst>

Copying files to the local system, using the "-IGNORECRC" option to verify the failed files, using the "-CRC" option, you can copy the file and CRC information.

Example:

• Hadoop fs-get/user/hadoop/file LocalFile
• Hadoop fs-get hdfs://nn.example.com/user/hadoop/file LocalFile

Returns a 0 success and nonzero error.

Getfacl

Use:Hadoop fs-getfacl [-R] <path>

Displays the ACL (Access control list) information for a file or directory. If the directory has a default ACL, the default ACL is also displayed.

Parameters:

• -R: Recursively get ACLs for all files and directories
• path: List of files or directories

Example:

• Hadoop fs-getfacl/file
• Hadoop fs-getfacl-r/dir

Returns a 0 success and nonzero error.

Setfacl

Use:Hadoop fs-setfacl [-R] [-B |-k-m |-x <acl_spec> <path>] |[ --set <acl_spec> <path>]

Sets the control list for files and directories (ACLs).

Parameters:

• -B: Remove all extended ACL information except basic ACL information (owner, group, others) will be retained to ensure compatibility of permissions;
• -K: Remove the default ACL
• -r: Apply these actions recursively to all files or directories.
• -M: The ACL is modified, the new entry is added to the ACL, and the old rule is preserved;
• -x: Deletes the specified ACL entry. Other ACL entries are reserved.
• --set: Replace the current ACL rule completely, replacing it with the new ACL rule. The new ACL rules must contain user, group, and other permission information to ensure compatibility;
• acl_spec: comma-delimited list of ACLs
• Path: The file or directory path that needs to be modified.

Example:

• Hadoop fs-setfacl-m user:hadoop:rw-/file
• Hadoop fs-setfacl-x User:hadoop/file
• Hadoop fs-setfacl-b/file
• Hadoop fs-setfacl-k/dir
• Hadoop fs-setfacl--set user::rw-,user:hadoop:rw-,group::r--, other::r--/file
• Hadoop Fs-setfacl-r-M User:hadoop:r-x/dir
• Hadoop fs-setfacl-m Default:user:hadoop:r-x/dir

Return value: 0 for success, 1 for failure.

getfattr

(Official documents have, Hadoop2.4.1 does not exist this command)

Use:Hadoop fs-getfattr [-r]-N name |-d [-e en] <path>

Displays the name and value of the extended property of the file or directory, if one exists.

Parameters:

• -R: Recursively lists the properties of all files and directories in the directory.
• -N Name: Dump the extended property of the specified name.
• -D: Dumps all extended properties of the associated path
• -E encoding: Encodes the retrieved value. The available encoding methods (encoding) include text, Hex, and base64. Using text, the retrieved values are displayed as strings in double quotation marks, and the retrieved values are shown in the coded form of 0x or 0s, using hex or base64.
• path: Files and directories.

Example:

• Hadoop fs-getfattr-d/file
• Hadoop fs-getfattr-r-N user.myattr/dir

Returns a 0 success and nonzero error.

setfattr

(Official documents have, Hadoop2.4.1 does not exist this command)

Use:hadoop fs-setfattr-n Name [-V value] |-X name <path>

Set an extended property for a file or directory: Name and value.

Parameters:

• -B: Remove all extended ACL information, basic ACL information (owner, group, others) will be retained to ensure compatibility of permissions;
  
• -N Name: Extended property name
  
• -V Value: The Extended property value. There are three different encoding methods for the value. If the argument is enclosed in double quotation marks, the value is a string. If the parameter is preceded by 0x or 0X, then it is a 16-digit number. If the parameter is preceded by 0s or 0S, then it is encoded in base64.
• -X Name: Remove Extended Properties
• path: file or directory

Example:

• Hadoop fs-setfattr-n user.myattr-v Myvalue/file
• Hadoop fs-setfattr-n User.novalue/file
• Hadoop fs-setfattr-x User.myattr/file

Return value: Returns 0 for success, 1 fails Getmerge

Use:hadoop fs-getmerge <src> <localdst> [ADDNL]

Takes a source directory and a local destination file as input, and connects all the files in the source directory to the local destination file. ADDNL is optional and is used to add a line break at the end of each file.

// Merge the files in the / fish folder of HDFS to the local /home/4.txt file.
hadoop fs -getmerge / fish fish: ///home/4.txt

Help

Use:Hadoop fs-help

Return to usage instructions

Ls

Use:Hadoop fs-ls [-d] [-h] [-r] [-T] [-S] [-r] [-u] <args>

Parameters:

• -D: listed like regular files.
  
• -H: Displays the file size in a more intuitive way (for example, 64.0m instead of 67108864).
• -r: Recursive subdirectory list.
• -T: Sort by modified time (last row first)
• -S: Sort by output file size
• -R: Reverse Sort
• -U: Time to use access time instead of display and sort.

The LS command returns the status of the file:

Permissions (permissions) Number_of_replicas (number of replicas) userid (User ID) GroupID (group ID) filesize (file size) modification_date (modified date) Modification_time (modification time) filename (filename)

For the table of contents, LS returns the first level of content in the directory, shown in the following format

Permissions (permissions) userid (User ID) GroupID (group ID) modification_date (modified date) Modification_time (modified) dirname (directory name)

Files in a directory are sorted by file name by default

Example:

• Hadoop fs-ls/user/hadoop/file1

Return value: Return 0 for success, 1 to fail

LSR

Use:Hadoop FS-LSR <args>

Recursive version of the LS command

Note: This tinnitus is out of date and can be used withHadoop fs-ls-r

Mkdir

Use:hadoop fs-mkdir [-p] <paths>

Create a directory with a URI as the path.

Parameters:

• -P is similar to Linux mkdir-p, where a parent path is created if the parent path does not exist.
  

Example:

[[email protected] bin]# hadoop fs -mkdir file:///home/a1 file:///home/a2
[[email protected] bin]# ll /home/
total 20
drwxr-xr-x   2 root    root   4096 Aug  8 09:45 a1
drwxr-xr-x   2 root    root   4096 Aug  8 09:45 a2

[[email protected] bin]# hadoop fs -mkdir /home/a1 /home/a2
[[email protected] bin]# hadoop fs -ls /home/
Found 2 items
drwxr-xr-x   - root          supergroup          0 2015-08-08 09:47 /home/a1
drwxr-xr-x   - root          supergroup          0 2015-08-08 09:47 /home/a2

Return value: Return 0 for success, 1 to fail

Movefromlocal

Use:hadoop fs-movefromlocal <localsrc> <dst>

In addition to the local path, it is deleted after being copied. Other similar to put commands

Movetolocal using:Hadoop fs-movetolocal [-CRC] <src> <dst>

It has not been implemented so far.

Mv

Use:hadoop fs-mv uri [uri ...] <dest>

Moving files from the source path to the target path (both files and directories can be moved), this command allows multiple sources, but the destination path must be a directory, and moving files across file systems is not allowed.

Example:

• Hadoop fs-mv/user/hadoop/file1/user/hadoop/file2
• Hadoop fs-mv hdfs://nn.example.com/file1 hdfs://nn.example.com/file2 hdfs://nn.example.com/file3 hdfs:// Nn.example.com/dir1

Return value: Return 0 for success, 1 to fail

Put

Use:Hadoop fs-put <localsrc> ... <dst>

Copying a single source from the local file system, or multiple sources to the target file system, can also be written to the target file system from standard input.

• Hadoop fs-put Localfile/user/hadoop/hadoopfile
• Hadoop fs-put Localfile1 Localfile2/user/hadoop/hadoopdir
• Hadoop fs-put LocalFile Hdfs://nn.example.com/hadoop/hadoopfile
• Hadoop fs-put-hdfs://nn.example.com/hadoop/hadoopfileReads the input from stdin.

Return value: Return 0 for success, 1 to fail

Rm

Use:Hadoop fs-rm [-F] [-R |-r] [-skiptrash] uri [uri ...]

Deletes the file specified by the parameter.

Parameters:

• -F forcibly removed. When performing the deletion, the analysis information of the file to be deleted is displayed, and the return information is not adjusted when the file does not exist;
• -R recursive removal of all contents under directory
  
• -R and-R as
• -skiptrash This parameter ignores the Recycle Bin and, if enabled, is completely deleted, which is useful when the directory size is huge. If this parameter is not used, the deleted file will be found in the Recycle Bin.

Example:

• Hadoop fs-rm Hdfs://nn.example.com/file/user/hadoop/emptydir

Return value: Return 0 for success, 1 to fail

RmDir

Using:Hadoop fs-rmdir [--ignore-fail-on-non-empty] uri [uri ...]

Delete Directory

Parameters:

• --ignore-fail-on-non-empty: When using wildcard characters, if a directory also contains files, the failure will not be deleted.
  

Example:

• Hadoop Fs-rmdir/user/hadoop/emptydir

RMRUse:Hadoop fs-rmr [-skiptrash] uri [uri ...]

Recursive deletion

Note: This command is out of date and can be replaced with thefs-rm-r command.

Setrep

Use:Hadoop fs-setrep [-R] [-W] <numReplicas> <path>

Change the replication factor for a file, and if path is a path, the command recursively modifies the replication factor for all files

Parameters:

• -W using this parameter, the command waits for the replication factor modification to complete, which can take a long time.
  
• -R backwards compatible, no actual effect.

Example:

[[email protected] bin] # hadoop fs -stat "% b% F% u:% g% o% y% n% r" /fish/1.txt
14 regular file root: supergroup 134217728 2015-08-07 07:50:15 1.txt 3 // The number of starting copies is 3
[[email protected] bin] # hadoop fs -setrep -w 4 /fish/1.txt // Modify the number of copies to 4
Replication 4 set: /fish/1.txt
Waiting for /fish/1.txt ............... ........................................ ........................................ ........................................ ........................................ ........................................ setrep: `/fish/1.txt ': No such file or directory
[[email protected] bin] # hadoop fs -stat "% b% F% u:% g% o% y% n% r" /fish/1.txt<pre name = "code" class = "java"> 14 regular file root: supergroup 134217728 2015-08-07 07:50:15 1.txt 4 // Number of ending copies is 4 

Return value: Return 0 for success, 1 to fail

Stat

Use:hadoop fs-stat [format] <path> ...

Prints the statistics in the specified format according to the file/directory. Accepted formats: all block file Size (%b), type (%F), owning Group (%g), name (%n), block size (%o), number of replicas (%r), owning user name (%u), and modified time (%y,%y),%y Show "Yyyy-mm-dd HH:mm:ss" The format of UTC time,%y displays the number of milliseconds from 1970 to the present. If no date format is specified, the default is%y

Example:

• Hadoop fs-stat "%F%u:%g%b%y%n"/file

Return value: Return 0 for success, 1 to fail

Tail

Use:Hadoop fs-tail [-f] URI

In standard output, the content of 1K at the end of the file is displayed

Parameters:

• -F file growth, output true and false content.

Example:

• Hadoop Fs-tail Pathname

Return value: Return 0 for success, 1 to fail

Test

Using:Hadoop fs-test-[defsz] URI

Parameters

• -D: If the path is a directory, then 0 is returned,
• -E: Returns 0 if the path exists.
• -F: Returns 0 if the path is a file.
• -S: returns 0 if the path is not empty
• -Z: Returns 0 if the file length is 0

Example:

• Hadoop fs-test-e filename

Text

Use:Hadoop fs-text <src>

Output the source file as text format. The allowed formats are zip and Textrecordinputstream.

Touchz

Use:hadoop fs-touchz uri [uri ...]

Create a file with a length of 0.

Example:

• Hadoop fs-touchz Pathname

Return value: Return 0 for success, 1 to fail

Usage

Usage:hadoop fs-usage Command

Returns help for a single command.

Example: Hadoop fs-usage du

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Hadoop File System Shell