Hadoop HDFS Command

Command basic format:

Hadoop Fs/dfs-cmd < args >

1.ls

Hadoop Fs-ls/

Lists directories and files under the root directory of the HDFs file system

Hadoop fs-ls-r/

List all directories and files of the HDFs file system

2.put

Hadoop fs-put < local file > < HDFs file >

The parent directory of the HDFs file must exist, otherwise the command will not execute

Hadoop fs-put < local file or dir >...< hdfs dir >

HDFs dir must exist, otherwise the command will not execute

Hadoop Fs-put-< HDSF file>

Read input from keyboard to HDFs file, press Ctrl+d to end input, HDFs file cannot exist, otherwise the command will not execute

2.1.moveFromLocal

Hadoop fs-movefromlocal < local src > ... < hdfs DST >

Similar to put, the source file local src is deleted after the command executes, and can be read from the keyboard to the HDFs file

2.2.copyFromLocal

Hadoop fs-copyfromlocal < local src > ... < hdfs DST >

Similar to put, you can also read input from the keyboard to the HDFs file

3.get

Hadoop fs-get < HDFs file > < local file or dir>

The local file cannot be the same as the HDFs file name, otherwise the file will be prompted to exist, and the file with no duplicate names will be copied to the local

Hadoop Fs-get < HDFs file or dir > ... < local dir >

When copying multiple files or directories locally to a folder path
Note: If the user is not root, the local path is the path under the user folder, or a permissions issue will occur.

3.1.moveToLocal

This command is not yet implemented in the current version

3.2.copyToLocal

Hadoop fs-copytolocal < local src > ... < hdfs DST >

Similar to get

4.rm

Hadoop fs-rm < HDFs file > ... hadoop fs-rm-r < HDFs dir> ....

You can delete multiple files or directories at a time

5.mkdir

Hadoop Fs-mkdir < HDFs path>

Only the first level of the building directory, the parent directory does not exist the use of this command will be an error

Hadoop Fs-mkdir-p < HDFs path>

The directory created if the parent directory does not exist, create the parent directory

6.getmerge

Hadoop fs-getmerge < HDFs dir > < local file >

After sorting all the files in the HDFs specified directory into a file specified by local, the file will be created automatically when it is not present, and the contents will be overwritten when the file is present.

Hadoop fs-getmerge-nl < HDFs dir > < local file >

When NL is added, a row is vacated between the HDFs files that are merged into the local file

7.cp

Hadoop FS-CP < HDFs file > < HDFs file >

The target file cannot exist, otherwise the command cannot be executed, which is equivalent to renaming and saving the file, the source file still exists
Hadoop FS-CP < HDFs file or dir, ... < HDFs dir >

Destination folder to exist, otherwise the command cannot be executed

8.mv

Hadoop FS-MV < HDFs file > < HDFs file >

The target file cannot exist, otherwise the command cannot be executed, which is equivalent to renaming and saving the file, the source file does not exist

Hadoop FS-MV < HDFs file or dir, ... < HDFs dir >

When there are multiple source paths, the destination path must be a directory and must exist.
Note: Moving across file systems (local to HDFs or vice versa) is not allowed

9.count

Hadoop fs-count < HDFs path >

Count the number of directories, number of files, total size of files in HDFs corresponding path
Display as number of directories, number of files, total file size, input path

10.du

Hadoop Fs-du < HDSF path>

Displays the size of each folder and file in the HDFs corresponding path

Hadoop Fs-du-s < HDSF path>

Displays the size of all files and sizes in the HDFS corresponding path

Hadoop Fs-du-h < HDSF path>

Displays the size of each folder and file under the HDFs path, and the size of the file is expressed in a convenient reading format, e.g. 64M instead of 67108864

11.text

Hadoop Fs-text < HDSF file>

Output text files or non-text files in some formats in text format

12.setrep

Hadoop fs-setrep-r 3 < HDFs path >

Change the number of copies of a file in HDFs, the number 3 in the above command is the number of copies set, and the-r option allows you to recursively change the number of copies of all directories + files in a directory

13.stat

Hdoop fs-stat [format] < HDFs path >

Returns the status information for the corresponding path
[format] Optional parameters are:%b (file size),%o (block size),%n (file name),%r (number of replicas),%y (last modified date and time)
This can be written in Hadoop fs-stat%b%o%n < HDFs path >, but not recommended, so that the results of each character output are not too easy to understand

14.tail

Hadoop fs-tail < HDFs file >

displaying 1KB data at the end of a file in standard output

15.archive

Hadoop Archive-archivename name.har-p < HDFs parent dir > < src >* < HDFs DST >

Command parameter name: ZIP file name, take;< hdfs parent dir >: The parent directory where the compressed file;< src >: file name to compress;< HDFs DST;: Compressed file storage path
* Example: Hadoop archive-archivename hadoop.har-p/user 1.txt 2.txt/des
The example compresses the file 1.txt,2.txt in the/user directory in HDFs into a file named Hadoop.har stored in the/des directory in HDFs, if 1.txt,2.txt does not write/ All directories and files in the user directory are compressed into a file named Hadoop.har, which is stored in the/des directory in HDFs
The contents of the Har can be displayed with the following command:

Hadoop Fs-ls/des/hadoop.jar

Show Har compressed is those files can use the following command

Hadoop fs-ls-r Har:///des/hadoop.har

Note:The Har file cannot be compressed two times. If you want to give a. har plus file, you can only find the original file and recreate it. The data of the original file in the Har file does not change, and the real role of the Har file is to reduce namenode and datanode excessive space waste.

16.balancer

HDFs Balancer

If an administrator discovers that some datanode hold too much data and some datanode save less data, you can start the internal equalization process manually using the above command

17.dfsadmin

HDFs Dfsadmin-help

Administrators can manage HDFs through dfsadmin, and usage can be viewed through the above command
HDFs Dfsadmin-report

Display the file system's basic data

HDFs Dfsadmin-safemode < Enter | Leave | Get | Wait >

Enter: Enter safe mode; leave: leave safe mode; get: Learn whether to turn on Safe mode;
Wait: Waiting to leave safe mode

18.distcp

Used to copy data between two HDFs

Hadoop HDFS Command