Shell operations for HDFS in Hadoop framework

Since HDFs is a distributed file system for accessing data, the operation of HDFs is the basic operation of the file system, such as file creation, modification, deletion, modification permissions, folder creation, deletion, renaming, etc. The operations command for HDFS is similar to the operation of the Linux shell on files such as LS, mkdir, RM, and so on.

When we do the following, make sure Hadoop is working correctly, and use the JPS command to ensure that each Hadoop process is seen.

We execute the commands shown in Hadoop fs,4-1.

Figure 4-1

A number of command options are shown in the figure. Not all of them, I have fully listed the supported command options in table 4-1.

Option name	Use formatting	Meaning
-ls	-ls < paths >	View the current directory structure for the specified path
-lsr	-LSR < paths >	Recursively view the directory structure of a specified path
-du	-du < paths >	Statistics directory next file size
-dus	-dus < paths >	Size of file (clip) in summary statistics directory
-count	-count [-Q] < paths >	Number of statistical files (clips)
-mv	-MV < Source path > < destination path >	Move
-cp	-CP < Source path > < destination path >	Copy
-rm	-RM [-skiptrash] < paths >	Delete files/blank folders
-rmr	-RMR [-skiptrash] < paths >	Recursive deletion
-put	-put < files on multiple Linux >	Uploading files
-copyfromlocal	-copyfromlocal < files on multiple Linux >	Copy from local
-movefromlocal	-movefromlocal < files on multiple Linux >	Move from local
-getmerge	-getmerge < Source path > <linux path >	Merge to Local
-cat	-cat	View File Contents
-text	-text	View File Contents
-copytolocal	-copytolocal [-IGNORECRC] [-CRC] [HDFs source path] [Linux destination Path]	Copy from local
-movetolocal	-movetolocal [-CRC]	Move from local
-mkdir	-mkdir	Create a blank folder
-setrep	-setrep [-R] [-W] < number of copies > < paths >	Modify the number of replicas
-touchz	-touchz < file paths >	Create a blank file
-stat	-stat [format] < paths >	displaying file statistics
-tail	-tail [-F] < files >	View File trailer Information
-chmod	-chmod [-R] < permission mode > [path]	Modify Permissions
-chown	-chown [-R] [genus Master][:[]] path	Modify Owner
-chgrp	-CHGRP [-R] Genus group name path	Modify Genus Group
-help	-help [Command Options]	Help

Note: The path in the table above includes the path in HDFs and the path in Linux. For areas prone to ambiguity, the "Linux path" or "HDFs path" is highlighted. If it is not explicitly stated, it means the HDFs path.

Here we describe the use of each command option.

-ls Show current directory structure

The command option represents the current directory structure for viewing the specified path, followed by the HDFs path, as shown in 4-1.

Figure 4-1

The path in is the HDFs root, and the content format displayed is very similar to the content format displayed by the Linux command Ls–l, which parses the content format of each line:

L The initial letter indicates the folder (if "D") or the file (if "-");

The 9-bit character behind the L represents the permission;

The number followed by or "-" indicates the number of copies. If it is a file, use a number to represent the number of copies;

The "root" in the back indicates the owner;

L "supergroup" in the back of the group;

The "0", "6176", and "37645" in the back of the file indicate the size of the files, in bytes;

L The time after the change, the format is month and day time;

The last entry represents the file path.

The visible root directory has four folders and two files.

If the command option is not followed by a path, the/user/< current user > directory is accessed. We log in using the root user, so we will access the/user/root directory of HDFs, as shown in 4-2.

Figure 4-2

If you do not have this directory/user/root, you will be prompted for the error that the file does not exist.

-LSR recursive display of directory structure

This command option indicates that the directory structure of the current path is recursively displayed, followed by the HDFs path. As shown in 4-3.

Figure 4-3

There is a root directory under the/user directory, and the root directory has the file ABC.

-du the size of each file in the statistics directory

The command option displays the file size under the specified path, in bytes, as shown in 4-4.

Figure 4-4

-dus file size under Summary statistics directory

This command option displays the file size of the specified path, in bytes, as shown in 4-5.

Figure 4-5

Please compare the differences between figure 4-4 and figure 4-5, and experience the different meanings of the two command options.

-count number of statistical files (clips)

This command option displays the number of folders, the number of files, and the total file size information under the specified path, as shown in 4-6.

Figure 4-6

In Figure 4-6 There are two commands, the following command is to support the correctness of the above command.

-MV Moving

This command option represents moving HDFs files into the specified HDFs directory. followed by two paths, the first represents the source file, the second represents the destination directory, and 4-7 shows.

Figure 4-7

There are three commands in Figure 4-7 to reflect the changes before and after the move.

-CP Copy

This command option indicates that the file specified by the HDFs is copied into the specified HDFs directory. followed by two paths, the first one is the copied file, the second is the destination, 4-8 is shown.

Figure 4-8

There are three commands in Figure 4-8 to reflect the changes before and after replication.

-rm Delete files/blank folders

This command option indicates the deletion of the specified file or an empty directory, as shown in 4-9.

Figure 4-9

In Figure 4-9, the first three commands are to reflect the changes before and after the execution. The fourth command is to delete the non-empty "/user/root" directory, the operation failed, indicating that the non-empty directory cannot be deleted.

-RMR Recursive deletion

This command option indicates that all subdirectories and files under the specified directory are deleted recursively, as shown in 4-10.

Figure 4-10

-put Uploading Files

This command option indicates that files on Linux are copied to HDFs, as shown in 4-11.

Figure 4-11

-copyfromlocal Copy from local

The operation is consistent with-put, no longer an example.

-movefromlocal move from local

This command indicates that the file is moved from Linux to HDFs, as shown in 4-12.

Figure 4-12

-getmerge Merge to local

The meaning of this command option is to merge all the file contents of the HDFs specified directory into a local Linux file, as shown in 4-13.

Figure 4-13

-cat View File Contents

The command option is to view the contents of the file as shown in 4-14.

Figure 4-14

-text View File Contents

The command option can be thought of as acting and using the same as-cat, here slightly.

-mkdir Create a blank folder

The command option indicates that the folder is created, followed by the folder in HDFs that will be created, as shown in 4-15.

Figure 4-15

-setrep set Number of replicas

The command option is to modify the number of copies of the saved file, followed by the number of copies, followed by the file path, 4-16.

Figure 4-16

In Figure 4-16, we modified the number of copies of the file/install.log, changed from 1 to 2, meaning that one more copy, HDFs will automatically perform the file copy work, generating a new copy.

If the last path represents a folder, then you need to follow option-R, which indicates that all files in the folder are modified by a copy, as shown in 4-17.

Figure 4-17

In Figure 4-17, we do the/user folder, using the option-R, then the number of copies of the file ABC under/user/root has changed.

Another option is-W, which indicates that the pending copy operation ends before exiting the command, as shown in 4-18.

Figure 4-18

Readers are asked to compare the results before and after using-Q in the above two graphs.

-touchz Create a blank file

The command option is to create a blank file in HDFs, as shown in 4-19.

Figure 4-19

-stat display statistics for a file

This command option displays some statistics about the file, as shown in 4-20.

Figure 4-20

In Figure 4-20, the command options can be formatted with quotation marks. The format "%b%n%o%r%Y" In the example shows file size, file name, block size, number of replicas, and access time in turn.

-tail viewing the contents of a file trailer

This command option displays the contents of the last 1 K bytes of the file. Typically used to view logs. If you have the option-F, the file contents are automatically displayed when the content changes, as shown in 4-21.

Figure 4-21

-chmod Modify File Permissions

The use of this command option is similar to the chmod usage in the Linux shell, with the effect of modifying the file's permissions, as shown in 4-22.

Figure 4-22

In Figure 4-22, the permissions for the file/emptyfile are modified.

If you add option-r, you can modify permissions on all files in the folder, as shown in 4-23.

Figure 4-23

-chown Modify Owner

This command option indicates the owner of the modified file, as shown in 4-24.

Figure 4-24

The owner of the file/emptyfile is changed from root to Itcast.

You can also modify the genus group at the same time, as shown in 4-25.

Figure 4-25

In Figure 4-25, the owner and group of the file/emptyfile are modified to itcast, and if only the genus group is modified, you can use ": Itcast".

With the option-r, you can recursively modify the owner and group information of all the files in the folder.

-chgrp Modify Genus Group

The purpose of this command is to modify the genus Group of the file, which is equivalent to the usage of "Chown: Genus", as shown in 4-26.

Figure 4-26

-help Help

The command option displays the Help information, followed by the command options that need to be queried, as shown in 4-27.

Figure 4-27

In Figure 4-27, the use of the RM is queried.

The command option displays content that is not completely accurate, such as the result of a query count is inaccurate, but instead displays the usage of all command options, as shown in 4-28. Hopefully the new version will be improved later.

Figure 4-28

For a short time, readers will not be able to master all command usage, please focus on the use of LS (r), RM (R), mkdir, put, get.

-----------------------------------------------------------------------------

SOURCE Link: https://wenku.baidu.com/view/3406704cf01dc281e53af0f8.html

Shell operations for HDFS in Hadoop framework