Shell command interface of storage system based on Key/value+hadoop HDFS design

&http://www.aliyun.com/zixun/aggregation/37954.html ">nbsp; Parse all of the commands in the Hadoop HDFS (where the operation process is their own idea and disagree.)

Interface Name

function

Operation Process

Get

Copy the file to the local file system. If more than one source file is specified, the local destination must be a directory.

(1) According to the above mechanism, read K in the storage Engine on config server, until the filename (or large file metadata information) is obtained;

(2) According to the filename (or large file metadata information) to the corresponding data server to obtain the corresponding file (large files need stitching);

(3) writes the obtained file to the local file system.

Put

Copies files or directories from the local file system to the destination file system.

(1) To get the key/value data of the target folder by layer parsing <dst> absolute path according to the above chapters;

(2) According to the rules mentioned in the chapter, the corresponding key/value are generated by combining the label of the target folder and the filename of the source file;

(3) According to the steps (2) generated key/value upload source files related information;

(4) Upload the source file contents to the system.

Rm

Deletes the specified file and deletes only non-empty directories and files.

(1) To obtain the key/value of the specified file at the location of the target file, as described in the above section;

(2) Locate the information stored on the datanode of the specified file or directory according to key;

(3) Delete the information;

(4) Write the delete record to edit log.

mkdir

Creating a directory creates all the missing parent directories in the path.

(1) According to the above mechanism, resolve to find the parent directory, get the tag, if the parent directory does not exist the directory on the creation path recursively;

(2) Create a new directory by making the tag of the parent directory The parent node field section of the key.

Appendtofile

Add source files from the local file system to the destination file system.

(1) To get the key/value data of the target folder by layer parsing <dst> absolute path according to the above chapters;

(2) According to the rules mentioned in the chapter, the corresponding key/value are generated by combining the label of the target folder and the filename of the source file;

(3) According to the steps (2) generated key/value upload source files related information;

(4) Upload the source file contents to the system.

Cat

The path specifies the contents of the file to output to stdout.

(1) According to the above mechanism, parsing path to the target file, obtaining the key/value of the file;

(2) According to the key value in the found file content output to stdout;

(3) Display the result in the terminal.

Chgrp

Change the group of files and directories.

(1) According to the above mechanism, read K in the storage Engine on config server, until the file name/directory (or large file metadata information) is being manipulated;

(2) Modify the value of this k to represent the field of the group and reinsert the new K.

chmod

Change access rights to files and directories.

(1) According to the above mechanism, read K in the storage Engine on config server, until the file name/directory (or large file metadata information) is being manipulated;

(2) Modify the field that represents the access rights of value in this k and reinsert the new K.

Chown

Change the owner of the file and directory.

(1) According to the above mechanism, read K in the storage Engine on config server, until the file name/directory (or large file metadata information) is being manipulated;

(2) Modify the field representing the owner of value in this k and reinsert the new K.

Copyfromlocal

Copying files from the local file system is similar to the put command.

The operation process is shown in the put interface.

Coptolocal

Copy the file to the local file system, similar to the Get command.

The operation process is shown in the Get interface.

Count

Displays the number of subdirectories determined by the path, the number of files, the number of bytes used, and all file directory names.

(1) Resolving to the target path and obtaining the relevant key/value;

(2) Gets the number of subdirectories in value, the number of files, and the fields using related information such as strings.

cp

Copies the file from the source path to the target path. This command allows multiple original paths, at which point the target path must be a directory.

(1) According to the above section, parsing <dest>, obtaining the location information of the target file;

(2) Obtaining the contents of the source file;

(3) Copy the source file information to the target file block by piece.

Du

Displays the file size, and if path is a directory, displays the size of all files in that directory.

(1) According to the above mechanism, read K in the storage Engine on config server, until the filename or directory is obtained;

(2) If the file reads the corresponding Key/value, reads the field representing the file size in value, reads the tag if it is a directory, and the recursive query belongs to all the files in the directory.

Dus

Displays the size of the file, but when used as a directory, the file size is displayed.

(1) According to the above mechanism, read K in the storage Engine on config server, until the filename or directory is obtained;

(2) If the file reads the corresponding Key/value, reads the field representing the file size in the value, reads the tag in the directory, and the recursive query belongs to all the file value information in the directory, adding the field information about the file size in value to the sum.

Expunge

Empty the Recycle Bin.

(1) According to the above mechanism, resolves to the HDFS system Recycle Bin directory (/user/username/. Trash/current);

(2) Delete (Reference interface RM).

Getfacl

Displays access control lists for files and directories.

(1) To obtain the Key/value data of the target file or path according to the above mentioned sections;

(2) Gets the field information about the ACL in value.

Getmerge

Gets all the files specified by the source directory, merges them into a single file, and writes to the LOCALDST in the local file system.

(1) According to the above mechanism, the level of analysis, to obtain the source directory file tags;

(2) Looking for a subdirectory under the directory file in config server according to the tag;

(3) Create the target file and merge the contents of the child files into the target file;

(4) Write the target file to the local file system.

ls

Lists directory and file information.

(1) According to the above mechanism, each layer resolves to obtain the mark of the target file or folder;

(2) Query subdirectories or file related information according to the Mark;

(3) Listing information.

Lsr

Loops list directories, subdirectories, and file information.

Operation flow See reference LS interface.

Movefromlocal

Similar to put, only local sources are deleted after replication.

(1) To get the key/value data of the target folder by layer parsing <dst> absolute path according to the above chapters;

(2) According to the rules mentioned in the chapter, the corresponding key/value are generated by combining the label of the target folder and the filename of the source file;

(3) According to the steps (2) generated key/value upload source files related information;

(4) Uploading the source file contents to the system;

(5) Delete the local source file.

Movetolocal

Similar to get command.

The operation process is shown in the Get interface.

mv

Move the file from the source address to the destination address.

(1) According to the above mechanism, the label of the target folder is obtained by layer parsing <dest> absolute path;

(2) Layer to source file to obtain Key/value data;

(3) Changing the key value in the source file to change the parent node label field to the label of the destination folder.

RMr

Recursively deletes the specified files and directories.

(1) According to the above mechanism, parse to the location of the target file to get the mark of the specified file;

(2) using recursive algorithm to delete the specified file or directory according to the tag.

Setfacl

Sets the files and directories for access control lists (ACLs).

(1) According to the above mechanism, the key/value data of the target file or path is obtained by layer analysis;

(2) Set the ACL in value.

Setrep

Change the target copy factor of the file.

(1) To obtain the Key/value data of the target file according to the above mentioned sections;

(2) Modify the field in value for the copy factor.

Stat

Displays statistics in the file.

(1) To obtain the Key/value data of the target file according to the above mentioned sections;

(2) Gets the related field of the statistics about the file in value.

Tail

Displays the last 1KB data in the destination file.

(1) To obtain the Key/value data of the target file according to the above mentioned sections;

(2) According to the resulting Key/value query the last piece of 1KB information, if the last piece of less than 1KB, then query the penultimate block of information, splicing into 1KB. Returns information.

Test

Check the path.

(1) According to the above section, the target path is analyzed by layer, obtaining the key/value data of the target file;

(2) Check the relevant fields in the Key/value in the target file to detect whether the file exists, whether the size is 0, and whether the directory path exists.

Text

Displays the text content of the file, which is equivalent to cat when the file is a text file. When a file is in a compressed format, it is decompressed first.

Operation flow see Cat interface.

Touchz

Create a file of length 0.

The operation process is shown in the MkDir interface.

Original link: http://demo.netfoucs.com/captain_wl9083/article/details/37663327