hdfs--command-line interface detailed

now we'll go through the command line with HDFS interaction. HDFS also has many other interfaces, but the command line is the simplest and most familiar to many developers.

when we set up a pseudo-distribution configuration, there are two properties that need further explanation. First,Fs.default.name, set tohdfs://localhost/,used forHadoopsets the default file system. The file system is made up ofURIspecified, here we have used aHDFs URIto configureHDFSto beHadoopthe default file system. HDFSThe daemon will use this property to determineHDFSthe host and port of the name node. We will be inlocalhostrun on the default port of8020. this way,HDFSThis property allows the user to know where the name node is running to make it easier to connect to it.

a second property dfs.replication , we set the 1 , in this way, HDFS The file system block is not copied by default settings 3 A copy. When running on a single data node,HDFS cannot replicate the block to 3 data nodes, so there is insufficient copy of the persistent warning block. This setting resolves this issue.

Basic File System operations

the filesystem is ready and we can perform all other file system operations, such as reading files, creating directories, moving files, deleting data, listing index directories, and so on. Enter the hadoop fs-help command to see a detailed help file for all commands.

first copy a file from the local file system to HDFS :

1. % hadoopfs-copyfromlocal Input/docs/quangle.
TXT hdfs://localhost/user/tom/quangle.txt

the command callsHadoopof the file systemShellCommandFS, a series of subcommands are provided. Here, we are doing-copyfromlocal. Local fileQuangle.txtis copied to run on thelocalhoston theHDFSin the entity/user/tom/quangle.txtfile. We can actually omitURIformat with the host and select default settings, i.e. omitHdfs://localhost,just likeCore-site.xmlas specified in the.

1. % Hadoop fs-copyfromlocal input/docs/quangle.
Txt/user/tom/quangle.txt

you can also use relative paths and copy files to Home directory, which is /user/tom :

1. % Hadoop fs-copyfromlocal input/docs/quangle.txt quangle.txt

We copy the files back to the local filesystem to see if they are the same:

1. % Hadoop fs-copytolocal quangle.txt quangle.copy.txt

2. % MD5 input/docs/quangle.txt quangle.copy.txt

3. MD5 (input/docs/quangle.txt) = A16f231da6b05e2ba7a339320e7dacd9

4. MD5 (quangle.copy.txt) = A16f231da6b05e2ba7a339320e7dacd9

MD5 The analysis results are the same, indicating that the file HDFS the journey to survive and complete.

Finally, let's look at HDFS file list. Let's create a table of contents to see how it appears in the list:

1. % Hadoop fs-mkdir Books

2. % Hadoop fs-ls.

3. Found 2 Items

  drwxr-xr-x    - tom supergroup          0 
2009-04-02  22:41 /user/tom/books  

5. -rw-r--r--1 Tom SuperGroup 118
2009-04-02 22:29/user/tom/quangle.txt

The returned information results are associated withUnixCommandls-lthe output is very similar and only slightly different. The first column shows the file format. The second column is the number of copies of this file(this inUnixthe file system is not.). Because the default number of copies we set is within the site scope for1, so it's all shown here.1. The beginning of this column is empty because the concept of a copy is not applied-- The directory is the metadata and exists in the name node, not the data node. The third and fourth columns show the users and groups to which the files belong. The fifth column is the size of the file, shown in bytes, with a directory size of0. Columns sixth and seventh are the date and time when the file was last modified. The last eighth column is the absolute path to the file or directory.

HDFS the file license in

HDFS for files and directories with POSIX a very similar licensing model.

(W) and execute license (x) hdfs ( and posix different )

Each file and directory has a user, group, and mode of ownership. This mode is comprised of the license of the owning user, the license of the other members of the group, and the license of other users.

username ( name ) and groups ( group ( dfs.permissions

If a license check is enabled, both the user license and the group license are checked to confirm that the user's user name is the same as the user's license and that he or she is a member of this user group, and if not, check for additional licenses.

Here is the concept of a super user, which is the identity of the name node process. The system does not perform any license checks for super users.

hdfs--command-line interface detailed