Using the command line to manage files on hdfs--reprint

Original address: http://zh.hortonworks.com/hadoop-tutorial/using-commandline-manage-files-hdfs/

In this tutorial we'll walk through some of the basic HDFS commands you'll need to manage files on HDFS. To the tutorial you'll need a working HDP cluster. The easiest to has a Hadoop cluster is to download the Hortonworks Sandbox.

Let ' s get started.

Step 1:let ' s create a directory in HDFS, upload a file and list.

Let's look at the syntax first:

Hadoop Fs-mkdir:

• It'll take path URI ' as argument and creates directory or directories.

    Usage:Hadoop FS-Mkdir<paths> Example:Hadoop FS-mkdir  /user//dir1  /user/hadoop/dir2 Hadoop FS Span class= "pun" >-mkdir hdfs ://nn1.example.com/user/hadoop/dir      

Hadoop Fs-ls:

• Lists the contents of a directory
• For a file returns stats of a file

    Usage: Hadoop Fs -Ls <Args> Example: Hadoop Fs -Ls /User/Hadoop/Dir1 /User/Hadoop/Dir2 Hadoop Fs -Ls /User/Hadoop/Dir1/FileNametxt hadoop fs  -ls hdfs< Span class= "p" >://< Hostname>: 9000/user /hadoop/< Span class= "com" >dir1/        

Let's use the following commands as follows and execute. You can ssh into the sandbox using Tools like Putty. You could download putty.exe from the Internet.

Let ' s touch a file locally.

$ touch filename.txt

Step 2:now, let's check how to find out space utilization in a HDFS dir.hadoop fs-du:

• Displays sizes of files and directories contained in the given directory or the size of a file if its just a file.

    Usage:Hadoop FS-Du URIexample: Hadoop FS Span class= "pun" >-du / user///user /hadoop/dir1/sample .txt                

Step 4:

Now let's see how to upload and download files from and to Hadoop Data File System (HDFS)
Upload: (We have already tried this earlier)

Hadoop fs-put:

• Copy single src file, or multiple src files from local file system to the Hadoop data file system

    Usage:Hadoop FS-Put<localsrc> ... <Hdfs_dest_path> Example:Hadoop FS-Put/Home/ec2-user/samplefile.txt ./ambari .repo /user /hadoop /dir3/      

Download:
Hadoop Fs-get:

• Copies/downloads files to the local file system

    Usage:Hadoop FS-Get  <localdst> example: Hadoop FS Span class= "pun" >-get/user /hadoop//samplefiletxt //           

Step 5:let ' s look at quickly, Features.hadoop Fs-getmerge

• Takes a source directory files as input and concatenates files in src to the destination local file.

    Usage:Hadoop FS-Getmerge<src> <localdst> [Addnl] Example:Hadoop FS-Getmerge/User/hadoop/dir1 / ./ samplefile2. txt option:  Addnl:set to enable adding a newline on end each file      

Hadoop distcp:

• Copy file or directories recursively
• It's a tool used for large inter/intra-cluster copying
• It uses MapReduce to effect its distribution copy, error handling and recovery, and reporting

 usage :  Hadoop distcp <srcurl>  <desturl> example:  Hadoop distcp hdfs:// <namenode1>:8020/user/hadoop/dir1/\  Hdfs ://<namenode2>:8020/user/hadoop/dir2/  

You could use the following steps to perform Getmerge and DISCP.
Let's upload both files for this exercise first:

# Touch Txt1 Txt2# Hadoop Fs -Put Txt1 Txt2 /User/hadoop/< span class= "n" >dir2/# hadoop fs -ls /user /hadoop/< Span class= "com" >dir2/        

Step 6:getting Help

You can use the Help command to get list of commands supported by Hadoop Data File System (HDFS)

    Example:hadoopFS-     Help

Hope This short tutorial is useful to get the basics of file management.

Using the command line to manage files on hdfs--reprint