HDFS-how to copy files from HDFS to a local device

The following two commands download files from HDFS to a local machine.

Get

Usage: Hadoop fs-get [-ignorecrc] [-crc]

Copy the file to the local file system. You can use the-ignorecrc option to copy files that failed CRC verification. Use the-crc option to copy the file and CRC information.

Example:

Hadoop fs-get/user/hadoop/file localfile

Hadoop fs-get hdfs: // host: port/user/hadoop/file localfile

Return Value:

0 is returned for success, and-1 is returned for failure.

CopyToLocal

Usage: hadoop fs-copyToLocal [-ignorecrc] [-crc] URI

Except that the target path is a local file, it is similar to the get command.

You can also use the following program to download files on HDFS to a local device.

Import java.net. URI;
Import java. io. OutputStream;
Import java. io. BufferedInputStream;
Import java. io. FileInputStream;
Import java. io. FileOutputStream;
Import org. apache. hadoop. conf. Configuration;
Import org. apache. hadoop. fs. FileSystem;
Import org. apache. hadoop. fs. FSDataInputStream;
Import org. apache. hadoop. fs. Path;
Import org. apache. hadoop. io. IOUtils;
Public class FileCopy2Local
{
Public static void main (String [] args) throws Exception
{
String dest = "hdfs: // localhost: 9000/user/laozhao0/cite2.txt ";
String local = "/home/laozhao0/cite2.txt ";
Configuration conf = new Configuration ();
FileSystem fs = FileSystem. get (URI. create (dest), conf );
FSDataInputStream fsdi = fs. open (new Path (dest ));
OutputStream output = new FileOutputStream (local );
IOUtils. copyBytes (fsdi, output, 4096, true );
}
}

Copy local files to HDFS

Download files from HDFS to local

Upload local files to HDFS

Common commands for HDFS basic files

Introduction to HDFS and MapReduce nodes in Hadoop

Hadoop practice Chinese version + English version + Source Code [PDF]

Hadoop: The Definitive Guide (PDF]