Javaapi of HDFs

Configure the Hadoop environment for the Windows platform

To do HDFS client app development on Windows, you need to set up a Hadoop environment and require a Windows platform-compiled Hadoop, or you will report the following error:

null\bin\winutils.exe in the Hadoop binaries.

Ref: 72896484

Create a MAVEN project that introduces Pom dependencies

<Dependencies>    <Dependency>        <groupId>Org.apache.hadoop</groupId>        <Artifactid>Hadoop-common</Artifactid>        <version>2.7.4</version>    </Dependency>    <Dependency>        <groupId>Org.apache.hadoop</groupId>        <Artifactid>Hadoop-hdfs</Artifactid>        <version>2.7.4</version>    </Dependency>    <Dependency>        <groupId>Org.apache.hadoop</groupId>        <Artifactid>Hadoop-client</Artifactid>        <version>2.7.4</version>    </Dependency></Dependencies>

Client Object

Working with HDFS in Java mainly involves the following Class:

Configuration: The object of this class is marshaled to the client or server;

FileSystem: The object of this class is a file system object, which can be manipulated by some methods of the object, and obtained by FileSystem's static method get.

FileSystem fs = filesystem.get (conf)

The Get method determines what type of file system is specific from the configuration value of a parameter Fs.defaultfs in Conf. If FS.DEFAULTFS is not specified in our code, and the project Classpath is not given a corresponding configuration, the default value in conf comes from Core-default.xml in the jar package of Hadoop, with the default value: file:///, is not an instance of Distributedfilesystem, but a client object for the local file system.

Sample code

1  Public classTesthdfs {2 3      Public Static voidMain (string[] args)throwsexception{4Configuration conf =NewConfiguration ();5         //specifies that the HDFs file system is used6 //conf.set ("Fs.defaultfs", "HDFs://node-1:9000 ");7 8         //filesystem is the core class of the Hadoop operational file system9         //get the file system client object by filesystem static methodTen //FileSystem fs = Filesystem.get (conf); One         //set uri,conf, user identity AFileSystem fs = Filesystem.get (NewURI ("hdfs://node-1:9000"), conf, "root"); -  -         //Create a folder the //Fs.mkdirs (New Path ("/createbyjava")); -         //Uploading Files - //Fs.copyfromlocalfile (New Path ("D:\\test.txt"), New Path ("/createbyjava/test.txt")); -         //Download File + //Fs.copytolocalfile (New Path ("/createbyjava/test.txt"), New Path ("D:\\test\\test.txt")); -  +         //stream form to read a local file AFileInputStream in =NewFileInputStream (NewFile ("D:\\test.txt")); at         //Create a file -Fsdataoutputstream out = Fs.create (NewPath ("/test.txt")); -         //stream in the form of uploading the local file to HDFs - ioutils.copy (in,out); -  - fs.close (); in     } -}

Javaapi of HDFs