HDFS API Detailed-very old version

Due to the recent need to make a network disk system, so the collection.

About the file operation classes are basically all in the "Org.apache.hadoop.fs" package, these APIs can support operations include: open files, read and write files, delete files and so on.

The ultimate user-supplied interface class in the Hadoop class library is filesystem, which is an abstract class that can only be obtained by getting the class's Get method. The Get method has several overloaded versions, which are commonly used:

Static FileSystem Get (Configuration conf);

This class encapsulates almost all of the file operations, such as Mkdir,delete. The library framework for manipulating files can be basically derived:

Operator ()

{

Get the Configuration object

Get FileSystem Object

Working with files

}

6.1 Uploading Local files

With Filesystem.copyfromlocalfile (Path src,patch DST), local files are uploaded to the location where HDFs is made, where SRC and DST are the full paths to the files. Specific examples are as follows:

Package com.hebut.file;

Import org.apache.hadoop.conf.Configuration;

Import Org.apache.hadoop.fs.FileStatus;

Import Org.apache.hadoop.fs.FileSystem;

Import Org.apache.hadoop.fs.Path;

public class CopyFile {

public static void Main (string[] args) throws Exception {

Configuration conf=new configuration ();

FileSystem hdfs=filesystem.get (conf);

Local file

Path src =new path ("D:\\hebutwinos");

Until HDFs

Path DST =new Path ("/");

Hdfs.copyfromlocalfile (SRC, DST);

System.out.println ("Upload to" +conf.get ("Fs.default.name"));

Filestatus Files[]=hdfs.liststatus (DST);

for (Filestatus file:files) {

System.out.println (File.getpath ());

}

}

}

The results can be viewed from the console, project browser, and SECURECRT, as shown in 6-1-1, figure 6-1-2, figure 6-1-3.

1) Console Results

Figure 6-1-1 Running results (1)

2) Project Browser

Figure 6-1-2 Running Results (2)

3) SECURECRT Results

Figure 6-1-3 Running Results (3)

6.2 Creating an HDFs file

You can create a file on HDFs with "filesystem.create (path F)", where F is the full path to the file. The specific implementation is as follows:

Package com.hebut.file;

Import org.apache.hadoop.conf.Configuration;

Import Org.apache.hadoop.fs.FSDataOutputStream;

Import Org.apache.hadoop.fs.FileSystem;

Import Org.apache.hadoop.fs.Path;

public class CreateFile {

public static void Main (string[] args) throws Exception {

Configuration conf=new configuration ();

FileSystem hdfs=filesystem.get (conf);

byte[] buff= "Hello Hadoop world!\n". GetBytes ();

Path dfs=new path ("/test");

Fsdataoutputstream outputstream=hdfs.create (DFS);

Outputstream.write (buff,0,buff.length);

}

}

The running results are shown in 6-2-1 and figure 6-2-2.

1) Project Browser

Figure 6-2-1 Running results (1)

2) SECURECRT Results

Figure 6-2-2 Running Results (2)

6.3 Creating an HDFs directory

You can create a folder on HDFs with "filesystem.mkdirs (path F)", where F is the full path to the folder. The specific implementation is as follows:

Package com.hebut.dir;

Import org.apache.hadoop.conf.Configuration;

Import Org.apache.hadoop.fs.FileSystem;

Import Org.apache.hadoop.fs.Path;

public class Createdir {

public static void Main (string[] args) throws exception{

Configuration conf=new configuration ();

FileSystem hdfs=filesystem.get (conf);

Path dfs=new path ("/testdir");

Hdfs.mkdirs (DFS);

}

}

The running results are shown in 6-3-1 and figure 6-3-2.

1) Project Browser

Figure 6-3-1 Running results (1)

2) SECURECRT Results

Figure 6-3-2 Running Results (2)

6.4 Renaming an HDFs file

The specified HDFs file can be renamed with Filesystem.rename (path Src,path DST), where SRC and DST are the full paths to the file. The specific implementation is as follows:

Package com.hebut.file;

Import org.apache.hadoop.conf.Configuration;

Import Org.apache.hadoop.fs.FileSystem;

Import Org.apache.hadoop.fs.Path;

public class rename{

public static void Main (string[] args) throws Exception {

Configuration conf=new configuration ();

FileSystem hdfs=filesystem.get (conf);

Path frpaht=new path ("/test"); The old file name

Path topath=new path ("/test1"); The new file name

Boolean Isrename=hdfs.rename (Frpaht, Topath);

String result=isrename? " Success ":" Failure ";

System.out.println ("File rename result is:" +result);

}

}

The running results are shown in 6-4-1 and figure 6-4-2.

1) Project Browser

Figure 6-4-1 Running results (1)

2) SECURECRT Results

Figure 6-4-2 Running Results (2)

6.5 Deleting files on HDFs

Filesystem.delete (Path F,boolean recursive) removes the specified HDFs file, where F is the full path to the file that needs to be deleted, and recuresive is used to determine whether recursive deletion is possible. The specific implementation is as follows:

Package com.hebut.file;

Import org.apache.hadoop.conf.Configuration;

Import Org.apache.hadoop.fs.FileSystem;

Import Org.apache.hadoop.fs.Path;

public class DeleteFile {

public static void Main (string[] args) throws Exception {

Configuration conf=new configuration ();

FileSystem hdfs=filesystem.get (conf);

Path delef=new path ("/test1");

Boolean isdeleted=hdfs.delete (Delef,false);

Recursive deletion

Boolean isdeleted=hdfs.delete (Delef,true);

System.out.println ("Delete?") +isdeleted);

}

}

The running results are shown in 6-5-1 and figure 6-5-2.

1) Console Results

Figure 6-5-1 Running results (1)

2) Project Browser

Figure 6-5-2 Running Results (2)

6.6 Deleting a directory on HDFs

As with the deletion of the file code, just change to delete the directory path, if there is a file in the directory, to be recursive delete.

6.7 See if an HDFs file exists

Use Filesystem.exists (path f) To see if the specified HDFs file exists, where F is the full path to the file. The specific implementation is as follows:

Package com.hebut.file;

Import org.apache.hadoop.conf.Configuration;

Import Org.apache.hadoop.fs.FileSystem;

Import Org.apache.hadoop.fs.Path;

public class Checkfile {

public static void Main (string[] args) throws Exception {

Configuration conf=new configuration ();

FileSystem hdfs=filesystem.get (conf);

Path findf=new path ("/test1");

Boolean isexists=hdfs.exists (FINDF);

System.out.println ("Exist?") +isexists);

}

}

The running results are shown in 6-7-1 and figure 6-7-2.

1) Console Results

Figure 6-7-1 Running results (1)

2) Project Browser

Figure 6-7-2 Running Results (2)

6.8 View the last modified time for the HDFs file

You can view the modified time for the specified HDFs file by using "Filesystem.getmodificationtime ()". The specific implementation is as follows:

Package com.hebut.file;

Import org.apache.hadoop.conf.Configuration;

Import Org.apache.hadoop.fs.FileStatus;

Import Org.apache.hadoop.fs.FileSystem;

Import Org.apache.hadoop.fs.Path;

public class Getltime {

public static void Main (string[] args) throws Exception {

Configuration conf=new configuration ();

FileSystem hdfs=filesystem.get (conf);

Path Fpath =new path ("/user/hadoop/test/file1.txt");

Filestatus Filestatus=hdfs.getfilestatus (Fpath);

Long Moditime=filestatus.getmodificationtime ();

SYSTEM.OUT.PRINTLN ("File1.txt Modification Time" is "+moditime");

}

}

Running results 6-8-1 shows.

Figure 6-8-1 Console Results

6.9 Read all files in a directory in HDFs

"Filestatus.getpath ()" Allows you to view all files in a directory in the specified HDFS. The specific implementation is as follows:

Package com.hebut.file;

Import org.apache.hadoop.conf.Configuration;

Import Org.apache.hadoop.fs.FileStatus;

Import Org.apache.hadoop.fs.FileSystem;

Import Org.apache.hadoop.fs.Path;

public class Listallfile {

public static void Main (string[] args) throws Exception {

Configuration conf=new configuration ();

FileSystem hdfs=filesystem.get (conf);

Path LISTF =new path ("/user/hadoop/test");

Filestatus Stats[]=hdfs.liststatus (LISTF);

for (int i = 0; i < stats.length; ++i)

{

System.out.println (Stats[i].getpath (). toString ());

}

Hdfs.close ();

}

}

The running results are shown in 6-9-1 and figure 6-9-2.

1) Console Results

Figure 6-9-1 Running results (1)

2) Project Browser

Figure 6-9-2 Running Results (2)

6.10 Finding the location of a file in the HDFs cluster

The Filesystem.getfileblocklocation (filestatus file,long start,long len) is used to find the location of the specified file on the HDFs cluster, where file is the full path Start and Len to identify the path to the lookup file. The specific implementation is as follows:

Package com.hebut.file;

Import org.apache.hadoop.conf.Configuration;

Import org.apache.hadoop.fs.BlockLocation;

Import Org.apache.hadoop.fs.FileStatus;

Import Org.apache.hadoop.fs.FileSystem;

Import Org.apache.hadoop.fs.Path;

public class FileLoc {

public static void Main (string[] args) throws Exception {

Configuration conf=new configuration ();

FileSystem hdfs=filesystem.get (conf);

Path fpath=new path ("/user/hadoop/cygwin");

Filestatus filestatus = Hdfs.getfilestatus (Fpath);

blocklocation[] blklocations = hdfs.getfileblocklocations (filestatus, 0, Filestatus.getlen ());

int blocklen = Blklocations.length;

for (int i=0;i<blocklen;i++) {< span= "" >

String[] hosts = blklocations[i].gethosts ();

System.out.println ("Block_" +i+ "_location:" +hosts[0]);

}

}

}

The results are shown in 6-10-1 and 6.10.2.

1) Console Results

Figure 6-10-1 Running results (1)

2) Project Browser

Figure 6-10-2 Running Results (2)

6.11 Get all node name information on the HDFS cluster

All node names on the HDFs cluster are obtained through "datanodeinfo.gethostname ()". The specific implementation is as follows:

Package com.hebut.file;

Import org.apache.hadoop.conf.Configuration;

Import Org.apache.hadoop.fs.FileSystem;

Import Org.apache.hadoop.hdfs.DistributedFileSystem;

Import Org.apache.hadoop.hdfs.protocol.DatanodeInfo;

public class GetList {

public static void Main (string[] args) throws Exception {

Configuration conf=new configuration ();

FileSystem fs=filesystem.get (conf);

Distributedfilesystem HDFs = (distributedfilesystem) fs;

datanodeinfo[] Datanodestats = Hdfs.getdatanodestats ();

for (int i=0;i<datanodestats.length;i++) {< datanodestats.<= "" p= "" >

System.out.println ("Datanode_" +i+ "_name:" +datanodestats[i].gethostname ());

}

}

}

Running results 6-11-1 shows.

Original address: http://doudouclever.blog.163.com/blog/static/1751123102012861591374/

HDFS API Detailed-very old version