Java-api operation of HDFs file system (i)

Important Navigation

• Example 1: Accessing the HDFs file system using Java.net.URL
• Example 2: Accessing the HDFs file system using filesystem
• Example 3: Creating an HDFs Directory
• Example 4: Removing the HDFs directory
• Example 5: See if a file or directory exists
• Example 6: Listing a file or directory name under a directory
• Example 7: Viewing the file storage location
• Example 8: Writing a local file to HDFs

Accessing the HDFs file system using Java.net.URL

API usage Instructions for HDFS:
1. If you want to access the HDFS,HDFS client, you must have a configuration file for HDFs
That is Hdfs-site.xml, which reads Namenode information.
2. Each application must also have a jar file that accesses the Hadoop program
3. Operation HDFs, which is the read and write of HDFs, the most commonly used class filesystem

Back to navigation Example 1: Accessing the HDFs file system using Java.net.URL

/**

* Action: Displays the contents of the files in the HDFs folder
* 1. Use the Java.net.URL object to open the data flow
* 2. Use static code blocks to enable Java programs to identify the HDFs URL for Hadoop
*/

The operation code is as follows:

 1 package Testhdfs; 2 import java.io.InputStream; 3 import Java.net.URL; 4 import Org.apache.hadoop.fs.FsUrlStreamHa Ndlerfactory; 5 Import Org.apache.hadoop.io.IOUtils; 6/** 7 * @author Simonszhao 8 * HDFS API uses 9 * 1. If you want to access the HDFS,HDFS client, you must have a configuration file of HDFs 10 * that is hdfs-site.xml, which reads the Namenode information 。 11 * 2. Each application must also have a jar file that accesses the Hadoop program 12 * 3. Operation HDFs, which is the read and write of HDFs, the most commonly used class FILESYSTEM13 * Operation: Displays the contents of the file in the HDFs folder 14 * 1. Using Java.net.UR L Object Open Data stream 15 * 2. Use static blocks of code to make Java programs identify Hadoop's HDFs url16 */17 public class Mycat {static{19 Url.seturlstreamhand Lerfactory (New Fsurlstreamhandlerfactory ());}21 public static void Main (string[] args) {InputStream input=null;23 try {input = new URL (Args[0]). OpenStream (); Ioutils.copybytes (Input,sys             Tem.out,4096,false} catch (Exception e) {System.err.println ("Error");}finally{29 Ioutils.closestream (input);}31}32} 

0. Packager and long-passing to Linux

A. Exporting a file jar package through export

B. Select the storage path for the jar package

C. Specifying the main class

D. Upload the jar package via SECURECRT to the specified folder in Linux.

1. Create a sample file under the specified folder demo

[Email protected] filecontent]# VI Demo

　　　　

2. Upload the file to the data directory in HDFs and the data directory needs to be created first.

[Email protected] filecontent]# Hadoop dfs-put demo/data/

　　　　

3. See if the upload was successful

[Email protected] filecontent]# Hadoop dfs-ls/data/

4. Upload the packaged jar file to Linux and switch to the appropriate folder to run the Hadoop command execution

From the results can be seen to show the contents of the demo file

[Email protected] filecontent]# Hadoop jar Mycat.jar Hdfs://neusoft-master:9000/data/demo

Back to navigation Example 2: Accessing the HDFs file system using filesystem

/**
* Action: Writes the files of the local file system via JAVA-API to the HDFs file
*/

1. The specified directory should first be created in the local file system and in HDFs

Create file in linux command: mkdir test

Create folder in HDFs command: Hadoop dfs-mkdir/data/

String source= "/usr/local/filecontent/demo";//linux in the file path, the demo has a certain data, here is stored a line of English sentences, such as Welcome to .... String destination= "Hdfs://neusoft-master:9000/data/test";//hdfs's Path

2. Program source code

 1 package Testhdfs; 2 3 Import Java.io.BufferedInputStream; 4 Import Java.io.FileInputStream; 5 Import java.io.FileNotFoundException; 6 Import Java.io.InputStream; 7 Import Java.io.OutputStream; 8 Import Java.net.URI; 9 Import org.apache.hadoop.conf.configuration;11 Import org.apache.hadoop.fs.fsdataoutputstream;12 Import ORG.APACHE.HADOOP.FS.FILESYSTEM;13 Import org.apache.hadoop.fs.path;14 Import org.apache.hadoop.io.ioutils;15 16/** * @author SimonsZhao18 * Writes the file of the local file system through Java-api to the HDFs file */20 public class Filecopyfromlocal {The public static VO          ID Main (string[] args) throws Exception {a String source= "/usr/local/filecontent/demo"; file path in//linux, demo has a certain data 23 String destination= "hdfs://neusoft-master:9000/data/test";//hdfs path InputStream in = new bufferedinputs         Tream (source),//hdfs read-write configuration file, FileInputStream = new configuration (); 27 The Create method that calls filesystem returns the Fsdataoutputstream object 28//The object is not allowed in theThe file is positioned because HDFs only allows an open file sequence to be written to or appended with FileSystem fs = Filesystem.get (uri.create (destination), conf); EAM out = fs.create (new Path (destination)), Ioutils.copybytes (in, out, 4096, true); 32}33}

3. The program is packaged and transmitted to the Linux file system

Please refer to the packaging process for example 1

4. program operation and Result analysis

A. To see if the specified jar package was successfully uploaded, use LS or LL command in Linux

B. Execute jar command

C. Results show welcome to .... Instructions are working correctly

Back to navigation example 3: Creating an HDFs Directory

* Create an HDFs directory
* Example: HDFs create test2 Directory

1. Explicitly create a specific address of the directory in the HDFs file system, providing user input through the args[0] parameter in the program, as

Hdfs://neusoft-master:9000/data/test2

2. Program source code

1 package Testhdfs; 2 Import Java.net.URI; 3 Import org.apache.hadoop.conf.Configuration; 4 Import Org.apache.hadoop.fs.FileSystem; 5 Import Org.apache.hadoop.fs.Path; 6/** 7  * @author Simonszhao 8  * Create HDFS Directory 9  * instance: HDFs Create test2 directory  * Hadoop jar Createdir.jar Hdfs://neusoft -master:9000/data/test211  */12 public class Createdirction {$ public     static void Main (string[] args) {         / /hdfs path: hdfs://neusoft-master:9000/data/test215         String uri=args[0];//input Path parameter from keyboard         Configuration conf = new Configuration ();         try {             FileSystem fs = Filesystem.get (new Uri (URI), conf), and             path DFS = new path (URI);             Fs.mkdirs (DFS);         catch (Exception e) {             e.printstacktrace ();         }finally{24             System.out.println ("sucess");         }26     }27}

3. Upload the jar package to Linux

Refer to the process of exporting the jar package for the first program.

4. program operation and Result analysis

[Email protected] filecontent]# Hadoop jar Createdir.jar Hdfs://neusoft-master:9000/data/test2

Back to navigation Example 4: Deleting the HDFs directory

1 package Testhdfs; 2 Import Java.net.URI; 3 Import org.apache.hadoop.conf.Configuration; 4 Import Org.apache.hadoop.fs.FileSystem; 5 Import Org.apache.hadoop.fs.Path; 6/** 7  * @author Simonszhao 8  * Delete the file above HDFs 9  */10 public class DeleteFile {one public     static void main (St Ring[] (args) {a         String uri= "Hdfs://neusoft-master:9000/data/test2";         Configuration conf = new configuration ();         try {             FileSystem fs =filesystem.get (new Uri (URI), conf), and             path f = new path (URI)             ; Recursively delete all files under folder             isdelete= Fs.delete (f, true);             //recursively delete all files under folder             //boolean isdelete= Fs.delete (F, false);             String str=isdelete? " Sucess ":" Error ",             System.out.println (" delete "+str), and the (         Exception e) {             System.out.println (" Delete Error ~ ");         }26     }27}

3. Upload the jar package to Linux

Refer to the process of exporting the jar package for the first program.

4. program operation and Result analysis

After executing the program, via Hadoop dfs-ls/See if the file above HDFs was successfully deleted

Back to navigation example 5: see if a file or directory exists

1 package Testhdfs; 2 Import Java.net.URI; 3 Import org.apache.hadoop.conf.Configuration; 4 Import Org.apache.hadoop.fs.FileSystem; 5 Import Org.apache.hadoop.fs.Path; 6/** 7  * @author Simonszhao 8  * See if the file exists 9  */10 public class Checkfileisexists {one public     static void Mai N (string[] args) {         //string uri= "hdfs://neusoft-master:9000/data/test2/";//Specify Directory in         String uri= "hdfs:// Neusoft-master:9000/data/test2/hello ";//Specify file         Configuration conf = new configuration ();             FileSystem fs = Filesystem.get (new Uri (URI), conf),             path PATH = new Path (URI), and             Boolean isexists=fs.exists ( path);             String str=isexists? " Exists ":" Not Exists ";             System.out.println (" specified file or directory "+STR);         catch (Exception e) {             E.printstacktrace ();         }24     }25}

3. Upload the jar package to Linux

Refer to the process of exporting the jar package for the first program.

4. program operation and Result analysis

If the file exists in Linux, it will appear as follows:

"Specify file or directory exists"

Back to navigation example 6: List the file or directory name under directory

1 package Testhdfs; 2 Import Java.net.URI; 3 Import org.apache.hadoop.conf.Configuration; 4 Import Org.apache.hadoop.fs.FileStatus; 5 Import Org.apache.hadoop.fs.FileSystem; 6 Import Org.apache.hadoop.fs.Path; 7/** 8  * @author Simonszhao 9  * List the file or directory name under directory  */11 public class Listfiles {The public static void main (Strin G[] (args) {     uri= "hdfs://neusoft-master:9000/data";     Configuration conf = new configuration (); 15     try {         FileSystem fs=filesystem.get (new Uri (URI), conf), and the         path PATH = new path (URI);         filestatus Status[] = fs.liststatus (path), + for         (int i = 0; i < status.length; i++) {             System.out.println (status[i].ge Tpath (). toString ());         }22     } catch (Exception e) {         e.printstacktrace ();     }25}26}

3. Upload the jar package to Linux

Refer to the process of exporting the jar package for the first program.

4. program operation and Result analysis

Back to navigation example 7: View File storage location

 1 package Testhdfs; 2 Import Java.net.URI; 3 Import org.apache.hadoop.conf.Configuration; 4 Import org.apache.hadoop.fs.BlockLocation; 5 Import Org.apache.hadoop.fs.FileStatus; 6 Import Org.apache.hadoop.fs.FileSystem; 7 Import Org.apache.hadoop.fs.Path; 8 9/**10 * @author SimonsZhao11 * File storage location */13 public class Loactionfile {+ public static void main (string[] A RGS) {String uri= "Hdfs://neusoft-master:9000/data/demo";//hello to file Configuration conf = new Configura tion (), FileSystem fs=filesystem.get (new Uri (URI), conf), and the path PATH = new path                     (URI); Filestatus filestatus = fs.getfilestatus (path); Blocklocation blklocation[]=22 FS.GETFILEBLOCKLOCATIONS23 (filestatus, 0, Filestatus.getlen ()); = 0; i < blklocation.length; i++) {string[] hosts=blklocation[i].gethosts (); System.out.printlN ("Block_" +i+ "_location:" +hosts[0]);}28} catch (Exception e) {e.printstacktrace (); 3 0}31}32}

3. Upload the jar package to Linux

Refer to the process of exporting the jar package for the first program.

4. program operation and Result analysis

Because of the pseudo-distributed environment, block block storage is 1, so this shows only the host hostname of 1 block blocks

Display:block_0_location:neusoft-master

Back to navigation example 8: Writing a local file to HDFs

 1 package Testhdfs; 2 Import Java.io.BufferedInputStream; 3 Import Java.io.FileInputStream; 4 Import Java.io.InputStream; 5 Import Java.io.OutputStream; 6 Import Java.net.URI; 7 Import org.apache.hadoop.conf.Configuration; 8 Import Org.apache.hadoop.fs.FileSystem; 9 Import org.apache.hadoop.fs.path;10 Import org.apache.hadoop.io.ioutils;11/**13 * @author SimonsZhao14 * files on the local file system  Write to HDFs file via Java-api */16 public class Filecopyfromlocal {~ public static void Main (string[] args) throws Exception {source= string "/usr/local/filecontent/demo"; file path in//linux, demo exists a certain data String destination= "Hdfs://neu         Soft-master:9000/data/test ";//hdfs's path InputStream in = new Bufferedinputstream (new FileInputStream (source)); 21 HDFs read-write configuration file conf = new configuration (); 23//Call FileSystem The Create method returns the Fsdataoutput Stream object 24//The object is not allowed to be located in the file because HDFs allows only one open file order to be written or appended to FileSystem fs = Filesystem.get (Uri.create (destination ),conf); OutputStream out = Fs.create (new Path (destination)); Ioutils.copybytes (in, out, 4096, true); 28 }29}

Java-api operation of HDFs file system (i)