HDFS schematic diagram:
Let's write an HDFs-based demo that basically implements the ability to read one of the contents of a file on HDFs and save it to another file.
1. Auxiliary class
This class is primarily used to get the HDFs file system connection
Public classHdfsutils {/** * @return * @throwsException*/ Public StaticFileSystem Getfilesystem ()throwsexception{Configuration conf=NewConfiguration (); Conf.set ("Fs.defaultfs", "hdfs://192.168.1.109:8020"); FileSystem FileSystem=filesystem.get (conf); returnFileSystem; } /** * @paramPopenuri *@paramPuser *@return * @throwsException *@throwsinterruptedexception *@throwsurisyntaxexception*/ Public StaticFileSystem Getfilesystembyuser (String popenuri,string puser)throwsException, Interruptedexception, urisyntaxexception{Configuration conf=NewConfiguration (); Conf.set ("Fs.defaultfs", "hdfs://192.168.1.109:8020"); FileSystem FileSystem= Filesystem.get (NewURI (Popenuri), Conf, puser); returnFileSystem; } /** * @paramPuser *@return * @throwsException *@throwsinterruptedexception *@throwsurisyntaxexception*/ Public StaticFileSystem Getfilesystembyuser (String puser)throwsException, interruptedexception, urisyntaxexception{String Fileuri= "/home/test/test.txt" ; Configuration conf=NewConfiguration (); Conf.set ("Fs.defaultfs", "hdfs://192.168.1.109:8020"); FileSystem FileSystem= Filesystem.get (NewURI (Fileuri), Conf, puser); returnFileSystem; } }
2. Main class
This class is primarily used for file read and write and creation.
Public classHdfsfstest { Public Static voidMain (string[] args) {String Fileuri= "/home/test/test.txt"; String Fileoutputurl= "/home/test/out.txt"; Try{Writefiletohdfs (Fileuri, Fileoutputurl); System.out.println ("Done!"); } Catch(Exception e) {e.printstacktrace (); } } Public Static voidWritefiletohdfs (String Popenuri, string poutputurl)throwsException {FileSystem FileSystem=NULL; Fsdatainputstream FileInputStream=NULL; Fsdataoutputstream FileOutputStream=NULL; intBuffsize = 4096; Try{FileSystem=Hdfsutils.getfilesystem (); FileInputStream= Filesystem.open (NewPath (Popenuri)); FileOutputStream= Filesystem.create (NewPath (Poutputurl)); Ioutils.copybytes (FileInputStream, FileOutputStream, Buffsize,false); } Catch(Exception e) {Throwe; } finally{ioutils.closestream (FileInputStream); Ioutils.closestream (FileOutputStream); } }}
3. Running Results
Run successfully!
Hadoop Programming implementation of HDFS