Java Operations for Hadoop HDFs

This article was posted on my blog

This time to see how our clients connect Jobtracker with URLs. We've built a pseudo-distributed environment and we know the address. Now we look at the files on HDFs, such as address: Hdfs://hadoop-master:9000/data/test.txt. Look at the following code:

    Static final String PATH = "Hdfs://hadoop-master:9000/data/test.txt";    public static void  Test () throws Exception {        url url = new URL (PATH);        InputStream in = Url.openstream ();        Ioutils.copybytes (in, System.out, 1024x768, true);    }

Even Hadoop supports the HDFS protocol, we first access by URL, run the above code result report error:

Unknown Protocol:hdfs

Read this to know that the original URL is not support this hdfs://, that can now change, ask a friend can set the way to solve the URL, the code is as follows:

    public static void  Test () throws Exception {               url url = new URL (PATH);        Url.seturlstreamhandlerfactory (New Fsurlstreamhandlerfactory ());               InputStream in = Url.openstream ();        Ioutils.copybytes (in, System.out, 1024x768, true);    }

Compile run or error, analyzed the next URL object is not set successfully, because it is set after the new object, I modified under:

    public static void  Test () throws Exception {          url.seturlstreamhandlerfactory (new Fsurlstreamhandlerfactory ()) ;              URL url = new URL (PATH);        InputStream in = Url.openstream ();        Ioutils.copybytes (in, System.out, 1024x768, true);    }

The running results read the HDFs file correctly and are now simply read and are implemented using Hadoop's own ioutils tool, since there is no easy API to use with Hadoop tools, of course, Hadoop provides this API, Hadoop provides filesystem to manage the operation HDFs, see the source code of filesystem it implements the method of operation HDFs, such as mkdirs (path F), Copyfromlocalfile (Boolean delsrc, Path SRC, Path DST) and so on, take a look at an example of creating a new class Hdfstest:

public class Hdfstest {static final String ROOT = "Hdfs://hadoop-master:9000/";        Static final Configuration config = new configuration (); /** * Create folder * @param path relative to paths.  such as "/test/" * @return whether to create a success * @throws Exception */public static Boolean Mkdir (String path) throws Exception        {FileSystem FileSystem = Getfilesystem ();        Boolean iscreate = filesystem.mkdirs (path);    return iscreate; }/** * Delete folder * @param path folder path * @return successfully deleted */public static Boolean RemoveDir (Stri        Ng path) {Boolean result = false;        Path strpath = new Path (path);            try {FileSystem FileSystem = Getfilesystem ();            if (Filesystem.isdirectory (strpath)) {result = Filesystem.delete (strpath); } else{throw new NullPointerException ("Logical Exception: Error deleting folder properties, failed to delete folder, if delete file should use DeleteFile (String path)"            ); }} catch (Exception e) {            System.out.println ("Exception:" + E.getmessage ());    } return result; }/** * Delete files * @param path file paths * @return successfully deleted */public static Boolean DeleteFile (Strin        G path) {Boolean result = false;        Path strpath = new Path (path);            try {FileSystem FileSystem = Getfilesystem ();            if (Filesystem.isfile (strpath)) {result = Filesystem.delete (strpath);             } else{throw new NullPointerException ("Logical exception: Error deleting file attribute, failed to delete file, if delete folder should use RemoveDir (String path)");        }} catch (Exception e) {System.out.println ("exception:" + E.getmessage ());    } return result; /** * to a list of files or folders * @param path files or folder paths * @return return filestatus[] */public static Filesta            Tus[] List (String path) {return List (path,false); /** * to the list of files or folders * @param path * @param isfull true: Recursive call folder path gets; false: Single fetchThe first layer does not make a recursive call * @return */public static filestatus[] List (String path,boolean isfull) {list<filestatus& Gt        ArrayList = new arraylist<filestatus> ();        Path strpath = new Path (path);            try {FileSystem FileSystem = Getfilesystem ();            Filestatus[] result = Filesystem.liststatus (strpath);                for (int i = 0; i < result.length; i++) {Filestatus item = Result[i];                                    Arraylist.add (item);                    if (Item.isdir ()) {filestatus[] results = List (Item.getpath (). ToString (), true);                        for (int j = 0; J < Results.length; J + +) {Filestatus filestatus = results[j];                    Arraylist.add (Filestatus); }}}} catch (Exception e) {System.out.println ("exception:" + E.get        Message ()); } filestatus[] results = new Filestatus[arraylist. Size ()];        Arraylist.toarray (results);    return results; }/** * Upload file * @param src Local Disk File * @param tar upload file to HDFs path * @return Successful upload */P        Ublic Static Boolean putfile (String src,string tar) {Boolean result = false;        Path Srcpath = new path (SRC);        Path Tarpath = new Path (TAR);            try {FileSystem FileSystem = Getfilesystem ();            Filesystem.copyfromlocalfile (Srcpath, Tarpath);        result = true;        } catch (Exception e) {System.out.println ("exception:" + E.getmessage ());    } return result; /** * Download file to Local Disk * @param path hdfs path * @return return Local Disk path * * public static String down        File (String path) {int index = Path.lastindexof ('. ') > 0 path.lastindexof ('. '): 0;        String name = path.substring (index);        SimpleDateFormat format=new SimpleDateFormat ("Yyyymmddhhmmss"); String Target =string.format ("{0}\\{1}{2} ", System.getproperty (" User.dir "), Format.format (New Date ()), name);        Path strpath = new Path (path);        Path Tarpath = new path (target);            try {FileSystem FileSystem = Getfilesystem ();        Filesystem.copytolocalfile (strpath, Tarpath);        } catch (Exception e) {System.out.println ("exception:" + E.getmessage ());    } return target; }/** * Get FileSystem instance * @return * @throws Exception */public static FileSystem Getfilesys    TEM () throws exception{return Filesystem.get (New URI (ROOT), config); }}

In the main function:

 public static void Main (string[] args) throws Exception {String Strdir = "/data/";        String strdirback = "/data/back";        String strfile = "f:\\05 note. txt";        String Strputfilepath = "/data/05 note. txt"; if (Hdfstest.mkdir (Strdir)) {if (Hdfstest.putfile (strfile, Strputfilepath)) {if (Hdfstest.mkdir (str                    Dirback)) {filestatus[] list = Hdfstest.list (Strdir);                        for (int i = 0; i < list.length; i++) {Filestatus filestatus = list[i];                        Final String name = Filestatus.getpath (). GetName ();                        Final String Path = Filestatus.getpath (). toString ();                        Final long length = Filestatus.getlen (); Final String dir = Filestatus.isdir ()?                        "D": "-";                        Final short replication = Filestatus.getreplication (); Final String Permission = Filestatus.getpermission (). Tostring ();                        Final String group = Filestatus.getgroup ();                        Final String owner = Filestatus.getowner ();                    System.out.println (dir + permission + "\ T" + replication + "\ T" + group + "\ T" + owner + "\ T" + path); }                                    }            }        }            }

come here first this time. Keep a record of every bit of drip!

Java Operations for Hadoop HDFs