Hadoop Diary Day9---hdfs Java Access interface

First, build the Hadoop development environment

The various codes that we have written at work are run on the server, and the operation code of HDFS is no exception. In the development phase, we use eclipse under Windows as the development environment to access HDFS running in the virtual machine. That is, access to HDFS in remote Linux through Java code in local eclipse.
To access HDFs in the client using Java code from the host, you need to ensure the following:
(1) Ensure that the host and client network is interoperable
(2) Ensure that the host and client firewalls are turned off , because many ports need to pass, in order to reduce the firewall configuration, directly shut down.
(3) Ensure that the host is consistent with the JDK version used by the client. If the client is JDK6 and the host is JDK7, the code will run with an unsupported version error.
(4) The host's login username must be the same as the client's user name . For example, we use the root user of Linux, then Windows will use the root user, otherwise it will report permissions exceptions
Overwrite the org.apache.hadoop.fs of Hadoop in the Eclipse project. The Checkreturnvalue method of the Fileutil class , 1.1, is intended to avoid permission errors .

Figure 1.1

If the reader has issues with permissions during the development process, follow the prompts in this section to check your environment.

Ii. reading and writing data using the FileSystem API

In the HDFs operation of Hadoop, there is a very important API, which is ORG.APACHE.HADOOP.FS. FileSystem, this is the direct entry of our user code operation HDFs , which contains various methods of manipulating HDFs, similar to the direct entry of the operational database in JDBC is the Connection class .

So how do we get a FileSystem object ?

--------------------------------------------------------------------------------------------------------------- ----------------

            String uri = "Hdfs://10.1.14.24:9000/";
            Configuration conf = new Configuration ();
            FileSystem FS = Filesystem.get (uri.create (uri ), );

--------------------------------------------------------------------------------------------------------------- ----------------
In the above code, be aware that the call is filesystem static method get, pass two values to the formal parameter, the first access to the HDFS address , the address of the protocol is Hdfs,ip is 10.1.14.24, the port is 9000. The complete information for this address is specified in the configuration file Core-site.xml , and readers can use the settings in their own environment's configuration file. The second parameter is a configuration object .

1. Create a folder
Use the HDFs shell command to look at the file under the root directory , as shown in 2.1.

Figure 2.1

We created the folder under the root directory of HDFs and the code is as follows

--------------------------------------------------------------------------------------------------------------- ----------------

Final String pathstring = "/D1";
Boolean exists = fs. exists (New Path (pathstring));
if (!exists) {
Boolean result = fs. mkdirs (New Path (pathstring));
SYSTEM.OUT.PRINTLN (result);
}

--------------------------------------------------------------------------------------------------------------- ----------------

The above code should be placed in the main function ,

• The first row determines the folder full path created is "/d1".
• The second line of code is to use the method exitst to determine if the folder exists, or if it does not exist, perform the create operation.
• The third row creates the folder, the Mkdirs method is called, the return value is a Boolean value, and if true, indicates that the creation was successful, or false to indicate that the creation failed.

Now to see if it's successful, 3.2,3.3 is created successfully.

Figure 3.2

Figure 3.3

2. Writing files

We can write the file to HDFs with the following code:

--------------------------------------------------------------------------------------------------------------- ----------------

Final String pathstring = "/D1/F1";
Final fsdataoutputstream fsdataoutputstream = fs.create (new Path(pathstring));// Write it out.
Ioutils. copybytes (New Bytearrayinputstream("My name is Sunddenly". GetBytes ()),
Fsdataoutputstream, conf, True);

--------------------------------------------------------------------------------------------------------------- ----------------
The first line of code indicates that the file created is the file F1 under the D1 folder you just created;
The second line is to call the Create method creating a output stream leading to HDFs；
The third line is to send a string to the output stream by invoking the static method of Ioutils, a tool class for Hadoop copybytes.

The static method has four parameters:

• The first parameter of the input stream .
• The second parameter is an output stream .
• The third parameter is a configuration object .
• The fourth parameter is a Boolean value and if True indicates that the stream is closed after the data transfer is complete .

Now look at whether the creation was successful, as shown in 3.4.

Figure 3.4

3. read the file

Now we read the file "/d1/f1" that we just wrote to HDFs, the code is as follows:

--------------------------------------------------------------------------------------------------------------- ----------------
Final String pathstring = "/D1/F1";
Final fsdatainputstream fsdatainputstream = fs. Open (New Path (pathstring));// read in
Ioutils.copybytes (fsdatainputstream, System.out, conf, True);

--------------------------------------------------------------------------------------------------------------- ----------------

• The first line specifies the path of the file being read.
• The second line means the calling method open opens a specified file, and the return value is an input stream to the file ;
• The third line is also called the Ioutils.copybytes method, and the output destination is the console.

See Figure 3.5

Figure 3.5

4. View directory listing and file details

We can display all the files and directories under the root directory, the code is as follows

--------------------------------------------------------------------------------------------------------------- ----------------
Final Stringpathstring= "/";
Finalfilestatus[] Liststatus= fs.Liststatus(New Path (pathstring));
for (Filestatus filestatus:liststatus) {
Final Stringtype= Filestatus.isdir ()?" Directory ":" File ";
Final shortReplication= Filestatus.getreplication ();
Final StringPermission= Filestatus.getpermission (). toString ();
Final longLen= Filestatus.Getlen ();
Final PathPath= Filestatus.GetPath ();
System.out.println (type+ "\ t" +permission+ "\ T" +replication+ "\ T" +len+ "\ T" +path);
}

--------------------------------------------------------------------------------------------------------------- ----------------
Calling the Liststatus method will get all the files and folders under a specified path, each represented by a filestatus. We use a For loop to display each of the Filestatus objects. The Filestatus object represents the details of the file, which contains a lot of information about the type, number of copies, permissions, length, path, and so on, and we just show a part. As shown in result 3.6.

Figure 3.6

5. Delete files or directories

We can delete a file or a path, the code is as follows

--------------------------------------------------------------------------------------------------------------- ----------------
Final String pathstring = "/D1/F1";
Fs.delete (New Path ("/d1"), true);
Fs. Deleteonexit (New Path (pathstring));

--------------------------------------------------------------------------------------------------------------- ----------------
The third line of code represents the deletion of the file "/d1/f1", and the second line of commented-out code indicates the recursive deletion of the directory "/d1" and all of the contents below. In addition to the FS methods listed above, there are many ways to read the API yourself.

Hadoop Diary Day9---hdfs Java Access interface