[Hadoop's knowledge] -- HDFS's first knowledge of hadoop's Core

Today, HDFS, the core of hadoop, is very important. It is a distributed file system. Why does hadoop support massive data storage? In fact, it depends mainly on the HDFS capability, mainly on the ability of HDFS to store massive data.

1. Why can HDFS store massive data?

In the beginning, let's think about this problem. I don't need to talk about the basic concepts of HDFS ~ We focus on usage rather than "research ". The argument is that "expert research" has become a real derogatory term and is ironic. In this era ~ You understand ~ Because some of them have never really experienced it, but they have to say they want to "study "~ So we don't have to worry about the concept. We only need to use it ~

Back to the question, why? In fact, HDFS has two cores: namenode and datanode. namenode can only have one in the cluster, and datanode can have multiple. We need to know that, data is mainly stored on datanode, which can be understood literally ~ And datanode can be smoothly expanded ~ This is like data can be stored in a large datanode, so it supports the storage of massive data.

2. How to Use HDFS?

HDFS can be directly used after hadoop is installed. There are two methods:

One is imperative:

We know that there is a hadoop command in the bin directory of hadoop. This is actually a management command of hadoop. We can use this to operate on HDFS.

hadoop fs -lsr /

The preceding example recursively lists all files (folders) in the root directory of HDFS)

Explanations:

Hadoop does not need to be explained. FS refers to HDFS,-ls (R), a directory LIST Command similar to Linux, listing all files (folders), and r of arc indicates recursion, /indicates the root directory.

Other common Commands include:

Hadoop FS-mkdir [path] indicates creating a folder

Hadoop FS-put [path] indicates uploading files to HDFS

hadoop fs -put ~/hello.txt /tmp/input/

And so on ~

One is API:

Check the following program:

Public static void main (string [] ARGs) throws exception {URL. seturlstreamhandlerfactory (New fsurlstreamhandlerfactory (); Final URL url = new URL (pathconstant. file01); Final inputstream in = URL. openstream ();/*** @ Param in * output stream * @ Param out * output stream * @ Param buffsize * buffer size * @ Param close * whether to close the stream */ioutils. copybytes (in, system. out, 1024, true );}

public class PathConstant {public static final String FILE01 = "hdfs://192.168.0.167:9000/tmp/input/micmiu-01.txt";public static final String DIR_PATH = "hdfs://192.168.0.167:9000/tmp/";public static final String TEST_PATH = "hdfs://192.168.0.167:9000/tmp/test/";public static final String FILE_PATH = "hdfs://192.168.0.167:9000/tmp/test/demo.txt";}

Run the above program as follows:

Directly output the content of files in HDFS. Org. Apache. hadoop. Io. ioutils; this is an operation tool class provided in hadoop.

The above is the java. Socket method. In fact, the filesystem class is recommended in hadoop for operations:

Public static void main (string [] ARGs) throws exception {// create a folder // mkdir (); // upload a file // uploaddata (); // download the file // getcontent (); // delete the file (folder) // Delete ();} public static void Delete () throws ioexception {filesystem testfilesystem = filesystem. get (URI. create (pathconstant. test_path), new configuration ();/*** @ Param f * directory * @ Param recursive * recursion */Boolean flag = testfilesystem. delete (New Path (pathconstant. test_path), true); system. out. println (FLAG);} public static void uploaddata () throws ioexception, filenotfoundexception {filesystem testfilesystem = filesystem. get (URI. create (pathconstant. test_path), new configuration (); fsdataoutputstream out = testfilesystem. create (New Path (pathconstant. file_path); fileinputstream in = new fileinputstream ("F: \ hadoop. file \ demo \ hello.txt "); ioutils. copybytes (In, out, 1024, true);} public static void getcontent () throws exception {filesystem = filesystem. get (URI. create (pathconstant. file_path), new configuration (); inputstream in = filesystem. open (New Path (pathconstant. file_path); ioutils. copybytes (in, system. out, 1024, true);} public static void mkdir () throws exception {filesystem = filesystem. get (URI. create (pathconstant. dir_path), new configuration (); Boolean flag = filesystem. mkdirs (New Path (pathconstant. test_path); system. out. println (FLAG );}