Hadoop has an abstract file system concept, and HDFs is just one of those implementations. The Java abstract class Org.apache.hadoop.fs.FileSystem shows a file system for Hadoop and has several implementations, as shown in table 3-1.
File system |
Rr. Scheme |
Java Real Now (all in Org.apache.hadoop) |
Describe |
Local |
File |
Fs. LocalFileSystem |
For a client-side checksum The local Area Connection disk uses File system. For no There is a checksum of this To the file system using Rawlocalfilesystem. See Chapter 4th for details |
Hdfs |
Hdfs |
Hdfs. distributed- FileSystem |
Hadoop's distributed File system. Hdfs Designed to be used in combination Map-reduce achieves high Effective work |
Hftp |
Hftp |
Hdfs. Hftpfilesystem |
A reference to HTTP. For HDFs read-only visits The file system (although Its name is hftp, but It has nothing to do with FTP). Wildcard Often used in conjunction with DISTCP (See Chapter 3rd), in the Lines of different versions of HDFs Replicating data between clusters |
Hsftp |
Hsftp |
Hdfs. hsftp- FileSystem |
Provide on HTTPS on the HDFS read-only access File System (IBID., with FTP-independent) |
HAR |
Har |
Fs. Harfilesystem |
One build in other text System up to the archived text File system of the software. Hadoop Archives are generally in HDFs File when you archive it, To reduce the use of name node memory |
KFS (Cloud-store) |
Kfs |
fs.kfs.kosmos- Flesystem |
Cloudstore (its predecessor was Kosmos file System) is similar to HDFs or Google's GFS files System, using C + + to compile Write. Details Please refer to HTTP://KOSMOSFS. sourceforge.net/ |
Ftp |
Ftp |
fs.ftp.ftp- FileSystem |
Supported by the FTP server File system |
S3 (local) |
S3n |
fs.s3native.native- S3filesystem. |
Supported by Amazon S3 Documents held by System. Refer to http://wiki.apache.org /hadoop/amazons3 |
S3 (based on Block |
S3 |
Fs.s3.S3FileSystem |
Supported by Amazon S3 The file system that holds the Storing files in block format (Very similar to HDFS) To solve the S3 5 GB File Size Limits |
Hadoop provides interfaces to many file systems, which typically use URI schemes to select the appropriate file system Instance interactions. For example, the file system shell we studied in the previous section can operate on all Hadoop file systems. List the files in the root directory of the local file system, and enter the following command:
1.% Hadoop fs-ls file:///
While it is possible (and sometimes convenient) to run a MapReduce program that can access any file system, it is still necessary to select a distributed file system with optimal local data, such as HDFs or KFS, when processing large amounts of data.
Hadoop file System Detailed--(1)