Hadoop file System Detailed--(1)

Hadoop has an abstract file system concept, and HDFs is just one of those implementations. The Java abstract class Org.apache.hadoop.fs.FileSystem shows a file system for Hadoop and has several implementations, as shown in table 3-1.

File system	Rr. Scheme	Java Real Now (all in Org.apache.hadoop)	Describe
Local	File	Fs. LocalFileSystem	For a client-side checksum The local Area Connection disk uses File system. For no There is a checksum of this To the file system using Rawlocalfilesystem. See Chapter 4th for details
Hdfs	Hdfs	Hdfs. distributed- FileSystem	Hadoop's distributed File system. Hdfs Designed to be used in combination Map-reduce achieves high Effective work
Hftp	Hftp	Hdfs. Hftpfilesystem	A reference to HTTP. For HDFs read-only visits The file system (although Its name is hftp, but It has nothing to do with FTP). Wildcard Often used in conjunction with DISTCP (See Chapter 3rd), in the Lines of different versions of HDFs Replicating data between clusters
Hsftp	Hsftp	Hdfs. hsftp- FileSystem	Provide on HTTPS on the HDFS read-only access File System (IBID., with FTP-independent)
HAR	Har	Fs. Harfilesystem	One build in other text System up to the archived text File system of the software. Hadoop Archives are generally in HDFs File when you archive it, To reduce the use of name node memory
KFS (Cloud-store)	Kfs	fs.kfs.kosmos- Flesystem	Cloudstore (its predecessor was Kosmos file System) is similar to HDFs or Google's GFS files System, using C + + to compile Write. Details Please refer to HTTP://KOSMOSFS. sourceforge.net/
Ftp	Ftp	fs.ftp.ftp- FileSystem	Supported by the FTP server File system
S3 (local)	S3n	fs.s3native.native- S3filesystem.	Supported by Amazon S3 Documents held by System. Refer to http://wiki.apache.org /hadoop/amazons3
S3 (based on Block	S3	Fs.s3.S3FileSystem	Supported by Amazon S3 The file system that holds the Storing files in block format (Very similar to HDFS) To solve the S3 5 GB File Size Limits

Hadoop provides interfaces to many file systems, which typically use URI schemes to select the appropriate file system Instance interactions. For example, the file system shell we studied in the previous section can operate on all Hadoop file systems. List the files in the root directory of the local file system, and enter the following command:

1.% Hadoop fs-ls file:///

While it is possible (and sometimes convenient) to run a MapReduce program that can access any file system, it is still necessary to select a distributed file system with optimal local data, such as HDFs or KFS, when processing large amounts of data.

Hadoop file System Detailed--(1)