Hadoop file System Detailed--(1)

Source: Internet
Author: User
Tags hadoop fs

Hadoop has an abstract file system concept, and HDFs is just one of those implementations. The Java abstract class Org.apache.hadoop.fs.FileSystem shows a file system for Hadoop and has several implementations, as shown in table 3-1.

File system

Rr.
Scheme

Java Real

Now (all in
Org.apache.hadoop)

Describe

Local

File

Fs. LocalFileSystem

For a client-side checksum

The local Area Connection disk uses

File system. For no

There is a checksum of this

To the file system using

Rawlocalfilesystem.

See Chapter 4th for details

Hdfs

Hdfs

Hdfs. distributed-

FileSystem

Hadoop's distributed

File system. Hdfs

Designed to be used in combination

Map-reduce achieves high

Effective work

Hftp

Hftp

Hdfs. Hftpfilesystem

A reference to HTTP.

For HDFs read-only visits

The file system (although

Its name is hftp, but

It has nothing to do with FTP). Wildcard

Often used in conjunction with DISTCP

(See Chapter 3rd), in the

Lines of different versions of HDFs

Replicating data between clusters

Hsftp

Hsftp

Hdfs. hsftp-
FileSystem

Provide on HTTPS on the

HDFS read-only access

File System (IBID., with

FTP-independent)

HAR

Har

Fs. Harfilesystem

One build in other text

System up to the archived text

File system of the software. Hadoop

Archives are generally in HDFs

File when you archive it,

To reduce the use of name node memory

KFS (Cloud-store)

Kfs

fs.kfs.kosmos-
Flesystem

Cloudstore (its predecessor was

Kosmos file System)

is similar to HDFs or

Google's GFS files

System, using C + + to compile

Write. Details

Please refer to HTTP://KOSMOSFS.
sourceforge.net/

Ftp

Ftp

fs.ftp.ftp-
FileSystem

Supported by the FTP server

File system

S3 (local)

S3n

fs.s3native.native-
S3filesystem.

Supported by Amazon S3

Documents held by

System. Refer to

http://wiki.apache.org
/hadoop/amazons3

S3 (based on
Block

S3

Fs.s3.S3FileSystem

Supported by Amazon S3

The file system that holds the

Storing files in block format

(Very similar to HDFS)

To solve the S3 5 GB

File Size Limits


Hadoop provides interfaces to many file systems, which typically use URI schemes to select the appropriate file system Instance interactions. For example, the file system shell we studied in the previous section can operate on all Hadoop file systems. List the files in the root directory of the local file system, and enter the following command:

1.% Hadoop fs-ls file:///

While it is possible (and sometimes convenient) to run a MapReduce program that can access any file system, it is still necessary to select a distributed file system with optimal local data, such as HDFs or KFS, when processing large amounts of data.

Hadoop file System Detailed--(1)

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.