Today's popular Distributed File system parade

Excerpted from Http://www.lupaworld.com/portal.php?mod=view&aid=205722&page=all

In this paper, several kinds of distributed file system are introduced briefly. The currently popular distributed file systems include: Lustre, Hadoop, MogileFS, FreeNAS, Fastdfs, NFS, OpenAFS, Moosefs, PNFS, and Googlefs.

Lustre ( www.lustre.org )

Lustre is a large-scale, secure and highly available clustered file system developed and maintained by Sun Corporation. The main purpose of the project is to develop the next generation of cluster file systems, which can support more than 10,000 nodes and a number of petabytes of storage systems.

Lustre is an open source cluster file system, with the GPL license agreement, currently in the cluster computer, the data exchange between the computer and the disk can not keep up with the speed of microprocessor and memory growth, which also drag the performance of the application, a new cluster file system software to improve the I/O speed, It is possible to reduce the cost of purchasing storage devices and change the way businesses purchase storage, which is already in use at universities, labs and supercomputer research centers and is about to enter the general business computing market. The new clustered file system employs open-source lustre technology, developed by the U.S. Department of Energy (Department), and offers business support from Hewlett-Packard Corporation (HP). It significantly increases the speed of input and output (I/O), which has already had a certain impact in universities, national laboratories and supercomputing research centers, and it is likely to enter the common commercial computer sector in the coming years.

Running under Linux, developing the language C + +

Hadoop (hadoop.apache.com)

Hadoop is not just a distributed file system for storage, but rather a framework for implementing distributed applications on large clusters of common computing devices.

As the architecture of Hadoop:

Licensing protocol Apache, the development of language Java, resource consumption is a bit large.

MogileFS (www.danga.com)

Mogile FS is an open-source Distributed File system with key features including

1, the application layer of the components

2. No single point of failure

3. automatic file copying

4. Better reliability than raid

5, without RAID nigukefs support, run under Linux.

FreeNAS ( www.openqrm.org )

FreeNAS is a dedicated operating system for network attached storage (NAS) services (a simplified version of FreeBSD). Based on the M0n0wall firewall, the system provides disk management and RAID software that allows the user home to convert the PC to a NAS server, supporting the FTP/NFS/RSYNC/CIFS/AFP/UNISON/SSH Sourceforge.net/pro protocol , designed to get people to reuse old hardware.

Fastdfs (CODE.GOOGLE.COM/P/FASTDFS)

Fastdfs is an open source Distributed File system, which manages files, including file storage, file synchronization, file access (file upload, file download), and so on, which solves the problem of large capacity storage and load balancing. Especially suitable for document-based online services, such as photo album sites, video sites and so on. The FASTDFS server has two roles: Tracker (tracker) and storage node (storage). The tracker mainly does the work of dispatching, which plays the role of load balancing on the access. Storage node storage file, complete all the functions of file management:

Storing, synchronizing, and providing access interfaces, Fastdfs also manages meta data for the file. The so-called meta data of a file is the relevant attribute of the file, denoted by a key value pair (key value pair), such as: width=1024, where the key is width,value to 1024. File meta data is a list of file attributes that can contain multiple key-value trees.

Develop the language C + +, run under Linux system.

　　NFS (www.tldp.org/HOWTO/NFS-HOWTO/index.html)

The network file system is one of the file systems supported by FreeBSD, also known as NFS.

NFS allows a system to share directories and files with other people on the network. By using NFS, users and programs can access files on the remote system as if they were accessing local files. The benefits of this are:

1. Local workstations use less disk space because the usual data can be stored on a single machine and can be accessed over the network.

2. Users do not have to have a home directory in the machine on each network. The home directory can be placed on an NFS server and is available everywhere on the network.

3. Storage devices such as floppy drives, CDROM, and zip can be used by other machines on the network. You can reduce the number of removable media devices across your network.

Develop a language C + +, which can be run across platforms.

OpenAFS ( www.openafs.org )

OpenAFS is an open source Distributed file system that allows systems to share files and resources across local and wide area networks. OpenAFS is organized around a set of file servers called cells, where the identity of each server is usually hidden in the file system, and users logged on from AFS clients will not be identified as running on that server because, from a user's perspective, They want to run on a single system with recognized UNIX file system semantics.

File system content is usually replicated across cells, and one hard disk failure does not compromise the operation on the OpenAFS client. OpenAFS requires a high-capacity client cache of up to 1GB to allow access to frequently used files. It is a very secure Kerbero-based system that uses access control lists (ACLs) for fine-grained access, not based on the usual Linux and UNIX security model. The development Protocol, IBM public, runs under Linux.

Moosefs (derf.homelinux.org)

The Moose file system is a fault-tolerant network Distributed file system that distributes data across different servers in the network, and Moosefs makes it look like a UNIX filesystem through fuse. But a little bit of a problem, it still does not solve the problem of single point of failure. Language Perl is developed to operate across platforms.

PNFS ( www.pnfs.com )

Network File System (FILESYSTEM,NFS) is an important part of most local area networks (LANs). But NFS is not suitable for demanding input bookcase intensive programs in high-performance computing, at least in the past. The NFS standard crime modification incorporates parallel NFS (PNFS), which is a parallel implementation of file sharing, which increases the transfer rate by several orders of magnitude.

Develop the language C + +, run under Linu.

Googlefs

It is said to be a good one extensible Distributed file system for large, distributed, large-scale data access applications. It runs on inexpensive ordinary hardware, but it can provide fault-tolerant functionality, which can provide high performance services to a large number of users. Developed by Google itself.

Today's popular Distributed File system parade