The current popular Distributed file system reading

The current popular Distributed file system reading

A brief introduction to the Distributed File system of logarithmic type

In this paper, several kinds of distributed file system are introduced briefly. The currently popular distributed file systems include: Lustre, Hadoop, MogileFS, FreeNAS, Fastdfs, NFS, OpenAFS, Moosefs, PNFS, and Googlefs.

Lustre (www.lustre.org)

Lustre is a large-scale, secure and highly available clustered file system developed and maintained by Sun Corporation. The main purpose of the project is to develop the next generation of cluster file systems, which can support more than 10,000 nodes and a number of petabytes of storage systems.

Lustre is an open source cluster file system, with the GPL license agreement, currently in the cluster computer, the data exchange between the computer and the disk can not keep up with the speed of microprocessor and memory growth, which also drag the performance of the application, a new cluster file system software to improve the I/O speed, It is possible to reduce the cost of purchasing storage devices and change the way businesses purchase storage, which is already in use at universities, labs and supercomputer research centers and is about to enter the general business computing market. The new clustered file system employs open-source lustre technology, developed by the U.S. Department of Energy (Departmentofenergy), and offers business support from Hewlett-Packard Corporation (HP). It significantly increases the speed of input and output (I/O), which has already had a certain impact in universities, national laboratories and supercomputing research centers, and it is likely to enter the common commercial computer sector in the coming years.

Running under Linux, developing the language C + +

Hadoop (hadoop.apache.com)

Hadoop is not just a distributed file system for storage, but rather a framework for implementing distributed applications on large clusters of common computing devices.

As the architecture of Hadoop:

Licensing protocol Apache, the development of language Java, resource consumption is a bit large.

MogileFS (www.danga.com)

Mogile FS is an open-source Distributed File system with key features including

1, the application layer of the components

2. No single point of failure

3. automatic file copying

4. Better reliability than raid

5, without RAID nigukefs support, run under Linux.

FreeNAS (www.openqrm.org)

FreeNAS is a dedicated operating system for network attached storage (NAS) services (a simplified version of FreeBSD). Based on the M0n0wall firewall, the system provides disk management and RAID software that allows the user home to convert the PC to a NAS server, supporting the FTP/NFS/RSYNC/CIFS/AFP/UNISON/SSHSOURCEFORGE.NET/PRO protocol , designed to get people to reuse old hardware.

Fastdfs (CODE.GOOGLE.COM/P/FASTDFS)

Fastdfs is an open source Distributed File system, which manages files, including file storage, file synchronization, file access (file upload, file download), and so on, which solves the problem of large capacity storage and load balancing. Especially suitable for document-based online services, such as photo album sites, video sites and so on. The FASTDFS server has two roles: Tracker (tracker) and storage node (storage). The tracker mainly does the work of dispatching, which plays the role of load balancing on the access. Storage node storage file, complete all the functions of file management:

Storing, synchronizing, and providing access interfaces, Fastdfs also manages meta data for the file. The so-called file metadata is the relevant properties of the file, in key-value pairs (key Valuepair) mode, such as: width=1024, where the key is Width,value 1024. File metadata is a list of file attributes that can contain multiple key-value trees.

Develop the language C + +, run under Linux system.

Nfs

The network file system is one of the file systems supported by FreeBSD, also known as NFS.

NFS allows a system to share directories and files with other people on the network. By using NFS, users and programs can access files on the remote system as if they were accessing local files. The benefits of this are:

1. Local workstations use less disk space because the usual data can be stored on a single machine and can be accessed over the network.

2. Users do not have to have a home directory in the machine on each network. The home directory can be placed on an NFS server and is available everywhere on the network.

3. Storage devices such as floppy drives, CDROM, and zip can be used by other machines on the network. You can reduce the number of removable media devices across your network.

Develop a language C + +, which can be run across platforms.

OpenAFS (www.openafs.org)

OpenAFS is an open source Distributed file system that allows systems to share files and resources across local and wide area networks. OpenAFS is organized around a set of file servers called cells, where the identity of each server is usually hidden in the file system, and users logged on from AFS clients will not be identified as running on that server because, from a user's perspective, They want to run on a single system with recognized UNIX file system semantics.

File system content is usually replicated across cells, and one hard disk failure does not compromise the operation on the OpenAFS client. OpenAFS requires a high-capacity client cache of up to 1GB to allow access to frequently used files. It is a very secure Kerbero-based system that uses access control lists (ACLs) for fine-grained access, not based on the usual Linux and UNIX security model. Development protocol Ibmpublic, run under Linux.

Moosefs (http://www.moosefs.org/)

Moose filesystem is a fault-tolerant network Distributed file system that distributes data across different servers in the network, moosefs through fuse to make it look like a UNIX filesystem. But a little bit of a problem, it still does not solve the problem of single point of failure. Can be operated across platforms.

PNFS (www.pnfs.com)

The Network File System (NETWORKFILESYSTEM,NFS) is an important part of most local area networks (LANs). But NFS is not suitable for demanding input bookcase intensive programs in high-performance computing, at least in the past. The NFS standard crime modification incorporates PARALLELNFS (PNFS), which is a parallel implementation of file sharing, which increases the transfer rate by several orders of magnitude.

Develop the language C + +, run under Linu.

Googlefs

It is said to be a good one extensible Distributed file system for large, distributed, large-scale data access applications. It runs on inexpensive ordinary hardware, but it can provide fault-tolerant functionality, which can provide high performance services to a large number of users. Developed by Google itself.

Original from →blog.sina.com.cn

The current popular Distributed file system reading