Some popular Distributed file systems (Hadoop, Lustre, MogileFS, FreeNAS, Fastdfs, Googlefs)

1 , the origin of the story

Time passes quickly, and the massive upgrades and tweaks to the last project have been going on for years, but the whole feeling happened yesterday, but the system needs to be expanded again. The expansion of data scale, the complication of operating conditions, the upgrading of the operational security system, there are many content needs to be adjusted, the use of a suitable distributed file system has entered our vision.

On the Internet to find some information, but also invited some experts in the industry to talk and research, we finally chose the Distributed File system is Hadoop, incidentally also some other research system also recorded, in the future to use can also be used as a reference.

The current popular Distributed File system includes: Hadoop, lustre, MogileFS, FreeNAS, Fastdfs, Googlefs. Some of the content from the network, over time, some features may change, please note.

2, Hadoop (hadoop.apache.com)

Hadoop is a distributed system infrastructure developed by the Apache Foundation.

Users can develop distributed programs without knowing the underlying details of the distribution. Make full use of the power of the cluster for high-speed operation and storage.

Hadoop implements a Distributed File system (Hadoop distributedfile System), referred to as HDFs. HDFs is characterized by high fault tolerance and is designed to be deployed on inexpensive (low-cost) hardware, and it provides high throughput (throughput) to access application data for applications with large datasets (large data set). HDFs relaxes the (relax) POSIX requirement to access the data in the file system (streaming access) in the form of streaming.

The most central design of Hadoop's framework is: HDFs and MapReduce. HDFS provides storage for massive amounts of data, MapReduce provides calculations for massive amounts of data.

Advantages

High reliability. The ability of Hadoop to store and process data in bits is a trustworthy one.

High scalability. Hadoop allocates data between the clusters of available computers and completes computing tasks, which can easily be extended to thousands of nodes.

Efficiency. Hadoop can dynamically move data between nodes and ensure the dynamic balance of each node, so the processing speed is very fast.

High fault tolerance. Hadoop can automatically save multiple copies of data and automatically reassign failed tasks.

Low cost. Compared to data marts such as All-in-one, commercial data warehouses and Qlikview, Yonghong Z-suite, Hadoop is open source and the software cost of the project is greatly reduced.

Hadoop has a framework written in the Java language, so it's ideal to run on a Linux production platform. Applications on Hadoop can also be written in other languages, such as C + +.

Hadoop is one of the most popular distributed file systems at the moment, and the data available is quite rich.

From a usage point of view, Hadoop is a bit like using and using FTP.

3, Lustre (www.lustre.org)

Lustre, a parallel Distributed file system commonly used in mainframe computers and supercomputers. Lustre is a hybrid word from Linux and cluster. As early as 1999, Pitt Bram (English: Peter Braam) was created by the cluster file system company ("Cluster file Systemsinc.") began research and Development, published in 2003 Lustre 1.0. Use GNU GPLV2 source code authorization.

The lustre file system consists of three main functional units:

Meta-data server (metadata servers,mdses). A lustre file system typically has two metadata servers (active and standby), and a metadata server has several metadata destinations (metadata Targets,mdts). The metadata Target stores namespace metadata: Information such as file name, directory, access rights, file structure, and so on. Unlike distributed file systems, which are based on blocks, such as GPFS and Panfs, which control all block allocations by the metadata server, the lustre metadata server cares only for path searches and permission checks without involving any file I/O operations. This feature prevents the metadata server from becoming a bottleneck for cluster expansion. The fact that a single filesystem has multiple metadata targets is a new feature introduced from 2.4.

Object Storageservers,osses stores file data in one or more object storage destinations (object storage targets,osts). Depending on the server hardware, an object storage server typically has two to eight object storage targets, and each object storage target manages a local file system. The space of the lustre file system equals the total capacity of all objects to store the target.

The client (Clients) can access and use the data. Lustre provides a unified namespace for all clients.

The MDT, OST, and client can reside on the same node (typically for testing), but in a typical production environment these feature components are installed in different nodes that use the network for communication. The Lustre Network (lnet) layer can use multiple types of network interconnection, including InfiniBand, Ethernet, and so on. If a remote direct memory access (RDMA) transfer is available, lustre will use it to increase throughput and reduce CPU usage.

Run under Linux, the development of language C + +, feel very found not much information.

4, MogileFS (www.danga.com)

MogileFS is an open source Distributed file system that is used to build distributed file clusters, developed by the Danga team and also a memcached developer. The MogileFS storage engine is completely transparent to the application, and each of its nodes can be used as a lightweight HTTP Server to support get direct access to files. In addition, it includes support for load balancing during access.

The main features include

1, the application layer of components

2, no single point of failure

3. automatic file copy

4, with better reliability than raid

5, without RAID nigukefs support, running under Linux.

MogileFS does not have the concept of a directory, it requires that each file must have a unique key, For example, we copy a file text.txt to MogileFS Distributed file system, it will create a key for our file text.txt named text, when we want to read this file, we must also specify key.

This architecture determines that the mogilefs is suitable for storing files that are saved and read many times, such as images uploaded by users. Storage of frequently updated data is not an optimal option.

5, FreeNAS (www.openqrm.org)

FreeNAS is a free NAS server that turns an ordinary PC into a networked storage server. The software is based on Freebsd,samba and PHP, and supports CIFS (samba), FTP, Nfsprotocols, Software RAID (0,1,5) and web interface setting tools. Users can access the storage server through Windows, Macs, FTP, SSH, and Network File System (NFS), FreeNAS can be installed on the hard disk or mobile media USB Flash disk, less than 16MB of space. The FreeNAS server is promising; its development work is very active. 11 revisions were released in the first 4 months of 2006 alone. It is a great choice to build a simple networked storage server, eliminating the hassle of installing a complete set of Linux or FreeBSD. And Freddnas also has the role of monitoring network conditions,

Important point FreeNAS version 8 does not support system files and storage to use the same hard disk, system files use a separate device, the device can be a USB memory card, u disk, a separate hard disk, storage using another disk device

6, Fastdfs (CODE.GOOGLE.COM/P/FASTDFS)

Fastdfs is an open source lightweight Distributed File system that manages files, including file storage, file synchronization, file access (file uploads, file downloads), and so on, which solves the problem of large capacity storage and load balancing. Especially suitable for the document as the carrier of online services, such as photo albums, video sites and so on.

The FASTDFS server has two roles: Tracker (tracker) and storage node (storage). The tracker mainly does the dispatching work and plays the role of load balancing on the access.

Storage node storage files, complete file management of all functions: storage, synchronization and provide access to the interface, Fastdfs at the same time the metadata of the file management. The so-called file meta data is the file's related attributes, in the form of key-value pairs (key Valuepair), such as: width=1024, where the key is Width,value 1024. File metadata is a list of file attributes that can contain multiple key-value pairs.

Both the tracker and the storage node can consist of one or more servers. The servers in the tracker and storage node can be added or offline at any time without affecting the online service. All servers in the tracker are equivalent and can be increased or decreased according to the server's pressure.

To support large capacity, storage nodes (servers) are organized by volume (or grouping). The storage system consists of one or more volumes, the files between the volumes are independent of each other, and the file capacity of all volumes is the file capacity of the entire storage system. A volume can consist of one or more storage servers, the files in a storage server under a volume are the same, and multiple storage servers in a volume play a role of redundant backups and load balancing.

When the server is added to the volume, the synchronized files are automatically completed by the system, and after synchronization is completed, the system automatically switches the new server to the online service.

Volumes can be added dynamically when storage space is low or is about to run out.　　Only one or more servers need to be added and configured as a new volume, which expands the capacity of the storage system. Develop the language C + +, run under Linux.

7, Googlefs

GFS (Googlefs) is an extensible Distributed file system for large, distributed applications that access large amounts of data. It runs on inexpensive, common hardware, but it can provide fault tolerance. It can provide a large number of users with a higher overall performance of the service.

GFS has many of the same goals as the previous distributed file systems, but the GFS design has been driven by the workload and technical environment of current and anticipated applications, reflecting its apparent differing vision from earlier file systems. This requires a new examination of the traditional choice and a completely different design perspective.

The different views of GFS with previous file systems are as follows:

⒈ part errors are no longer treated as exceptions, but are treated as a common case. Because the file system consists of hundreds of machines that are used for storage, these machines are made up of inexpensive generic parts and are accessed by a large number of clients. The number and quality of parts make it possible for some machines to not work at any time and some may not be able to recover. So real-time monitoring, error detection, fault-tolerant, automatic recovery is essential to the system.

⒉ according to the traditional standard, the documents are very large. Files that are up to a few gigabytes in length are common. Each file usually contains a number of application objects. When dealing with fast-growing, tens of thousands of objects and terabytes of data sets, it is difficult to manage thousands of KB-scale blocks of files, even if the underlying file system provides support. Therefore, the parameters of the operation in the design, the size of the block must be reconsidered. The management of large files must be efficient and small files need to be supported, but not optimized.

⒊ Most files are updated by adding new data, rather than by changing existing data. Random operations in a file are virtually non-existent in practice. Once written, the file is readable and many of the data have these features. Some data may form a large warehouse for data analysis programs to scan. Some are data streams that are generated sequentially by a running program. Some are data of an archival nature, some are intermediate data that is generated on one machine and processed on a different machine. Because of the way these large files are accessed, adding operations becomes the focus of performance optimization and atomicity assurance. Caching data blocks in the client is not attractive.

The ⒋ workload consists of two reading operations: read operations on streaming of large amounts of data and random reading of small amounts of data. In the previous read operation, you might want to read hundreds of KB, usually up to 1MB and more. Continuous operations from the same customer usually read a contiguous area of the file. Random read operations usually read several KB at a random offset. Performance-sensitive applications typically categorize and batch read operations on small amounts of data so that the read operation is pushed forward steadily, rather than being read over and over again.

The ⒌ workload also contains a number of sequential writes to files that add data to a large amount of data. The size of the data written is similar to that of reading. Once written, the file is rarely changed. Write operations for small amounts of data are also supported at random locations, but not necessarily very efficient.

The ⒍ system must efficiently implement the semantics of adding operations to the same file for a well-defined number of customers at the same time.

7. High-sustainable bandwidth is more important than low latency