Parsing Linux new Technology object storage file system

With the evolution of high-performance computing from traditional host to networked cluster, the traditional host-based storage architecture has gradually developed to networked storage, and the trend of computing and storage separation is becoming more and more obvious. In view of the shortage of SAN and NAS, the research of the new file system-Object storage file system for Linux cluster has been carried out, this paper focuses on the architecture and technical features of the storage object file system, and has carried on the preliminary test to the Lustre object storage file system. The results show that the object storage file system has been significantly improved in terms of scalability, performance, ease of use and so on, with the continuous maturation of networked storage technology, object storage file system will become an important development direction.

First, the introduction

High-performance computing has evolved from traditional host mode to cluster mode, such as TOP500, only 2 systems are clustered in 1998 years, and by 2003 there are 208 clusters. With the development of high performance computing architecture, the traditional host-based storage architecture has become a new bottleneck and cannot meet the needs of cluster system. Cluster storage systems must address two key issues effectively: (1) Provide shared access data to facilitate cluster application authoring and storage load balancing, and (2) deliver high-performance storage that can meet the needs of hundreds of thousands of Linux clustered server aggregation accesses at the I/O level and data throughput rates. At present, networked storage has become an effective technical approach to solve the high performance storage of cluster systems.

There are two main types of networked storage architectures in the world, which are differentiated by command sets. The first category is the San (Storage area network) structure, which employs a set of SCSI block I/O commands, providing high performance random I/O and data throughput through data access at the disk or FC (Fiber Channel) level, with a bandwidth, low latency advantage, A niche in high-performance computing, such as SGI's CXFS file system, is based on SAN for high-performance file storage, but because of the high price of San systems and poor scalability, thousands of CPU-scale systems are not met. The second category is the NAS (Network attached Storage) architecture, which uses NFS or CIFS command sets to access data, file as a transport protocol, networked storage via TCP/IP, scalable, inexpensive, user-manageable, If the NFS file system is used in cluster computing, the high cost, low bandwidth and large latency of NAS are not conducive to the application in high performance cluster.

In response to the Linux cluster's high performance and data-sharing requirements for storage systems, new storage architectures and new file systems have been studied abroad in the hope of effectively combining the benefits of SAN and NAS systems, enabling direct access to disk to improve performance, and simplifying management through shared files and metadata. At present, object storage file system has become a hot research hotspot in Linux cluster system, such as lustre of cluster file systems Company, Activescale file system of Panasas company and so on. The lustre file system is based on object-based storage technology, which comes from the CODA Project research work of Carnegie Mellon University, released in December 2003 in Lustre 1.0, and is expected to release 2.0 editions in 2005. Lustre at the United States Department of Energy (U.s.department of Energy:doe), Lawrence Livermore National Laboratory, Los Alamos National Laboratory, Sandia National Laboratory, Pacific Northwest National Laboratory of High-performance Computing system has been a preliminary application, IBM is developing the Blue gene systems will also use lustre file system to achieve its high-performance storage. The Activescale file system technology comes from Dr. Carnegie Mellon University. Garth Gibson, the first NASD (network attached Secure disks) project supported by DARPA, is now the industry's more influential object storage file system, and won the Computerworld 2004 Innovation Technology Award.