Concept and Application of NFS protocol and pNFS

NFS is the foundation of network storage development and also the basic protocol for network sharing. Let's talk about the development in this field. First, we need to understand the basic concepts of NFS and some corresponding applications. Next we will introduce the knowledge of PNFS.

NFS was designed and developed by Sun Microsystems in 1984 to connect to their systems. in the course of its development, the storage industry has been revolutionalized by facilitating the connection between file systems, so that all files in an environment can be put under the public view.

Back in 1984, the High-Speed Backbone Network was 10 Mbit/s Ethernet, while the disk drive capacity was measured in Mb instead of GB. NFS meets requirements and is soon widely used in the industry and becomes a standard.

Since then, NFS has made some limited improvements, some of which are aimed at performance, but most of the improvements are minor-many for security and to keep up with other UNIX standards. we are all familiar with NFSv1, v2, and v3. Now it is v4, And we all complain about the performance of NFS, because the performance of network and storage is growing faster, but NFS is not keeping up with the pace.

Since the advent of NFS, the network has grown from 10 Mbit/s Ethernet to 10 Gbit/s Ethernet (three orders of magnitude of performance growth ), storage has grown from 3 Mb/s disk drives to 400 Mb/s Fiber Channel RAID (Redundant Array of Independent Disks), which is also a 133-fold increase in performance. at the same time, NFS only evolved from 4 KB data packets to 64 KB. for some poor networks, large data packets may also be a problem, but when we need to transmit a large amount of data on high-speed networks, we need better technology. the good news is that NFS is coming soon.

Don't worry, NFS v4.1 is coming.

The best way to understand NFSv4.1 (also called pNFS) is to read some articles. as part of the NFSv4.1 protocol, pNFS (parallel NFS) enables high-speed data movement between devices. this was originally developed in another group. pnfs.com provides some background knowledge about why pNFS is needed (read the questions listed below, they have pointed out in detail some problems of the existing NFS Protocol ).

This figure depicts the pNFS system architecture:

The most interesting picture is that IEFT (International Internet Engineering Task Force) is preparing to support the T10 OSD protocol (T10: SCSI Storage interface technical committee, OSD: Object-based storage device ), and use it as part of NFSv4.1. this means that with NFSv4.1, you can move data from the NFS server by block, T10 OSD object, or file. it is very important that, as NFSv4.1 was started, most vendors only started to implement one of these methods, so there were some problems.

Data and Metadata are separated as part of the transmission mechanism. therefore, for block-based storage, generally, index nodes and any non-direct extensions are clustered together, while block addresses are accessed by using DMA (Direct Memory Access) to directly obtain from the network. compared with NFSv4 and earlier versions of NFS to read files on the network, the performance of this method will be a thousand miles. of course, if you read files over 10 Mb/s Ethernet or 1 Gbit/s Ethernet, the difference will not be great, but you will also feel the improvements brought about by the adoption of new methods. now, if you have 10 Gbit/s Ethernet, or DDR (dual data rate) IB (InfiniBand), or a faster network in the future, so the existing NFS protocol can make these high-speed networks run like on 1 GB Ethernet.

Around 1991, I was called by a site for technical support. they have two Cray supercomputers connected through HiPPI (high-performance parallel interface, 800 MB), and they are also composed of an FDDI (Optical Fiber Distributed Data Interface, 100 Mb ). you may wonder why the speed of HiPPI is only 10% higher than that of FDDI. they asked why they spent so much money to deploy HiPPI, but only achieved 10% performance improvement. I cannot give a good answer. of course, ftp and rcp run faster on HiPPI. the only benefit of this visit is that I went to Germany because I could not give the customer a satisfactory response. at that time, NFS was not running fast even on high-speed networks. NFS has made some minor adjustments to make it acceptable on 1 GB Ethernet, but it still cannot reach the network speed. in the past 15 years, we are about to enter the commodity high-speed internet era, such as 10 GB Ethernet, but the current version of NFS is still not competent. the following describes the NFS problem:

"Customers' requirements for storage I/O bandwidth grow much faster than what the network file server can provide. when we install NFS, we will encounter more and more such problems. this problem can be solved by increasing the server bandwidth. this draft recommends improvements to the NFS file service so that the NFS file service can be expanded as the customer base expands. the recommended method is to centralize all possible bandwidth on a single file system through the parallel file service, the result is that multiple networks are connected to multiple server terminals that require data transmission. we need to implement this method within the NFS architecture, for example, through an NFSv4 Protocol version ".

Obviously, the IETF statement shows that it understands the current limits of NFS and is trying to solve them. almost all major vendors have been involved in the formulation of NFSv4.1 (pNFS), including but not limited to EMC, IBM, NetApp, Sun, and of course the Linux community. there are even some network rumors that Microsoft is preparing to consider NFSv4.1. I am skeptical about this in future Windows versions.

Limitations of NFSv4.1

On the contrary, NFS is a protocol, not a file system. NFS allows common external interfaces to access the file system. even if its name is like this, it is not a file system. this means that you are constrained by the performance of the underlying file system. for example, if Microsoft includes NFSv4.1 in a later version of Windows and your camera is connected to 10 GB Ethernet, however, this does not mean that you can use the FAT32 File System to export data on your old computer at a speed of 1 GB/second. there are many restrictions.

The fact is that our media cannot transmit data at 1 Gb/s. More importantly, you are subject to the performance of the file system. in real time, you have a media that can transmit data at 1 Gb/s. Given the current data format, the FAT32 file system cannot support this data rate.

I believe that NFSv4.1 will dwarf many file systems. now, these NFS file systems can be used as an excuse to say that they are limited by the current NFS performance limitations. This may be the case, but this is not the case in the future. one thing I 've been trying is to run performance tests on a local NFS file server to test how fast I can read/write. then I will observe the underlying hardware and determine whether the performance bottleneck lies in the file system or the hardware. during these times, I found that, more often, the performance bottleneck is on the file system rather than the hardware. due to the limitations of the current NFS protocol, while 10 GB Ethernet does not, the file system connected to the server on NFS can blame the protocol for performance problems, but this day will soon pass.

As a pessimistic commentator, I think some vendors will choose not to support NFSv4.1 and 10 GB Ethernet until they can make their file systems work perfectly with NFSv4.1 and 10 GB Ethernet, the speed at which data is transmitted over the network. no manufacturer wants to make himself look bad. vendors with high-performance file systems (we all know who they are) may be the first to support NFSv4.1. because you are always constrained by the slowest components in the data channel, therefore, those file systems that do not support high-performance data transmission may not support NFSv4.1. it is very interesting to observe the progress of NFS4.1 in the next few years, especially if you understand why the vendor is doing this.