IOFlow-from the Microsoft perspective

       

        Note:There are a lot of negative messages about software-defined storage on the Internet. Some people say that in the history of storage development, storage has long been more than just hardware, and software has a core position in storage. Some people think that software-hidden storage should be implemented, because software makes it difficult to manage storage, such software includes deduplication and automatic configuration streamlining. Some people also say that we do not need more software, we need to see less Software. Some people say that storage is software defined, and all storage needs software; especially when storage becomes a shared resource (storage of big data and data centers), it is often surrounded by complicated and mysterious software. Many people say that each time the system is updated, more and more storage software will be added, from pattern replication to automatic configuration streamlining.

When I read the abstract of the IOFlow paper [1], I was particularly surprised. All of the above questions about software-defined storage are dispelled.

Overview

In the current data center, the I/O path from the application to the storage device is particularly long, including many layers or stages. For details, click here, in addition, the interfaces between these layers are still not transparent (like tunnel), which makes enforce an end-to-end policy (for example, for different tenant, different storage bandwidth is required, for example, hot data and cold data, such as customers of different levels of storage services) is very difficult (compared to the control logic of software-defined networks, because the network packet header is well processed ). Compared with SDN OpenFlow, IOFlow is designed here for implementation in the hypervisor and storage server phases.

        Break:The question is that IOFlow does not simply add software, but uses software to manage non-transparent software and hardware for management and monitoring, of course, fewer software will be seen (only controller programs can also be easily monitored ). In addition, when updating the system, these storage software does not need to be updated at all, and the logic is on the controller.

But since it is software-defined storage, there must be a data plane, a controller, and then an application with a controller. So the question is, what are their respective roles? Then how does it interact? The following describes in detail.

Stage

In software defined storage, the data plane is different from the data plane of the Software Defined Network. In the storage field, the data plane is layered and deep. In the Software Defined Network, the data plane is plate-wide (there are many Forwarding routing algorithms ).

IOFlow calls the data plane a stage, and implement traffic based on queues and queuing rules. (Unlike the SDN flow table, the flow table needs to be forwarded according to the flow rules ). Queuing rules map the I/O requests of different headers (original addresses (such as VM IDs) and destination addresses (such as shared storage IDs) to different queues, configure the attributes of the queue as required. In this way, these queues can use the service feature to control the speed of the queue processing process, and then use the route feature to guide I/O requests to the next stage.

First, the discovery component of the controller uses getqueueinfo to obtain the io header stage (for example, the network header or the smb header). Then, the controller can use createqueuerule to create queuing rules, allocate the incoming IO packets to the queues of the stage. Then you can configure the route and speed attributes of the queue.

 

Note:Forwarding and management between these hierarchical modules is a bit incredible. The hierarchical modules are inherently interlocking, and there is no need for anyone. It is already determined how to forward them. Therefore, the Control function only controls the forwarding speed more accurately and sets the route function for security.

 

Note:IOFlow is made by Microsoft and is not open-source. The virtual storage of IOFlow is based on virtual disk, but I want to use the fcoe virtualization layer instead of virtual disk, but vm-based, therefore, the storage stages are different based on this situation.

 

Although the principle of stage can be clearly stated, if you want to ask how to control the flow rate? In addition to the flow rate, what mechanisms are used to control service features? How to control routing? Can you give an example? Is there any special processing in the stage? The answer is as follows.

 

How can we control the flow rate? In the stage, a mechanism called token bucket is used. The operating principle of the token ring is that when an information frame is transmitted, it can be transmitted if there is enough token. Otherwise, it cannot (here, many tokens represent higher bandwidth, if it is less, it indicates a lower rate ). The queue speed is also based on the token rate. However, the storage request is different from the network packets, so the design of the licensing ring is also different. For network packets, a byte represents a licensing ring. For storage, the processing methods for different requests, such as read, write, and create are different, therefore, the design of the ring should be based on tenant's requests, and the ring should be designed based on different requests. However, there is no linear relationship between them, because in the stored backend, the processing time of each request is related to the request size, data locality, and device type, so in order to measure the overhead of io requests, the controller has done many other things. After this is measured, the controller then calls the API to configure the number of queue tokens of the stage. This configuration is updated at intervals.

In addition to using a ring to control the flow rate, you also need to use the priority feature to configure each queue. For special queues, special priority is given. Then, the length of the queue is configured using the queue size parameter, which can cause request congestion and cause pressure on the backend.

 

So how does stage control routing? As the concern in note, queue certainly has a default next hop because stage is dependent on layers. But you can also control it to take different routes. For example, let the io streams from untrusted VMS enter the malware scanning stage.

 

For example, for a tenant, all its virtual machines span 10 hypervisor, and these virtual machines access a storage server together, in this case, the controller can throttle at each hypervisor to achieve greater concurrency, rather than at the storage server end (this may cause competition ).

Special considerations? For example, in the SMBc stage, large io requests (which may be difficult to determine io performance and affect token accuracy) are decomposed into small io requests.

 

Controller overview

The content to be controlled by the software-defined storage controller is related to storage. Storage is concerned with bandwidth guarantee, and it is not difficult to reach the destination, because the storage network looks simple, it is not like the network routing is so complex as the topology structure, it often leads to a very complex routing algorithm.

 

In the software-defined storage field, Controller has two main functions: one is to interact with the above Control application, and the other is to interact with the data plane to Control the data plane. Here, the controller is mainly used to discover the stages in the data center, then interact with them, and maintain a global stage graph topology. Then, he informs the upper-layer control application of the topology and the information of each stage. These Control applications are at the same level as the Control logic in the sdn field, such as hub or l2 switching.

 

Control application

In this paper, we mainly mention two types of applications. One is the performance control application, and the other is the Malware Scanning middlebox. Malware Scanning middlebox is based on the premise that some flows are trusted, while others are untrusted. Therefore, for untrusted flow, let them enter the malware scanning middlebox for detection. If the detection finds that they can be trusted, route them to other stages (otherwise they may be discarded ). The Controller uses API configureQueueRouting to process the stream sent by the untrusted virtual machine. In general, performance is more important, and performance control applications are also needed.

Performance Control mainly includes minBandwidth or exactBandwidth. The performance control application should be able to see a subgraph that he needs. The controller must configure a suitable bandwidth for any queue in the subgraph.

For example, for the figure on the left above, in this complicated situation, we must first ensure that the bandwidth is guaranteed, then, you may need to use the algorithm on the right to ensure the bandwidth (as described in the article) and configure the queue attributes.

Compared to the situations in the Software Defined Network, such as the hub, the logic is basically relatively simple, and the second layer of learning is also relatively complicated, if FCOE's forwarding logic (once analyzed) is quite complicated, the logic of SDS's control program is quite easy. The Control application does not seem to have much space to use (such as hub, l2 switching, or fcoe routing). It is just a tenant Control program in the data center.

Note:Through the above Control application, it is estimated that the understanding of the controller is also more in-depth, but the software defined network controller has many implementation versions, such as NOX, POX, and FLOODLIGHT. However, here, Microsoft's IOFlow is not open-source, and it is estimated that only the Microsoft version of the controller is available. The controller then provides a set of APIs to control application interaction.

 P.s.If you are lazy, you will not be able to write any comments on function evaluation, scalability, and performance. In general, the features are very good and the results are remarkable, but the scalability issue is worth worrying about. If you are interested, you can check it by yourself.

References

[1] Eno Thereska, Hitesh Ballani, Greg O 'Shea, Thomas Karagiannis, Ant Rowstron, Tom Talpey, Richard Black, and Timothy Zhu, IOFlow: A SoftwareDefined Storage Architecture, SOSP '13, November 2013

IOFlow-from the Microsoft perspective