Cross-Cloud application deployment: Shared data storage

As one of the core services of cloud computing, storage services are used by every user. However, the current situation is: most customers in addition to virtual machine data disk (persistent disk), and does not fully utilize the storage services. Let's look at a typical scenario: for example, a Web application needs a user to upload an invoice scan (image file). At the code implementation level, the image files uploaded by the user are usually saved directly to the disk file system. In a traditional physical server deployment environment, this is the best cost-effective implementation, and does not prevent high availability-Multiple physical servers can share a single storage device (San/nas). However, in a public cloud environment, this design is no longer applicable because public clouds typically do not have shared storage devices for similar san/nas. Of course, Azure and AWS Publish file service and EFS services separately, but:

• Azure's file service is based on the SMB 3.0 protocol and does not implement all of the functionality in the Protocol. At the same time, in the Linux environment, the support of SMB protocol is not perfect;
• AWS EFS is based on the NFS V4 protocol, which is good for both Windows and Linux environments, but EFS is still a preview phase of AWS International, and the AWS China version naturally does not have EFS available.

In order to achieve so-called "smooth migration" without modifying the code, there is a very unreliable "putty" scenario: Running an NFS service on a virtual machine provides shared storage services for other instances (virtual machines). Obviously, the virtual machine that runs the NFS service is a single point--if the virtual machine hangs, other instances and applications that rely on shared storage are all finished! In that case, let's make it a high-availability virtual machine that runs NFS, so you don't have to worry about it. Wait, with at least 2 virtual machines to provide shared storage services, eliminate deployment and maintenance costs not to mention, 2 virtual machines cost is obvious! So often customers complain that migrating applications to public clouds is not as good as running in their own data centers.

In this case, the best solution is to use the public cloud storage services instead of shared storage devices, namely: Write/Save the user's uploaded files to the storage service, not the disk file system, so as to get rid of the shared storage devices, reduce the cost, improve the overall application system availability and scalability!

Don't lie to me, this plan to change the code! Yes, the code must be modified, but:

• Almost all public cloud-based storage services provide SDKs for a variety of mainstream languages.
• In addition to the SDK, there are restful APIs that provide easy access to the storage service as long as the language in use has the ability to access HTTP (HTTPS) services.
• Using the storage service's SDK/API to replace the disk file system reads and writes, according to personal experience, the amount of code that needs to be modified usually does not exceed 100 lines.
• Mainstream storage services typically map read file operations to HTTP GET methods (for example, each file will have a unique URL address), which in some way reduces the complexity of the read file. For example, image files in the storage service that have permissions to public can be referenced directly on the image element of the Web page, eliminating the process of reading the image file from the disk file system and publishing the image file over the HTTP service.
• The cost of storage services is very low compared to virtual machine disks. And its stability, reliability is much higher than in the virtual machine's disk storage files.

Does the above-mentioned reason seem perfectly reasonable, or is there a problem, because the use of a specific public cloud storage service will lead to a public cloud provider "kidnapping" it? In fact, there is absolutely no need to worry about this problem:

• Although the industry does not define uniform standards and specifications for public cloud storage services, there is more or less reference to AWS S3 in terms of functionality and SDK/API. It is essential to think that AWS S3 APIs have become de facto industry standards.
• From a code design standpoint, whether it's a disk file system or a public cloud storage service, it's a place to persist files. Just as the same application can support multiple databases, it is easy to support multiple file storage mechanisms with simple factory and façade patterns.

An open source project is recommended here: S3proxy, project address: Https://github.com/andrewgaul/s3proxy

The highlight of this project is the ability to allow developers to access other cloud storage services using the S3 API! This project now supports the mainstream cloud storage services such as Azure Blob,google cloud storage and OpenStack Swift. At the Microsoft Build 2016 conference, there is also a live demo of accessing Azure blobs via S3proxy. However, the problem with this project is that the AWS China and Azure China editions are not supported by default. Because S3proxy relies on Apache Jclouds, the support for AWS China and Azure China will need to be modified for jclouds (the details of its modifications are not described in this article one by one).

The essence of S3proxy is to provide a local jetty-based Web service that forwards API call requests from AWS S3 to other corresponding cloud storage services (or API calls that translate AWS S3 API calls into other cloud storage services) through a local Web service. You can even access local files through the AWS S3 API. Therefore, S3proxy needs to define a local temporary folder to be used as a file transfer.

It would be worthwhile to take advantage of the features of the public cloud by making a very small code change. Of course, there is a situation that is very Jie (WU) stuffy: Because the development team of existing application system is disbanded, or the existing application system's vendors fail/run/do not provide maintenance, resulting in the code can not be modified! I've really met this situation, and I'm not in the minority!

Cross-Cloud application deployment: Shared data storage