Super-Large files uploaded to Azure Linux virtual machine best practices

When the customer is actually migrating, they often encounter the need to upload large files in the local data center, such as a single 200GB file, or a total of 1TB of thousands of files uploaded to Azure, especially to the Azure Linux virtual machine scenario, which includes:

• Upload large log files to Azure for analysis

• Database backup file Import and Export

• Big data file download to local and so on

In particular, to import database backup files to a Linux virtual machine for database recovery, the general user will often choose the traditional FTP method for processing, which usually takes the user more than 10 hours, even a few days of time for file transfer, but in the era of cloud computing, In fact, there are a lot of tools and methods can be used, allowing you to significantly reduce the time and cost, today we will be 250GB database files uploaded to the Linux virtual machine as an example to see how to use these tools.

Basic Flow:

1. If you install or use a Windows machine locally, the number of cores and memory may be better because Azcopy calculates the number of concurrent threads based on the number of cores

2. Installing Azcopy

3. Create a storage account on Azure, container container

4. Upload local files to blob storage using azcopy

5. Download cloud files to a local Linux virtual machine using the Azure CLI on a Linux virtual machine

6. If you are a Windows virtual machine on Azure, use azcopy to download

Limitations and optimization Recommendations:

1. a single block blob file cannot exceed 195GB, so if your local single file exceeds this size, you can choose:

   1. Split files into smaller parts

   2. Compress files to smaller sizes with compression software

2. If your local machine is relatively strong, and a single file is indeed relatively large, it is recommended that you local compression, can save the overall time, you start to compress, you can roughly know how long this compression takes,

3. Your virtual machine and your storage account are in special need, please put in a region, such as all in Beijing or Shanghai, while minimizing delay, increase speed

4. Azcopy will make the most of your data center bandwidth, and if you want to reduce bandwidth usage, you can limit the number of concurrent processing azcopy, using/NC to make the limit, by default, if you are manipulating the blob storage, The number of concurrent threads for azcopy is 8 times times the number of your processors, and for table operations, the default number of concurrent threads is your number of processors.

5. Upload download speed and your disk speed, network bandwidth and so on, if you need to deal with large-scale files, you can consider temporarily adjust your data center bandwidth size

In this example, the basic environment is as follows:

1. Backup file uncompressed before 210GB, 60GB after compression

2. User Data Center server, Windows server 2012,16 CORES,32GB

3. Azure Linux virtual machine: A4, SUSE SP3

The basic configuration steps are as follows:

1. To download and install Azcopy on the User Data center Windows Server, install the latest version:

   Http://aka.ms/downloadazcopy

2. Create a storage account on Azure and get the primary access key for the storage account:

650) this.width=650; "Src=" http://images2015.cnblogs.com/blog/845013/201608/845013-20160824214123792-1714662709. PNG "style=" border:0px; "/>

650) this.width=650; "Src=" http://images2015.cnblogs.com/blog/845013/201608/845013-20160824214124464-900981005. PNG "style=" border:0px; "/>

3. Create a container under the storage account named upload, which you can name according to your needs:

650) this.width=650; "Src=" http://images2015.cnblogs.com/blog/845013/201608/845013-20160824214125183-653770077. PNG "style=" border:0px; "/>

650) this.width=650; "Src=" http://images2015.cnblogs.com/blog/845013/201608/845013-20160824214125823-166473449. PNG "style=" border:0px; "/>

4. Using azcopy, upload your local file to BLOB storage, note that you need to use the last step to create the storage account and account key, assuming that the current upload file name is Full0606.zip, placed on your e-disk:

Azcopy.exe/source:e:\/dest:https://myhousestorage.blob.core.chinacloudapi.cn/upload/destkey:yourstoragekey/ Pattern: "Full0606.zip"

Parameter description:

/souce: Your source file address, write directory, specific file name written behind, such as C:\data

/dest: The container address of your storage account can be seen on your container page

/destkey: Storage account's primary access key

For more detailed usage and commands on azcopy, please refer to:

https://azure.microsoft.com/en-us/documentation/articles/storage-use-azcopy/

5. The traditional way, the user will use the FTP mode, after tuning, the approximate speed is 3.5mb/s, and the use of azcopy, because it has multi-threading, network optimization and other underlying features of the support, speed can reach near 11mb/s, user data Center bandwidth 100M, this speed is quite good:

650) this.width=650; "Src=" http://images2015.cnblogs.com/blog/845013/201608/845013-20160824214126714-738676582. PNG "style=" border:0px; "/>

6. After uploading data to Azure blob storage, you need to download it locally on a Linux virtual machine, have tools like powershell,azcopy on Windows, and on a Linux system, there are azure The CLI, as a cross-platform tool, can manage azure services, and for how to install the Azure CLI on Linux, see:

http://cloudapps.blog.51cto.com/3136598/1842181

7. On a Linux virtual machine, download the compressed file uploaded to the blob storage directly using the CLI command from Azure, in the same region, at a speed of approximately 30MB, but the actual speed depends on the network condition and the current load:

# Azure Storage blob download upload full0606.zip.

650) this.width=650; "Src=" http://images2015.cnblogs.com/blog/845013/201608/845013-20160824214127698-697294700. PNG "style=" border:0px; "/>

If you need to download large files on the Linux virtual machine to local, you can consider using the above method in reverse order, azcopy through network optimization, compression optimization and algorithm optimization, data processing on the Azure artifact:)

In the previous customer's scenario, it took about 17 hours for the file to be uploaded to the Linux virtual machine, and after selecting the appropriate method, data upload, and download for about a total of 2 hours, it is important to choose different tools and methods depending on the scenario.

Super-Large files uploaded to Azure Linux virtual machine best practices