How to use signiant skydrop for data migration?

In the media industry, Signiant is already famous for the migration of large documents. Similar companies, such as broadcasters, film companies, and gaming companies, use signiant Media shuttle,signiant media Exchange and signiant manager+agents tools to improve the flow of large document delivery. By analyzing the workloads that are expanding in the cloud, you can apply the solution of large document transfer to the data transfer of large data.

Signiantskydrop

Signiant's SkyDrop is an emerging service that provides AWS users with a large amount of data to move into Amazon S3, a service that allows AWS users to stop worrying about additional cloud infrastructure management. SkyDrop is a mix of software as a service (SaaS), which means Signiant Management Server side. The Amazon elastic Cloud Compute (AmazonEC2) instance runs on the SkyDrop server, the Amazon simple Storage Service (Amazon S3) transports the build, but the end user runs a light, The Client Agent's program. Users simply need to install an AWS authenticated local client and set up the Amazon S3 bucket to use before moving files into the monitoring folder. Users who frequently move large data workloads into Amazon S3 are much easier to use than Amazon EMR and Amazon Redshift,skydrop.

When a user sends a file to Amazon S3 using Signiant's SkyDrop, its background is automatically scaled during the high-capacity transfer cycle. SkyDrop's background spans multiple AWS-usable areas and is load-balanced across multiple Amazon EC2 platforms, so it becomes very reliable without complex-setting services.

Like the solution discussed earlier, Signiant's accelerated file Transfer protocol uses mixed TCP and UDP, and signiant can minimize the impact of WAN latency, resulting in relatively fast transfer speed, for long-distance large document transmissions. Signiant propaganda channel its transmission rate is 200 times times faster than FTP. Importantly, the Signiant document Transfer Protocol supports 2 features at the same time, which is not supported in tsunami UDP: AES-256 bit encryption and intelligent transfer retries. If the transmitted data is interrupted for any reason, the transmission will restart from the interrupted location (using a large number of document restart algorithms). If the document already exists in Amazon S3 and has not been changed, SkyDrop will not update the document.

Signiant SkyDrop also supports the use of inventory for bulk document transfer, so users can effectively transfer a large number of small documents. Users need to summarize the compressed data in a large number of small documents beforehand, in order to optimize large data performance by tightly matching the document with the default HDFs block size. If the user's data is compressed in a similar way as gzip, the data is not separable and allows multiple mappings to process the set data concurrently, which improves the performance of large data. Alternatively, the user simply loads the data entered by different files into the Amazon redshift table.

Once the document arrives on the AWS Signiant SkyDrop Backend, HTTPS uses the multi-part upload API to securely transfer the document to Amazon S3.

SkyDrop has a graphics client with a Windows or Mac, a command-line interface, and a variety of programming languages that provide an SDK.

Set Signiantskydrop

1 via AWS Marketplace Login Signiant SkyDrop

2 Create an IAM user with read/write permissions in the Amazon S3 bucket that requires uploading documents.

3 Install the SkyDrop client and add the user name you just created plus the Amazon S3 Bucket that uploaded the document.

4 Drag the document into the specified confidential folder that is configured in SkyDrop. A moment later, it will appear on the Amazon S3.

Set command line interface (CLI)

1 storage and critical config.cfg files for Amazon S3 by adding user credential configuration skydropcli.

2 Transfer a single file using CLI commands, just use the-D upload parameter. In the following example, Matt uses the M3.xlarge Amazon EC2 instance located in Us-east-1 to run without interference, based on the Amazon Linuxami of AWS. Matt transmits a 1-GIB uncompressed file and uses DD to raise the American Standard to the Amazon S3 Bucket. Importantly, this file is located in the EC2 instance store, so that the throughput of Amazon's resilient block storage (Amazon EBS) will not be a distortion test bottleneck. The average transmission rate in this case is about 630Mbps.

A more complex document transfer may involve a large number of documents in the list, in the list, one document per line:

In this case, interactive movement (-i) is used to see the statistics for real-time file transfers and to generate detailed transport statistics (-Z) at the end of the transmission.

Conclusion

Signiant's SkyDrop is a fast and easy way to move large data into the cloud. Because it's a SaaS solution, it's easy to use and users don't have to worry about deploying and maintaining a highly available and High-performance file transfer system architecture. SkyDrop uses Signiant's accelerated file Transfer protocol, which moves to the Amazon EC2 backend, and then optimizes the transfer to Amazon S3 so that users can significantly reduce the speed of the data transfer clouds. Finally, SkyDrop is retried in traffic and intelligent document delivery, while encryption means that users can safely and reliably transfer documents.

Original link: Http://blogs.aws.amazon.com/bigdata/post/Tx2QZODBZN1RD1I/Moving-Big-Data-Into-the-Cloud-using-Signiant-SkyDrop

If you need to know the latest AWS information or technical documentation to access the AWS Chinese technology community, if you have more questions please ask at the AWS Technology Forum and experts will answer later.

Subscribe to the "AWS Chinese technology Community" micro-credit public number, real-time command of AWS technology and product information!

The AWS Chinese technology community provides an Amazon Web service technical Exchange platform for the vast majority of developers, pushing the latest news, technical videos, technical documents, wonderful technical blogs and other related highlights from AWS, as well as having AWS community experts to communicate with you directly! Join the AWS Chinese technology community to quickly and better understand the AWS cloud computing technology.

(Translator/li Zebian/yuping)