Why is your Instagram photo not in the original place?

Instagram and Facebook engineers

According to an article written by foreign media today, about one year after the acquisition of Instagram was completed last year, Facebook started to migrate the photo sharing service from Amazon cloud to its own data center, involving the transfer of 20 billion photos. The project is complicated and takes about one year. What's even more surprising is that users are totally unaware of the Instagram migration process. Facebook exactly how to do it?

The following is the main content of the article:

Your Instagram photo is not in the original place.

Just about 200 million users were using Instagram on their cellphones this spring, a small team of engineers moved the photo-sharing service from Amazon's cloud computing service, built in 2010, to a data center operated by Facebook. "The user is still in the car they were sitting on when they started their journey," says Instagram founder Mike Krieger, a metaphor for the move. "But in fact we have replaced every part of the car , But people do not feel it at all. "

"Instagration"

Facebook referred to the move as "Instagration," an unprecedented move for CEO Mark Zuckerberg and the company. Facebook has moved other acquired assets (such as FriendFeed) to its own data center, but those are often small projects that shut down services and put them into the world of Facebook.

Instagration is a dynamic migration of a large and popular service. "There's no disruption to the Instagram service during that time," said George Cabrera, a Facebook engineer. Facebook did not disclose how many virtual machines are required to run Instagram on the Amazon cloud platform, just "thousands". The service now stores more than 20 billion digital photos.

For Instagram, the move gives it more efficient access to a long list of computing tools that have helped drive the massive online Empire of Facebook. For engineers responsible for running the Facebook global data center network, it's a template that blends the company's operations with the company's future acquisitions.

In the meantime, Instagration also gave the entire tech community one or two lessons. The community is increasingly building app services on cloud computing services such as Amazon AWS - those applications where they are likely to move to private data centers in the future. The key to Instagration is Amazon's proprietary service called Virtual Private Cloud ("VPC").

Complex engineering

In April 2013, about a year after Facebook spent $ 1 billion on Instagram, Jay Parikh, vice president of engineering for the social network, said the company plans to move Instagram to its own Computing facilities. The project was started at that time. The relocation took about a year, although it was a huge undertaking, but not many people were in charge. In 2013, Instagram had 8 engineers responsible for its infrastructure, which later grew to 20. Cabrera said the team was actually preparing for more than six months before it started a one-month data move.

Since 2010, Instagram has been running on the Amazon EC2 cloud service. To seamlessly migrate Instagram to Facebook's data center on the East Coast, Cabrera's team first created a copy of the software that supports the Instagram service. Once the copy is up and running on Facebook, the team can implement data transfer - including 20 billion photos.

This process is more complex than imagined. It involves developing a private computer network that runs across Facebook data centers and Instagram on the Amazon cloud - the best way to securely migrate all your data - but if you do not switch Instagram to another part of the Amazon cloud, Can not create such a network. In other words, Craig's team is going to have two relocations of Instagram.

First, they moved Instagram to Amazon's APC, a tool the Craig team used to create a logical network on Amazon that extends to Facebook's data center. Creating this network is especially important because it gives Facebook full control over the web addresses used by Instagram machines. Allegedly, if they were not moved to APC, they would not be able to customize those network addresses, which meant that they would face countless address conflicts during their move to their own data centers.

However, there's something more complicated in Instagration - the first to migrate Instagram from EC2 to VPC, the team also needed to set up a public network in those two environments. Amazon does not provide a solution for this. As a result, Facebook temporarily developed a networked tool called Neti. Neti's significance is that it is another important step in the Instagration process - an important lesson for those who develop services on Amazon and other cloud services.

VPC did not exist when Instagram was founded in 2010. Now, if other startups run the service on the VPC right from the start, they can eliminate the extra steps in Instagram migration. VPC can also help meet the need to move a portion of the infrastructure from the cloud to a private data center.

After really getting ready to migrate software and data, the Craig team used the increasingly popular Chef tool. This tool is used to write automated programs to load and configure digital content on a large number of machines. For example, the programs they write automatically load the appropriate software onto machines running Amazon VPC, and they then use a similar program to load almost the same software on a machine in a Facebook data center. One of the programs they developed was for installing software on each of Instagram's database servers and one for configuring cache servers to speed up the push of popular photos.

Meaning of transfer

Instagram last batch of software and data landing Facebook data center in late April. In mid-April, there was a disruption in service to Instagram users around the world, but the company said the incident had nothing to do with its migration.

Instagram's dedicated machines are now in the Facebook facility. According to Pedro Canahuati, a Facebook engineer, this makes the service more efficient, using only a third of the servers it serves on the Amazon cloud, and Instagram and the Facebook team Share a variety of data migration techniques, so Instagram's "data capture" time a full 80% reduction.

The move also means that Instagram can more easily and efficiently connect to other services running on Facebook's facilities, as well as Facebook's other data centers around the world. In addition, Facebook has built a massively massive data analytics system that Instagram now can leverage the "big data" infrastructure just as any other Facebook in-house service does. Instagram also takes advantage of Facebook's spam filtering tool.

However, Craig said Instagram does not share data with Facebook. Facebook, for example, can not use your Instagram behavior to accurately target ads on its social networking platform - and vice versa.

Instagration is not unusual. Adrian Cockcroft, former head of cloud infrastructure for Netflix, seems that everyone is moving to the cloud rather than moving away. "If your company is as big as Facebook, you will want to build your own data center," he said, "and most small businesses should not do that." But with the ever-expanding scale, games like Zynga The company moved a large part of its operations from the cloud to private data centers. There are certainly other companies that will do the same in the future - especially those companies that are under Facebook.

http://www.wired.com/2014/06/facebook-instagram/

Translator: Le Bang

Baidu News and NetEase science and technology manuscripts, please indicate the source.