A sketch of the dynomite of the new Netflix failover system

Neil Hunt is Netflix's chief product officer, but his job is far more than he can take. In essence, Netflix's final product is to push video streaming to the ipad or smart TV, but the content displayed on the terminal is based on a lot of work.

For example, Netflix has a high profile in the area of content recommendation algorithms, but the popularity of the resilient cloud computing architecture and the new type of video pricing system is far from being matched by its strength. In a recent interview, Hunt explains the efforts Netflix has made in this area and points out where it can be upgraded in the future.

First, Netflix's recommended system for survival

No matter how you speculate, Hunt thinks the importance of Netflix's recommendation engine is still underestimated. Each quarter, he explains, customers see more than 7 billion hours of video viewed by Netflix, dismantling the data and generating about 150 million events a day during user activity. Hunt points out:

If you can use a subset of these, say 10% (15 million events a day), you can instantly affect the retention behavior of a large number of users, creating an extra billion-dollar income for existing users without the cost of expanding new users.

And looking at the current results, Hunt is very low-key said they are only just getting started, digging only the fur of Jinshan. Hunt said the company carried out about 300 A/b tests last year, much of it on the recommendation system, and about half of that affected the user's choice.

Hunt says: "Roughly speaking, every two or three days our content recommendation has a measurable boost." In other words, even if we do so much work now, there is still a lot of room for improvement. Although not every promotion can bring a high income, it is noteworthy that each promotion has the opportunity to bring tens of millions of or hundreds of millions of of dollars of income. And when we choose a successful one from 10 Tests, we find that all the worthwhile ideas in this direction have been tested and the rest is not enough for us to invest. ”

In fact, Netflix is still learning the user's scoring behavior, such as commenting immediately after the movie is usually more accurate. At the same time, the rating does not represent everything, the reason why many companies will be rated as the ultimate basis for the recommendation is that users tend to confuse quality and enjoyment. Hunt says:

Hotel Rwanda or Schindler ' s list are great videos, and they also scored 5-star acclaim. But that doesn't mean that people will be interested in the movie after a day's hard work. If the user is really selected and "very serious" is being viewed, it is basically possible to speculate that the user is already asleep or unaware of the trace.

With all this in mind, Netflix has been working to learn how users actually watch, especially on the next on-demand learning, even if their search results are not in the Netflix video library. At the same time, according to the survey, if a user read a movie from start to finish, but the film playback process, he and the device did not have any interaction, it is likely that he is not interested in the film.

To this end, Netflix has created an automatic playback interceptor that alerts users when they watch an auto play video series for 3 hours and have no other behavior.

If this happens, and there is nothing else, it basically indicates that the user is asleep or is not next to the computer, and this operation will be defined as a murmur. From another point of view, if the user determines the viewing state, and then read the two episodes, then his love for this series of video can be imagined.

Establish a suitable and business-appropriate data organization

Hunt says Netflix's data experts are divided into 3 teams: an employee who builds algorithms and data science for the entire business, professionals who specialize in content discovery algorithms, and engineers who implement these algorithms on Netflix scale. Of course, the teams are not completely isolated, they have a lot of channels for exchanging information, so they can communicate when they need it.

But there is a certain difference between Los Gatos and California's product team and the Los Angeles content sourcing team. "In layman's terms, what we need to do is figure out how to get the most out of the video, which is to put the right things in front of the right people." ”

Whether good or bad, has been running in the cloud

Netflix's streaming infrastructure on AWS has been very well run, and Hunt says that, in some ways, these infrastructures are their products. Typically, when a developer clicks on the release button, the infrastructure is responsible for pushing the feature to everyone else. Hunt says:

It's not difficult to write a user's recommended algorithm in one night, but it's not easy to deliver 15 million choices a day, because each choice takes into account the ordering of headings in 10,000 Netflix video libraries.

Netflix's availability target is 4 9, which translates into a few minutes of downtime per month, and Hunt says they have not yet achieved that goal, although the Netflix team has done its best to do the resiliency and redundancy based on the AWS environment (such as developing "simian Army" Toolset for infrastructure testing under different failures, but there is still room for improvement.

A sketch of the dynomite of the new Netflix failover system

Meanwhile, when it comes to concerns about the stability of AWS, Hunt says leasing and buying are not the main issues. In fact, the worst impact in Netflix's history is not 2012 years of Christmas Eve downtime, but rather an error database for a disk upgrade when using your own server.

Hunt says the key to using a good cloud service is to put it in the content. In AWS, there are others who are trying to improve the experience of using the cloud. Netflix needed to restart 6000 virtual servers in AWS when the Xen virtual Machine monitor bug appeared in October, when only a small number of employees were stuck in service-monitoring positions, and most of them went to the 50 million-user party at Netflix because the problem was not very large.

We are now twice times more energetic than we were in the past when migrating all the infrastructure to AWS in 2008, but please don't overlook that we are 20 times times the size of 2008. In other words, efficiency has increased 10 times times. Another problem is that we're not just relying on AWS for stability, except for catastrophic failures, instead of attributing blame to AWS, we'd rather admit that it was a design failure.

Of course, there are a lot of features that the cloud brings with concern about downtime, such as the use of AWS for Pay-per-View and Out-of-the-click Features, and Netflix can perform frequent tests, and Netflix can test both algorithms in a full production environment and then close the inefficient These are based on very little spending.

AWS Community Hero. Adrian Cockroft is now the leader of the Netflix team, and Jeremy Edberg is also a member of the Netflix team.

4K and future

About 4K and the future of technology orientation, hunt bluntly says that compared to traditional cable TV and Best Buy, Netflix can do too much for them. (Please read the original text for details, given the scope of the Netflix service)

Original link: Why Marvell and content are inseparable at Netflix (Compile/Zhonghao revisers/wei)

CSDN invites you to participate in China's large data award-winning survey activities, just answer 23 questions will have the opportunity to obtain the highest value of 2700 Yuan Award (a total of 10), speed to participate in it!

National Large data Innovation project selection activities are also in full swing, details click here.

The 2014 China Large Data Technology Conference (Marvell conference 2014,BDTC 2014) will be held at Crowne Plaza Hotel, New Yunnan, December 12, 2014 14th. Heritage since 2008, after seven precipitation, "China's large Data technology conference" is currently the most influential, the largest large-scale data field technology event. At this session, you will not only be able to learn about Apache Hadoop submitter uma maheswara Rao G (a member of the project Management Committee), Yi Liu, and members of the Apache Hadoop and Tez Project Management Committee Bikas Saha and other shares of the general large data open source project of the latest achievements and development trends, but also from Tencent, Ali, Cloudera, LinkedIn, NetEase and other institutions of the dozens of dry goods to share. There are a few discount tickets for the current ticket purchase.

Free Subscribe to the "CSDN large data" micro-letter public number, real-time understanding of the latest big data progress!

CSDN large data, focus on large data information, technology and experience sharing and discussion, to provide Hadoop, Spark, Impala, Storm, HBase, MongoDB, SOLR, machine learning, intelligent algorithms and other related large data views, large data technology, large data platform, large data practice , large data industry information and other services.