Cloud computing platform injects new blood, Disney's first big data

Compared with the entertainment giants such as Disney, many small companies, relying on strong will and extraordinary wisdom, with a small team, using Hadoop, NoSQL database and other open source technology, can fully create a unique large data platform. And for Disney, once it comes to big data platforms, the entertainment giant looks more like a start-up. Such large companies are moving into large data areas, and the results are bad for us to judge, but at least they mean a completely different rule of "play".

Arun Jacob, head of Disney's Big Data technology and Services Solutions team, introduced Disney's Big Data platform at the Thursday Big Data Innovation Summit in Boston's IE group. Unlike other companies, Disney chose to build large data platforms from scratch, not from a software vendor. While high costs are an important factor, the key is the question of flexibility, which is the root cause of the final decision.

Reducing, reusing, recycling

In order to guarantee to bring the biggest value to the company, the Disney Big Data platform, condenses each staff's painstaking effort, it is everybody's everything, this is a formidable task. At first, Jacob said, "We think of ourselves as a small advisory body, just that we have something to sell." "But when a department wants to use the platform to develop a specific feature, Jacob acts immediately."

Architecturally, through this platform and a specific purpose component, it can refactor path data, or it can easily replace components, if any. Disney's Big Data platform is developed based on Hadoop,cassandra and MongoDB. The operating team can use the platform to view, analyze, and index error messages, and application developers can get the high throughput, low-latency data access they need, while the analysis team has access to the high latency data they need.

However, while Jacob also wants to use a set of Open-source software to cut costs, he does have the luxury of being able to afford to splurge, a budget that most startups don't have to play-outsourcing and unscheduled new products. When he needs the support of the Hadoop cluster, he can call Cloudera. When it is necessary to deploy Solandra (an open source search engine based on SOLR and Cassandra), he can completely buy the Enterprise version of DataStax based on Cassandra Products, but he did not.

Flexibility is not free

Solandra events are actually meant to weigh the need to use free Open-source software. "You can work on open source projects late into the night, you can learn to run them, but that doesn't make any sense." "Jacob says that if you are willing to devote time and energy, these things can be completely overcome."

But there are more problems for Disney-scale companies that have to be overcome. Jacob says that while the deployment process can address fault tolerance, high availability, and security in its own way, it ultimately needs to find a way to implement these things.

The best for the public

While the system can be built on open source software that all people can use, it also means that there is not enough framework to build a scalable and stable system, and that the system must meet the needs of thousands of different types and levels of in-house developers. Jacob said it would be easy for a start-up with six people to learn Hadoop for one months and then start using it to deploy large data platforms. But for a big business, this is absolutely not feasible.

His team made deployment easier

To eliminate the excuse that enterprise users cannot load their data into the system, they only need to point the file to a user-defined interface. Jacob said that Disney's data platform, although with 5TB of data per day dramatically, but there are still many other types of data need to be stored. Because they've encapsulated the technology, Jacob's team didn't talk much about Hadoop and MongoDB, just the parts of the analysis and the query. The data platform uses many kinds of programming languages to build the client's frame structure, so developers can interact with the platform without having to write restful API calls.

However, after doing all the preparation, Jacob began to devote his energy to Disney's big Data platform, and Jacob did not want it to be a duplication of other data-platform processes. As the tools for big data management are getting better, Jacob says he is still analyzing, building a new tool and buying a tool, and there is still time to make changes. While it is a good choice to build a custom tool when there is no choice, it is not always wise to buy something that is ready to save countless time and effort.