Facebook talks Big Data: It's not enough to have Hadoop

"The Hadoop programming framework may be synonymous with the" Big data "movement, says Ken Rudin, >facebook analyst at Http://www.aliyun.com/zixun/aggregation/1560.html. But Hadoop is not the only tool for companies to gain business insights from the unstructured information that is stored on a large scale.

"There are a lot of big data beliefs that need to be questioned," Rudin says. "The problem is that Hadoop is a technology, but big data isn't about technology, and big data is about business needs." ”

"In fact, large data should include Hadoop and relational databases and any other technology appropriate to the task at hand." "he added.

Facebook's business model relies on the processing of its user information and activity data for more than 1 billion social media users to provide targeted advertising. But, "Hadoop is not always the best tool for what we want to do." "Rudin said.

For example, it makes sense to make extensive exploratory analysis of a dataset in Hadoop, but relational storage is better for the discovery of operational analysis.

Rudin says Hadoop is not good for finding the lowest level of detail in a data set, but relational databases are more meaningful for storing data for conversion and aggregation.

"The conclusion is to use the right technology for whatever you need." "he said.

Rudin also has another assumption that the simple behavior of analyzing large data provides valuable insights. "The problem is to come up with more brilliant answers to the question," he said. "It's still an art to figure out what's right." ”

Facebook has been focused on hiring the right people to run its analytics business, not only with a PhD in statistics, but also in business.

"When you're interviewing, don't just focus on ' how do we calculate this metric, '" Rudin says, instead giving them a business case study and asking them which are the most important indicators.

Companies should also try to foster "everyone analysis," Rudin said.

Facebook runs an internal "data camp", a two-week program that teaches employees to analyze. Rudin says product managers, designers, engineers, and even financial staff are present. "Everyone participates in the meaning of what you give to everyone a common language of data that they can use to discuss issues and problems." "he said.

Facebook has also shaken the organization of statisticians and business teams. If statisticians remain independent, they tend to "sit there and wait for requests from the business sector to respond to them" rather than proactively. However, if statisticians are placed in business units, "you will find multiple groups trying to solve problems redundantly." "he said.

Facebook has adopted an "embedded" model to put analysts on the business team, but they report to a higher level of analysts, which helps avoid duplication of work.