Large data technology exists limited empirical intuition indispensable

Large data will be a new chapter in the history of human business and could replace ideas, paradigms, organizations and ways in which people think about the world, according to a print edition of The New York Times, published 30th. But at the same time, experience and intuition are indispensable.

The following is a summary of the article content:

"Big data is important and intuition is essential." "It was the subject of an industry meeting held at MIT earlier this month.

Andrew McAfee, chief scientist at MIT's Digital business center, says big data will be a new chapter in the history of human business Andrew Macafi. Eric Brinjolsen Erik Brynjolfsson, another professor at the center, says big data will replace ideas, paradigms, organizations and ways in which people think about the world.

These avantgarde predictions are premised on the fact that data such as web browsing, sensor signals, GPS tracking, and social networking information can be opened to unprecedented levels of measurement and monitoring of human and equipment behavior. Computer algorithms can predict many things in humans, such as shopping, dating or voting.

Industry experts predict that the end result is: The world is becoming more and more intelligent, the efficiency of enterprises more and more high, consumers get more and more quality of service, people make decisions more and more reasonable.

I've written a lot about big data before, but at this particular moment at the end of 2012, I think it's time to reflect, ask questions and question big data.

It is not new to excavate practical revelation from business evaluation. More than 100 years ago, Winslow (Frederick Winslow Taylor) 's masterpiece "Scientific Management Principle" is the predecessor of large data. Taylor's assessment tool is a stopwatch, which is timed and monitored for each employee's actions. Taylor and his aides used the "time and action" model to redesign the most effective way to work.

But if this method is exaggerated, it becomes the object of the irony of Chaplin's Modern Age (Xiandai times). Since then, the enthusiasm for this quantitative approach has also begun to fluctuate.

Typically, the internet has been used by large data advocates as an example of a successful data business, which is represented by Google. Today, many large data technologies, such as mathematical models, predictive algorithms and artificial intelligence software, have been widely used by Wall Street.

At the Massachusetts Institute of Technology this month, when asked about major failure cases in the Big data field, few people could say such a failure. Later, Robert Legeburn (Roberto Rigobon), a professor at the MIT Sloan School of Management (Sloan parochial of Management), said the financial crisis had no doubt affected the data business. "Hedge funds have failed all over the world," he said. ”

The problem is that the mathematical model is a simplification. This model is derived from the natural sciences, and according to the laws of physics, particle behavior in fluids can be predicted.

In so many large data applications, a mathematical model usually comes with accurate data about human behavior, interests, and preferences. The dangers of this approach in finance and other fields are evident, Mannuel Derman, director of the Department of Financial Engineering at Colunya University in the United States, Emanuel Derman in his book Models. Behaving. Badly the danger in detail.

"You can cheat yourself with data, and I'm worried about bubbles in big data," said Claudia Perlich, Media6degrees chief scientist at New York start-up. "He feared that many people called themselves" data scientists "but did not do enough homework to discredit the field.

The big data, Mr. Cooper Litcher, appears to be facing a labour bottleneck. "Our skills are not up to speed enough," she said. "The US needs 140,000 to 190,000 workers with" in-depth analysis "and 1.5 million more data-literate managers, whether retired or employed, according to a report published last year by McKinsey global Cato.

Thomas H. Davenport, a visiting professor at Harvard Business School, is writing a new book titled Keeping Up with the Quants Thomas Davenport to help managers deal with big data challenges. Davenport that an important part of managing large data projects is to ask the right question: How do you define the problem? What data do you need?

If modelers can think about issues such as ethical dimensions (ethical dimensions), it will serve society better, says Rachel Schutt, senior statisticians at Google Research (Rechel Chatt). "Models are not just predictions, they can actually make things happen," says chart. ”

Models can create data what scientists call "behavioral loops" (behavioral loop), if a person is provided with enough data, can guide their behavior.

Facebook, for example, uploads personal data to its Facebook page, and Facebook's software tracks your clicks and searches. The algorithm is used to evaluate the data and then provide a friend's advice.

But the behavior of tracking users through the software has caused privacy concerns, is the big data will usher in digital monitoring the arrival?

My personal biggest concern is that the current algorithm for determining our personal digital world is too simple and not smart enough. This is one of the issues discussed by Ailly Parice (Eli pariser), "The Filter bubble:what the Internet is hiding from."

Encouragingly, these thoughtful data scientists, like Percy and Chet, are aware of the limitations and deficiencies of large data technologies. They believe that listening to data is important, but experience and intuition are equally important.

At the Massachusetts Institute of Technology conference, Chart was asked how to become a good data scientist, she said, the need for computer science and math skills, with curiosity, innovative, with data and experience for action guidelines. "I'm not going to hallow the machine," she said. ”