Large data technology exists limited intuition indispensable

Lead: The New York Times print edition of the 30th article said that big data will be the history of human business in the chapter, is expected to replace ideas, examples, organizations and ways people think about the world. But at the same time, experience and intuition are indispensable.

The following is a summary of the article content:

"Big data is important and intuition is essential." "It was the subject of an industry meeting held at MIT earlier this month.

The big data will be a new chapter in the history of human business, said Andrew McAfee, chief scientist at the MIT Digital Business Center. Erik Brynjolfsson, another professor at the center, says big data will replace ideas, paradigms, organizations and ways in which people think about the world.

These avantgarde predictions are premised on the fact that data such as web browsing, sensor signals, GPS tracking, and social networking information can be opened to unprecedented levels of measurement and monitoring of human and equipment behavior. Computer algorithms can predict many things in humans, such as shopping, dating or voting.

Industry experts predict that the end result is: The world is becoming more and more intelligent, the efficiency of enterprises more and more high, consumers get more and more quality of service, people make decisions more and more reasonable.

I've written a lot about big data before, but at this particular moment at the end of 2012, I think it's time to reflect, ask questions and question big data.

It is not new to excavate practical revelation from business evaluation. More than 100 years ago, Frederick Winslow Taylor's masterpiece "Principles of scientific management" is the predecessor of large data. Taylor's assessment tool is a stopwatch, which is timed and monitored for each employee's actions. Taylor and his assistants use this "time and action" research model to redesign the most effective way to work.

But if this method is exaggerated, it becomes the object of the irony of Chaplin's Modern Age (Xiandai times). Since then, the enthusiasm for this quantitative approach has also begun to fluctuate.

Typically, the internet has been used by large data advocates as an example of a successful data business, which is represented by Google. Today, many large data technologies, such as mathematical models, predictive algorithms and artificial intelligence software, have been widely used by Wall Street.

At the Massachusetts Institute of Technology this month, when asked about major failure cases in the Big data field, few people could say such a failure. Later, Roberto Rigobon, a professor at MIT Sloan School of Management (Sloan parochial of Management), said the financial crisis had no doubt affected the data business. "Hedge funds have failed all over the world," he said. ”

The problem is that the mathematical model is a simplification. This model is derived from the natural sciences, and according to the laws of physics, particle behavior in fluids can be predicted.

In so many large data applications, a mathematical model usually comes with accurate data about human behavior, interests, and preferences. The dangers of this approach in finance and other fields are obvious, Emanuel Derman, director of the Department of Financial Engineering at Colunya University in the United States, Models in his book. Behaving. Badly the danger in detail.

"You can cheat yourself with data, I'm worried about bubbles in big data," says Claudia Perlich, chief scientist at Media6degrees, a new york-based start-up. "Perlich worry that many people call themselves" data scientists, "but do not do their homework, but discredit the field.

Perlich that big data appears to be facing a labor bottleneck. "Our skills are not up to speed enough," she said. "The US needs 140,000 to 190,000 workers with" in-depth analysis "and 1.5 million more data-literate managers, whether retired or employed, according to a report published last year by McKinsey global Cato.

Thomas H. Davenport, a visiting professor at Harvard Business School, is writing a new book titled "Keeping Up with the Quants" to help managers deal with big data challenges. Davenport that an important part of managing large data projects is to ask the right question: How do you define the problem? What data do you need?

If modelers can think about issues such as ethical dimensions (ethical dimensions), it will serve society better, says Rachel Schutt, senior statisticians at Google. "Models are not just predictions, they can actually make things happen," Schutt said. ”

Models can create data what scientists call "behavioral loops" (behavioral loop), if a person is provided with enough data, can guide their behavior.

Facebook, for example, uploads personal data to its Facebook page, and Facebook's software tracks your clicks and searches. The algorithm is used to evaluate the data and then provide a friend's advice.

But the behavior of tracking users through the software has caused privacy concerns, is the big data will usher in digital monitoring the arrival?

My personal biggest concern is that the current algorithm for determining our personal digital world is too simple and not smart enough. This is one of the issues discussed by Eli Pariser, "The Filter bubble:what the Internet is hiding".

Encouragingly, these thoughtful data scientists like Perlich and Schutt are aware of the limitations and deficiencies of large data technologies. They believe that listening to data is important, but experience and intuition are equally important.

At the Massachusetts Institute of Technology conference, Chart was asked how to become a good data scientist, she said, the need for computer science and math skills, with curiosity, innovative, with data and experience for action guidelines. "I'm not going to hallow the machine," she said. ”

(Responsible editor: The good of the Legacy)