New York Times: Big Data Age comes

Introduction: The New York Times website published today that the "big Data Age" has come and that people with expertise in this area are facing many opportunities. The article points out that "big data" is affecting every area. For example, in business, economics and other areas, decision making is increasingly based on data and analysis rather than on experience and intuition, and the ability to anticipate "big data" has emerged in areas such as public health, economic development and economic forecasts. The following is the full text of this article.

Are you good at numbers? You're fascinated by the data? Then the sound you hear is the chance of knocking at the door.

As a newly-graduated Yale MBA (MBA), Zhou Me was hired by IBM last summer to join the company's rapidly growing data-advisor team. The IBM Data Advisor's job is to help companies understand the meaning behind data explosions-network traffic and social-network reviews, and monitoring shipments, software and sensors for suppliers and customers-to guide decisions, cut costs, and increase sales. "I've always loved numbers. Zhou Me said she was a data analyst and was in line with the skills she had learned.

To develop a torrent of data, America will need many people like her. According to a report released last year by McKinsey Global, a consultancy at McKinsey's research department, the U.S. needs 140,000 to 190,000 workers with "deep analysis" expertise, and 1.5 million more data-savvy managers, Cato. Whether it is a retired person or an employed person.

The impact of data flooding is far beyond the corporate world. Justin Grimmer, for example, is a new generation of political scientists, aged 28 and assistant professor at Stanford University, Grimaud. In college and graduate studies, he links mathematics to political science, saying he sees "an opportunity because discipline is becoming increasingly data-intensive". His research involves computer-automated analysis of blog posts, congressional speeches and press releases, hoping to gain insight into how political ideas spread.

In many other areas, such as science and sports, advertising and public health, there is a similar situation-a shift towards data-driven discovery and decision-making. "This is a revolution, and we are really doing it," said Gary King, dean of the School of Quantitative Social Sciences at Harvard University (Cato for Quantitative Social). The quantitative changes brought about by the huge new data sources will spread rapidly in academia, business and politics. No area will be affected. ”

Welcome to the "Big Data Era" (age of "great"). Silicon Valley's upstart--initially at Google, and now in Facebook--is proficient in managing the relationship between Web data (Web search, postings, and information) and internet advertising. At the World Economic Forum held last month in Davos, Switzerland, big data was one of the themes of the discussion. The report, published in the forum entitled Big Data, big Impact, says data has become a new category of economic assets, like money or gold.

"One of the days of my Life" (Smolan Rick Smolan), the creator of the series of photographs, is planning to launch a new project later this year, the "Big Data Human Face" (The Human Data) will record the collection and use of the information. Smolan, a fanatic, believes that "big data" has the potential to be a "human dashboard"-an intelligent tool that can help people fight poverty, crime and pollution. Private-sector advocates, however, hold a pessimistic view, warning that "big data" and "dictator" (great Brother) expatiating, just put on the corporate cloak.

What is "Big data"? This, of course, is a glossary of cultural genes and marketing ideas, but it also reflects the evolving trend in the field of science and technology, which opens the door to a new way of understanding the world and making decisions. According to IDC, a technology research firm, the data has been growing at a rate of 50% a year, in other words, one-fold every two years. This is not a question of a simple increase in data, but a new issue. For example, in today's global industrial equipment, automobiles, electronic instruments and shipping boxes, there are countless digital sensors, which can measure and communicate the location, motion, vibration, temperature and humidity data, and even to measure the chemical changes in the air.

Connect these AC sensors to computational intelligence, and you'll see the so-called "IoT" (Internet of Things) or "Industrial Internet" (Industrial Internet). Making progress on the issue of access to information is also one of the reasons for the development of the "big data" trend. For example, government data-employment data and other information-have been steadily shifting to the network. In the 2009 years, the U.S. government opened the door to data by launching the Data.gov Web site, which provides a wide range of government data to the public.

Data is not just becoming more usable, but it is becoming more easily understood by computers. Much of the data added to the "big data" trend is generated in the natural environment, such as uncontrolled things like online speech, pictures and videos, and data from sensors. These are so-called "unstructured data" that are often not used in traditional databases.

But computer tools aimed at acquiring knowledge and insight from the vast "treasures" of unstructured data in the Internet age are rapidly evolving. At the forefront of this tool development is the rapid progress of artificial intelligence (AI) technologies, such as natural language processing, pattern recognition and machine learning.

These artificial intelligence techniques can be applied in many fields. For example, Google's search and advertising business and its experimental robotic cars all use a lot of artificial intelligence technology. On the California State Road, Google's robotic car has run thousands of miles. Both of Google's businesses have deterred the challenge of the "Big Data" era, which analyses large numbers of data and makes instantaneous decisions.

In turn, a great deal of new data is accelerating progress in computing, a virtuous circle in the "Big Data" era. For example, machine learning algorithms can be based on data to learn, the more data machines can learn more. In an example of Apple's Iphohne mobile phone Siri, which was launched last fall, the application could be traced back to a research project in the Pentagon, which was then detached and became a Silicon Valley start-up company. Apple acquired Siri in 2010 and continues to provide it with more data. Today, with hundreds of thousands of questions to offer, Siri is becoming an increasingly skilled personal assistant, providing services such as reminders, weather forecasts, catering advice, and answers to a number of questions.

Erik Brynjolfsson, a professor of economics at the MIT Sloan School of Management, says that if you want to understand the potential impact of "big data", take a look at the microscope example. Lunorfson The microscope was invented four centuries ago to allow people to see and measure things they had never seen before-at the cellular level. A microscope is a revolution in the field of measurement.

Lunorfson explains that data measurements are the equivalent of a modern version of a microscope. For example, Google Search, Facebook posts and Twitter messages make it possible to measure people's behavior and emotions in detail.

Lunorfson further points out that, in business, economic and other areas, decision making will increasingly be based on data and analysis rather than on experience and intuition. "We can begin to become far more scientific. He said.

There is a lot of anecdotal evidence that data-orientated thinking will bring high returns. One of the most notable examples is still the book "Penalty Gold" (Moneyball), published by Michael Lewis in 2003, The book records how low budget Oakland sports teams use analyzed data and arcane baseball statistics to find baseball players who are judged to be too low. Before the movie "Penalty Gold" starring Brad Pitt Pitt was moved to the screen last year, deep data analysis had not only become a standard in the field of baseball, but also in other sports such as the British Football League.

Retailers such as Wal-Mart (WMT) and Kohl are also starting to analyze sales, pricing and economics, demographics and weather data to select the right shelves in a particular chain of stores and based on these analyses to determine the timing of a sale. Freight companies such as UPS are also analyzing the delivery time and traffic patterns of trucks to fine-tune their transport routes.

Dating sites such as Match.com often take a closer look at the personal features, responses, and exchange information listed on their websites to improve their algorithms to provide better pairing for men and women who want to date. Throughout the United States, the Police department, headed by New York, is also using computerized maps and analyses of variables such as historical arrest patterns, payday, sports, rainfall and holidays, in an attempt to anticipate "hot spots" where the most likely crimes are to occur, and to advance the deployment of police forces in those areas.

Lunorfson and his two colleagues published a study last year saying data-directed management activities are spreading across the corporate world, and that management activity is starting to pay off. The three academics studied 179 large companies and found that companies using the "data-driven decision" model could raise their productivity by 5% to 6%, a productivity improvement that was difficult to explain with other factors.

In areas such as public health, economic development and economic forecasts, the ability to anticipate "big data" is being developed and emerging. The researchers found that there was an increase in the number of search queries on google for words such as "flu symptoms" and "flu treatments", and that a few weeks later there was an increase in the numbers of flu patients attending a hospital emergency room in a given area (and it should be noted that The emergency room of the hospital usually releases a report more than two weeks at the time of the patient's visit.

The United Nations has launched a new project called Global Pulse, which wants to use "big data" to boost global economic growth. The United Nations will conduct so-called "sentiment analysis" using natural language decryption software to analyze information in social networking sites and text messages to help predict unemployment, spending cuts or disease outbreaks in a given region, with the goal of using digital early warning signals to guide aid projects in advance, To prevent a region from falling back into poverty.

In the field of economic forecasting, there have been studies showing that, compared with the predictions made by real estate economists, the trend of increasing or decreasing the amount of search queries in the house is more accurate to predict the housing market trend in the next quarter. The Fed and other agencies have noted this. Last July, the National Bureau of Economic Research (according of Economic) hosted a seminar on "Opportunities in the big Data age" and its impact on the economic sector.

"Big Data" has also changed research into how social networks work. In the 1960s, Stanley-Milgram of Harvard University used parcels as a research medium for a famous experiment related to social networking. He sent parcels to volunteers in the Midwest to instruct them on how to bring parcels to strangers in Boston, but not directly; if the participants wanted to deliver the package by mail, the target would be someone they knew. The results showed that the average number of hand-swapping in a parcel was very low, only about 6 times. This is a classic interpretation of the so-called "Small World phenomenon", which forms the popular word for "Six Degrees of Separation" (six Degrees of separation).

Today, the content of social networking research involves how to collect a large collection of digitized data to illustrate the collective behavior on the network. The results of this study suggest that people you know but don't often contact--known in sociology as the "weak link" (weak ties)--are the best sources of job vacancy gossip, because these people are walking through a slightly different social world than close friends, So you can see the opportunities that you and your best friends are not able to see.

In the exchange of topics, researchers can also see their impact patterns and peaks-for example, by tracking the trend labels on Twitter. For a large number of users, Twitter, the online "Glass tank" is a window into the behavior of the reality. Jon Kleinberg, professor at Cornell University, said: "What I'm looking for is the ' hot spot ' in the data, which is an activity that I need to understand," Klenberg. Only by ' Big data ' can you do it. ”

There is no doubt that "big data" itself has some risks. Statisticians and computer scientists point out that the combination of "big data" and high-density measurements will increase the risk of "false discovery". Hasti Trevor Hastie, a professor of statistics at Stanford University, said that if a meaningful "needle" were to be found in the vast data "haystack", the problem would be "many straws look like needles".

In addition, "Big Data" also provides more raw materials for statistical hoaxes and biased fact-finding activities. "Big Data" provides a High-tech tool for an old trick--I know the truth, now let's find the facts. Goulding, a mathematician at George Mason University, Rebecca Goldin, said it was "one of the most harmful ways to use data."

The data has been tamed and understood by computers and mathematical models, which are like metaphorical rhetoric in literature, a simplified way of interpreting. These patterns are useful for understanding data, but they also have limitations. Private sector advocacy groups have warned that a web-based search model could find a correlation that would make unfair or discriminatory statistical inferences about the effects of products, bank loans and health insurance provided by pension funds.

Despite this warning, the advent of the "Big Data" era appears to be irreversible. The data has been seated in the driver's seat, where it is useful and valuable and even fashionable.

Friends have long been tired of talking about their work, but now they suddenly become curious, senior data analysts say. "Penalty gold" is one of the reasons for this change, but the real reason is far from so simple, analysts say. "Culture has changed," said Andrew Gelman, a statistician and political scientist at Columbia University, Germann. The idea now is that numbers and statistics are fun and a cool thing. ”

(Responsible editor: The good of the Legacy)