Big Data era coming opportunity knocking

Are you good with numbers? Obsessed with data? So what you hear is the opportunity to knock on the door.

Zhou Mu, Mo Zhou, who had just completed his MBA at Yale last summer, was snatched by IBM and joined the fast-growing data advisor of the technology company. They help companies figure out the meaning of the data explosion web traffic, comments on social networks, and monitoring data on goods, suppliers and customers ' software and sensors to provide decision guides, cut costs, and boost sales. "I've always had a passion for numbers," Miss said. She is a data analyst and this position matches her skills.

To exploit this torrent of data, America needs a lot of people like her. Last year, a survey by the McKinsey Global Institute of McKinsey, a consultancy, predicts that the United States needs 140,000 to 190,000 people with "deep analysis" expertise, and that the demand for managers with data knowledge is more than 1.5 million, both in recruitment and retraining.

The impact of data enrichment extends beyond business. Justin Grimmer, for example, is a member of the new generation of political scholars. As a 28-year-old assistant professor at Stanford University, he saw "an opportunity because discipline is becoming more and more data intensive", so he applied mathematics to political science in his own university and graduate studies. His research includes automated computer analysis of blog postings, congressional speeches and press releases, and news content in order to learn more about how political ideas are being disseminated.

Other areas, such as science, sports, advertising, and public health, are also similar-trends in data-driven discovery and decision-making. "It was a revolution," said Gary King, director of the Harvard Institute for Quantitative Social Sciences. But with the support of a huge new source of data, the pace of quantification will go all the way to academia, business and government. No field can be touched. ”

Welcome to the big Data age. The new darling of Silicon Valley, formerly Google and Facebook, is a master of Web data--all of which are adept at putting internet ads on online searches, articles and messages. Last month, at the World Economic Forum in Davos, Switzerland, big data was one of the framed themes. A report from the forum, big data, big impact, declares the data to be a new kind of economic asset, like money or gold.

Rick Smolland (Rick Smolan), author of a Day of life, is planning to launch a project called "The Human face of Big Data" later this year. Mr Smolan is a fanatic, and pro data could become a "human dashboard" that can help fight poverty, crime and pollution as an intelligent tool. Privacy advocates are skeptical, warning that large data is the Big Brother (note: Brother, who has seen George Orwell's "1984", who is not unfamiliar with the ' Great Brother '), is just putting on a corporate coat.

What is big data? This is a cultural gene (meme), a marketing term that is true, but also a generalization of trends in technology, a trend that opens the door to new ways of understanding the world and making decisions. According to IDC, a technology research institute, a large amount of new data is emerging, growing at a rate of 50% a year, or doubling every two years. It is not just that the torrent of data is growing, but also that there will be more and more new tributaries. For example, there are countless digital sensors worldwide attached to industrial equipment, automobiles, electric meters and crates. They can determine the direction, movement, vibration, temperature, humidity, and even chemical changes in the atmosphere, and can communicate.

By connecting these communication sensors with computational intelligence, you can see the rise of the so-called IoT (Internet of Things) or the industrial Internet (Industrial Internet). Improved access to information also contributed to large data trends. Other information, such as government data-employment figures, is being steadily transplanted to the web. In 2009, Washington opened the door to the data by launching Data.gov, which opened up various government data to the public.

Not only is data becoming more common, but it is also becoming more readable for computers. Most of the big data waves are wilder-all things that are hard to control, like the web and the text, images, and videos of the sensing data stream. This is known as unstructured data, and is usually not an abdominal object in a traditional database.

However, computer tools that harvest knowledge and insights from the vastness of the internet age of unstructured data are rapidly gaining popularity. At the forefront are the rapidly evolving AI technologies, such as natural language processing, pattern recognition, and machine learning.

Those artificial intelligence techniques can be applied to many fields. Google's search and advertising business, for example, and its experimental robotic car, which has been riding thousands of miles in California, use a whole host of artificial intelligence techniques. These are daunting big data challenges that need to parse a lot of data and make decisions right away.

In turn, the abundance of new data accelerates the progress of computing-a virtuous circle of large data. For example, the machine learning algorithm is learned from the data, the more data, the more machine learning. Let's take the example of the iphone conversation and question and answer app that Apple introduced in the last fall. The application's origins go back to a Pentagon research project and then split up a Silicon Valley start-up. Apple acquired Siri in 2010 and kept feeding it data. Now, with millions of questions being offered, Siri is becoming an increasingly sophisticated personal assistant, offering reminders, weather forecasts, hotel advice, and other services to iphone users, and the number of questions answered is expanding in the universe.

To understand the potential impact of big data, you have to look at the microscope, said Erik Brynjolfsson, an economist at MIT Sloan School of Management. The microscope, invented 4 centuries ago, allows people to watch and measure things--cell levels--at an unprecedented level. This is a revolution in measurement.

Professor Brynjolfsson explains that the measurement of data is the modern equivalent of a microscope. Google's search, Facebook's articles and Twitter messages, for example, make it possible to measure the behavior and emotions in a subtle way.

In other areas of business and economics, Brynjolfsson says, decisions will increasingly be based on data and analysis rather than on experience and intuition. "We can start a lot more scientifically," he commented.

There is plenty of anecdotal evidence that data-first thinking is rewarding. The most famous is still the penalty gold (Moneyball), the Michael Louise (Michael Lewis) 2003 book, which records how the Oakland sports team (Oakland a), with little budget, uses data and arcane baseball statistics to identify the story of an undervalued player. A great deal of data analysis has not only become the standard of baseball, but it is also the norm in other sports, including soccer, which was done long before the release of the same film starring Brad Pitt last year.

Retailers, such as Wal-Mart and Kohl ' s, analyze sales, pricing and economic, demographic, and weather data to select the right product for a particular store and determine the timing of the price cut. Logistics companies, such as UPS, excavate data on delivery time and traffic patterns to adjust the route.

Online dating services, like Match.com, are constantly looking at their personal characteristics, responses, and web lists of communication to improve the algorithms for dating between men and women. Under the leadership of the New York Police Department, national police stations are using computerized maps and analysing variables such as historical crime patterns, payday, sporting events, rainfall and holidays, with a view to predicting possible crime "hotspots" and deploying police forces in those places.

The study, conducted by Professor Brynjolfsson with two other colleagues, was published last year, which argues that data to guide management is spreading across the U.S. business community and is beginning to take effect. They studied 179 large companies and found that those who adopted "data-driven decision making" had a 5%-6% higher productivity than those that were explained by other factors.

The ability to predict large data is also being explored, and there is hope for success in areas such as public health, economic development and economic forecasting. Researchers have found that keywords such as "flu symptoms" and "flu treatments" in Google's search requests peak more than two or three weeks earlier than an increase in emergency room influenza in a regional hospital (and emergency room reports tend to be about two weeks slower than browsing).

Global Pulse, a new UN-sponsored action plan, hopes big data will leverage global growth. The group will use natural language deciphering software to conduct so-called emotional analyses of messages and text messages in social networks-to help predict unemployment, spending cuts or outbreaks in specific areas. The goal is to use digital early warning signals to advance aid planning, for example, to prevent backsliding in one region from poverty.

Research shows that in terms of economic forecasts, the increase or decrease in search volume on Google's property is a more accurate predictor of real estate economists ' predictions. The Fed, and others, have noted this. Last July, the National Bureau of Economic Research hosted a seminar entitled "Opportunities for Big Data" to explore its impact on economic majors.

Big data has shifted research into how social networks work. In the 60 's, in a famous social relations experiment, Milgram of Harvard University (Stanley Milgram) used parcels as their research medium. He sent parcels to volunteers in the Midwest to instruct them to send parcels to strangers in Boston, but not directly, and participants could only send parcels to someone they knew. The average number of times a parcel changed hands was unusually small, about 6 times. This is the "Small World phenomenon" of a classic embodiment, which also formed a popular phrase "six Degrees of separation."

Today, social networking research includes a digital dataset that digs out huge amounts of online collective behavior. Among the findings are: people you know but don't often contact-also known as "weak links" in sociology-are the best sources of internal information about job openings. They travel in a social world that is slightly different from your close friends ' circle, so you can see opportunities that you and your best friend can't see.

The researchers were able to see the pattern of the impact and know when the conversation on a topic was hottest-take the tagging trend of Twitter. This online transparent glass aquarium is a window into the real-time behavior of a huge crowd. "I need to understand the eruption of an activity and I'm looking for a hot spot in the data," says Prof Jon Kleinberg of Cornell University. "You can only do this with big data." ”

Admittedly, large data itself risks. Statisticians and computer scientists point out that massive datasets and fine-grained measurements can lead to an increased risk of "error detection". Trevor Hastie, a professor of statistics at Stanford University, says the trouble with looking for a meaningful needle in a massive data haystack is that "many straws look like needles (note: Well, it looks more difficult than a needle in a haystack because of the thousand people)".

Large data also provide more raw materials for parody and biased fact-finding studies. That's the old trick-the fact that I already know, now let's find out, the new trick-high tech. Rebecca Goldin, a mathematician at George Mason University, says this is one of the most harmful ways to use data. ”

With the use of computers and mathematical models, we have tamed and understood the data. These models, like the metaphor of literature, are a simplification of interpretation. They are useful for understanding, but there are limitations. Privacy advocates warn that, according to an online survey, the model has the potential to derive an unfair or discriminatory correlation and statistical inference that affects someone's products, bank loans and health insurance.

Despite these caveats, the trend seems to be irreversible. The data is sitting on the driver's seat. It's there, it's useful, it's valuable, it's even fashionable.

Senior data analyst, people who have long been bored by their friends when they talk about their work are suddenly becoming curious. These analysts say this is thanks to "penalty gold", but the reality is far from the case. "Culture has changed," said Andrew Gelman, a statistical and political scientist at Columbia University: "People think numbers and statistics are fun and fun." Now it's cool stuff. ”

(Responsible editor: The good of the Legacy)