The deep meaning of large data

Wen/Liu

Random forests, naïve Bayesian estimators, http://www.aliyun.com/zixun/aggregation/14172.html ">restful services, Gossip
Protocols, eventual consistency, data sharding, Anti-entropy, Byzantine
Quorum, erasure job, vector clocks ...

Can you guess where this string of dense terminology comes from?

This is the first letter from Amazon CEO Bezos to shareholders 2010 years ago. "Walking into an Amazon conference room, you may suddenly think you've strayed into a computer science lecture." Almost all of this letter is about technology, with the focus on large data processing. Data has become a new era of oil, large data processing capacity, indeed has become a competitive focus of enterprises.

In my August 2011 volume, I have sorted out the emergence of big data, the first in 2005 when Tim O ' Reilly presented the Web 2.0 concept. However, in writing this issue of the "Hall of Fame" Jim Gray article, I read a lot of information and found that things are far less simple.

As early as the 1940 's, cybernetics's father, Wiener, had begun to discuss a machine that could collect enough of the various types of information, produce, market, and human psychology, and then determine the probability of what happened. At that time, computers were not yet born.

Jim Gray recalls that he had worked with some colleagues to apply computer science to social issues when he was studying for a PhD in Berkeley before 1969. This is one of the themes that he has been studying since. His project name at Microsoft Research is called escience, and much of the work has yielded fruitful results by opening Microsoft's computing resources to academic peers in other disciplines to address those data-intensive topics.

In 2007, he went on a few months ago, in a speech to the National Scientific Research Council, pointing out that scientific research had entered the fourth stage-data exploration-after thousands of years of experience, hundreds of years of theoretical modelling and computational simulations decades ago. At this stage, scientists rely on a variety of instruments, sensors to obtain data, or through simulation to generate data, and then use the software for processing, the information/knowledge to be stored in the computer, and then by the scientists with a variety of statistical and data tools for analysis and visualization. This is basically the classic meaning of large data processing.

These days I'm looking through a 2007-year bestseller, Super crunchers, a popular brochure that is very important for data analysis. The rich examples in the book will give you an idea of the ubiquity of big data: predicting the quality of red wine, choosing baseball players, taking titles, judge justly, looking for objects ...

Large data analysis is often more reliable than an expert or yourself. The most impressive case is evidence-based medicine, frankly speaking, the traditional experience of the accumulated medical diagnosis and treatment of many practices and procedures, there is no data support, there is a great risk, should be used as much as possible statistical data for argument.

In part of the hospital experiments, more than a year to save the lives of 100,000 people.

This also reminds me of the premature death of the Xiaoxiang teacher, he died before the regular physical examination did not detect the problem, feel not timely and not pay attention to, missed the timely treatment. This tragedy should be avoided if we can develop appropriate technologies, monitor the vital organs of each person with tiny sensors, collect data continuously, analyze them in time, and warn them of danger.

Jim Gray has predicted that by 2047 all the information about real things, people, buildings, processes will be online. Let us work together to achieve it as soon as possible.

(Responsible editor: The good of the Legacy)