Large data concepts are not without solutions

When we held a scientific symposium on big Data last October, one representative said that so many experts had been talking for a day that even the concept of what was big data was not clear. I said no. We interpret this concept from different angles. As to the concept of "culture", ask 100 people immediately, can you come up with a unified answer? Maybe 100 people have 100 answers. But that does not mean that there is no consensus on the concept. Similarly, when it comes to the concept of spirituality, although everyone can understand pretty, it is difficult for 100 of us to give the same standard answer. If there is no absolute authority, such as the "Beijing Spirit", to give large data to clear a unique concept, we will be on the basis of pretty still each statement, each said the words.

Our research results show that the root of the phenomenon and concept of large data can be divided into three stages. One is from the 1980s to the middle of 90, it is the embryonic stage of large data cognition. 1980, the famous American futurist Toffler in the third wave of the book, the Big Data hailed as "the third wave of the CLS movement." The second is from the mid 1990s to the first 10 years of 21st century, is a big data widely concerned about the stage. Douglas Laney, analyst at Gartner Group, Grass Lanny a relatively clear definition of large data for the first time in 2001, emphasizing that large data must have 3V characteristics, namely, large capacity, diversification and speed. The third is 2010 to present, is the big Data strategy application is put on the schedule and the rapid development stage. The President's Scientific and Technical Advisory Council reported to President Obama and Congress in 2010 on the future of planning for digitization. McKinsey released its report on Big data: The next frontier of innovation, competition and productivity in 2011. 2012 is an important year. January, Davos, Switzerland 9374.html "Big Data Big Impact" published by the U.S. Obama administration in March, and the United Nations Secretary-General's Executive Office issued a report on "Big Data development: challenges and opportunities" in May; June, The OECD Statistical Commission at its 9th session issued a study on the use of large data for decision-making. 2013 can be called the big data of Chinese statistics year. July, "Large data Age statistics: Opportunities and challenges-the high-end forum for Chinese Statistics" held at Shanghai University of Finance and economics; in October, the 17th National Statistical Scientific Symposium on the theme "Statistics in the context of large data" took place in Hangzhou; November, the National Bureau of Statistics and Ali, Baidu and other 11 enterprises signed a large data strategic cooperation framework Agreement.

What is big Data? The McKinsey report defines "large data as a data base that is larger than the ability to crawl, store, manage, and analyze traditional database software tools." Wikipedia's statement is that "big data is difficult to use existing database management tools to deal with the characteristics of a large number of features and complexity of data integration." "The Shiji defines large data as those that are larger than the traditional scale, where the general software tool is difficult to capture, store, manage, and analyze data, and think it should generally be a" byte "order of magnitude. Mr. Ma, the founder of Ali, says big data is a service. And our colleagues in the study put forward: large data refers to the use of a variety of data collection methods, the collection of different data sources, through the use of modern information technology and architecture to high-speed analysis and processing of highly applied value and decision support functions of multiple types of data and technology integration.

From the perspective of existence: large data is divided into structured data that can be reflected by two-dimensional tables and unstructured data that cannot be reflected in two-dimensional tables, such as audio, video, pictures, etc. From the data source: Large data can be divided into administrative record data, business record data, Internet and search engine data three major categories. The administrative record data includes the personal information record data, the unit information record data and the natural and the resource record data and so on, the business record data includes the electronic Commerce transaction data, the enterprise production operation data and the information consultation report data and so on; The characteristics of large data, from the initial 3V has been summed up as 6V plus 1 c. The data volume is large (Volume), type diversity (produced), processing speed (velocity), the application value is great (value), the way of data acquisition and transmission is free and flexible (vender), accuracy (veracity) and processing and analysis is very difficult ( complexity).

Compared with traditional data, large data is automatic or semi-automatic generation, data collection, processing, storage and analysis capabilities have been greatly improved, data subjects and sources are becoming more and more diverse, unstructured data accounted for the vast majority of the need for a large number of filtering to extract useful value;