Big Data and Small Apps - An Irreversible New Wave

About big data, since last year, together with cloud computing, internet of things, 3D printing and the like all over the world, it has become a hot topic. But what exactly is big data? What exactly is big data? What should we do with big data? What kind of changes will it bring to our lives? The discussion of these issues is ongoing and many companies are thinking about how The application of big data in the enterprise IT construction, to achieve business operations innovation.

Big data, Baidu's definition is: refers to the amount of information involved is so large that it can not be captured, managed, processed and organized in a reasonable time by the current mainstream software tools to help enterprises run Information for making more positive ends.

Gartner gives this definition. Big Data is a massive, high-growth, and diverse information asset that requires new processing models for greater decision-making, insight and discovery and process optimization.

IBM 4V description of big data features is generally accepted by the industry: (1) Volume, huge volume of data. From the TB level, jumped to the PB level; (2) Variety, a wide range of data types. It includes not only traditional formatted data, but also weblogs, videos, photos, geo-location information, etc. from the Internet. (3) Value, low value density, high commercial value. Taking video as an example, the data that may be useful in continuous uninterrupted monitoring is only one or two seconds. (4) Velocity, processing speed. 1 second law. This last point is also essentially different from traditional data mining techniques.

If you simply click on the four characteristics to understand big data, big data may be understood as full data or holographic data. And such data applications, it seems only in the very large or large projects can be built up, and these and the traditional data warehouse What is the difference?

The three big data features given by Schoenberg, one of the first data scientists to see the trends in the Big Data era, may give us a better understanding of big data. Schoenberg's Big Data features can be described in three terms: more, more chaotic, and more relevant.

More here is for the research object itself, to consider the more object-related dimensions of information, rather than the traditional internal information, such as operators in the study of customer off-grid forecast, not only the study of customers Billing data, you can also add customer location information, and even speech information on the SNS network. So, big data is not always full (and who can define exactly what the full amount is?), But only increasing "more."

More chaotic, is the data collected more noise, and even in the study of a problem will have a greater disturbance to the prediction of the data dimension. This requires the use of the Internet's "trial and error" thinking, continuous study of possible noise in the acquisition and data processing, practice repeatedly, in the big data out of the most useful "small data." As mentioned above, oil price has long been known for application. One of the developers' experience is the constant adjustment of the contextual processing of SNS text messages. The noise removed includes the interference of other topics on oil prices, etc., which makes the small data set more accurate . For example, a related big V when discussing the issue of taxi prices said that if the price of a taxi goes up, then the price of oil must have risen. The human brain can quickly determine such a language, the theme is to talk about the issue of taxi prices, and the machine is hard to understand this. If you get the message of rising oil prices from such sentences, then judging the entire oil price is a distraction.

Relevance, is to find the correlation between the data, the development of the research object to make a better prediction. An example of how Google engineers can predict an earlier flu than the U.S. official health department is a good illustration. Google's data engineers are not pathologists, they can not know what the causes of influenza are, but they are predicting the imminent arrival of influenza through some information related to the flu.

From the above three characteristics and examples, the application of big data is not only a big application such as national strategy and business strategy, but also can be continuously promoted by countless "small applications" closely related to our life. Go down the altar, into the real market applications.

The author of WeChat on a shared account, "oil prices known to know," pushed such a message: "Oil prices know early Friendship tips: According to the oil price public opinion tracking analysis, June 22 morning oil prices or up (probability of more than 70%), the increase of about At 100 yuan / ton. "The next day, oil prices have long been known to continue to suggest news of oil price increases, and given an increase of 0.1 yuan / liter, June 21, oil prices have known to announce the news already released by the NDRC oil price increase notice .

Oil prices already know another three days in advance forecast the oil price adjustment information, from the line, their forecast accuracy has exceeded 95%! This is a typical example of the application of big data, which is what the author saw, landing in China A good example of a big data applet.

However, as big data gradually come into our daily life, we should also clearly understand that the development of any technology is a process in which norms, systems, and applications continue to cooperate with each other and work together. The recent "Prism Gate" incident has given the public a cool understanding of big data. On June 17, the author wrote a remark on Weibo: "The Snowden Event finally mentioned the" data right "to public view. Who wants to live under the rule of Big Brother in 1984? Someone wants to become Big brother, but the public is not the last century, the first hurdle of big data or the first development breakpoint gradually emerged.

Therefore, as soon as possible to achieve "norms (systems), technology, application" should also be the responsibility of each practitioner. Manufacturers who manipulate big data technology can participate more in the study of basic norms, and application explorers can accumulate experience in the process of deepening applications and participate in the construction of basic theories. The relevant departments that bear the direction of state information should attach great importance to the construction of big data standards (systems). After all, this is not an industry or an enterprise that can accomplish it alone.

Small data applications, are turbulent influx of people into our lives, oil prices have known for a long time is a good example, with this exploration, I believe that the health of our lives have known, tourism has long known, traffic I already knew that stocks knew ... not far away.