Big data: The stormy waters of the mobile internet age

We are not sure whether everything is the same, but in the mobile internet age, at least, people have pushed through the door: to gain business and social value through efficient analysis of massive amounts of data. Large data has brought new value to the mobile Internet, and laid the foundation for moving towards things networking.

A data analyst appears on the stage as a "conductor". With the forceful waving of his arm, a string of data was tamed. He was a man of his own, but with more data and more ferocity. He's a little hard, he swings around, he's scrambling! A wave of waves came over, and he stood as skiff annihilated in the sea of data ...

The same scene, another analyst appeared, or the waving arm, or a huge amount of data on the face, but he is at ease, calm and easy to command, as if it is not messy data, but the music master composed of the handed down movement.

On the July 13, 2012, in the "Big Data World Forum" at the landmark mansion, Dr Zhang Lei, chief consultant of SAS, played an exciting video before his speech. The whole audience was at a full battle.

Mobile Internet ushered in big data

"It's the most exciting moment for data people like me, because databases and big data have become the center of change, and in fact can be a revolution in it, manufacturing, retailing, government management, technology, and big data that changes the way the world works." So what we call the new world of big data. ”

Sun Boke as if still immersed in just the atmosphere. He is Microsoft Asia Pacific Research and Development Group chief technology officer. Sun Boke says there are several major factors driving big data changes, such as low cost storage and cloud computing capabilities, such as "The explosion of mobile devices everyone realizes, everybody has one or more mobile devices, and now the world's mobile device users may have reached 5.5 billion." "And," it's not just about web users, it's about sensors, we're expecting 10 billion of our sensors to connect to the Web, all of which brings us to new changes in big data. ”

In other words, writing this digital note is not Schubert, Mozart, but the mobile internet age of every netizen and every smart phone, computer, sensor. Large data is the resulting unstructured data that is different from the traditional structured data, or the traditional technology cannot analyze the data of valuable results. It accounts for 85% of total data.

For the source of large data, Ling, director of China's Industry Cooperation and Solutions division of Intel, replaced a more concise version in addition to the data generated by social networks and traditional business "human-trader, human-communication", there is a class of data generated by machines and machines, existing smart device networks, This is going to be bigger, and the Internet is bound to be an inevitable trend, and as time goes on, people will see more and more data coming from the web of things.

Global data use will grow 44 times times by 2020, Ling said, and the main source of growth for 35.2ZB is big data.

Compared with traditional data, large data has the following four typical characteristics, namely diversity (produced), Volume (volume), velocity (velocity) and values (value), that is, it is complex structure, large volume and fast processing in one, And ultimately produce enormous commercial and social value.

As Forresterresearch senior analyst Cao Yuchin said, big data is not a whole new technology, "the big data is mainly to help enterprises to the existing data, the resulting data to do some integration, corresponding to make a small amount of input and get a greater return." We can even say that the traditional internet has produced a lot of big data, but the rapid development of mobile internet, cloud computing, and the initial application of IoT have powerfully promoted the advent of the big Data age, which is beyond doubt.

The difficulty of large data analysis

Imagine you have ordered a product in Jingdong mall or Amazon, then the machine will be your ID number, shipping address, cell phone, phone, e-mail and receipt time and so on all records. If you submit an item comment, or share it with a friend on a microblog, it is also recorded.

Insight into all this means the business of the dream.

Therefore, Sun Boke asserts: "Two companies, one company using large data technology while the other has not adopted, then the future of their financial situation will be significantly different, big data has become a competitive advantage to maintain the competitiveness of enterprises." ”

In Microsoft's view, he says, big data means managing the end-to-end lifecycle of large data, "manage data, how to capture, store, and protect security data; Next, how to protect your data, how to clean, discover the relevant data, how to connect other data with it, and finally, how to gain insight in the data. ”

But the problem is that analysis of big data is not easy.

Ling that traditional storage architectures need to be upgraded to extensible storage architectures, otherwise can not adapt to the existing large data storage, at the same time, for the support of the Distributed file system, the need for real-time flow processing, and "Traditional data analysis is more structured, the amount of data is limited, centralized processing, batch processing, Can not meet the needs. ”

Liu, general manager of the Software Research and Development (Beijing) Co., Ltd., said that in the big data age, data analysis means the key, "but the traditional data analysis capabilities, can not handle such a large number of data." When we analyze tens of millions of data, it will take dozens of hours to get the results. When your data reaches 1 billion, the software will not run, sometimes it will take dozens of days, this speed is unacceptable. ”

He cites some of the confusion caused by traditional analytical techniques. For example, because of the limitation of the analysis means, the sample number is not large enough to make full use of all the data, which destroys the integrity of the information, for example, limited by the analytical ability to get the answer to complex problems, limited by time and had to adopt a simple modeling technology, the same, Also because there is not enough time to perform multiple iterations, you have to compromise the model's clarity and so on.

Liu said that in the 50 's, American pilots found that they could win the war effectively by Ooda methods when battling with the enemy, that is, "If your decision is faster than your opponent, you can take advantage of it".

Here the speed was referred to the strategic height. The same is true of the analysis of large data.

Product friends Interaction is China's largest digital advertising internet technology companies, the first in advertising technology to adopt large data research methods, set up a number of Hadoop based cloud computing platform. Product has interactive CEO Huang in the "network leader" reporter interview, said, as the largest demand-side platform (DSP), Friends of the mutual processing of tens of billions of data per day, its DSP platform can butt AD trading platform (Adexchange) for real-time bidding and intelligent algorithm, 50 milliseconds to complete the response.

Visible, no speed, then valuable data can only be a pile of money can not be circulated.

(Responsible editor: Liu Fen)