Database cannot afford the most important data landing talent shortage

Founded in 2008, Vpon is a mobile advertising company whose main business is to accept advertisers to post ads on mobile apps launched by its partners. Last month, Vpon launched a system advertising analysis system called 3S (Sales supporting system), a large data mining, analysis and processing system built on the Hadoop platform. The most important function of this system is to be able to accurately statistic and analyze the time, place and various preferences of the user's click and move application, so as to help advertisers make more effective advertising decisions. Founded in 2009, Beijing Percentile Information Technology Co., Ltd. (%) is doing something similar to vpon. The percentage point introduces a tool called personalized search engines that E-commerce sites can use to analyze visitor clicks, identify visitor preferences, and recommend products.　　Similarly, the search engine is based on Hadoop development. There are a growing number of companies in the Internet arena, such as Vpon, which use Hadoop, and so on, to use large data-related techniques to conduct visitor-click behavior Analysis, including a vast array of start-ups that are sensitive to emerging technologies, as well as Google,　　A famous internet company like Facebook. In fact, the "Big Data", the data-analysis boom that originated in internet companies, has now gone beyond the internet, and some of the traditional companies that have dared to "taste it" have begun to deploy big data-related technologies. According to IDC's latest global Data Market Forecast, the big data will grow from $3.2 billion trillion in 2010 to $16.9 billion in 2015, with a composite growth rate of 40% per cent annually. Good expectations for the market are attracting investors, too, with more than $500 million trillion in investment in big data: Cloudera, the main publisher of the Hadoop version at the end of last year, received $40 million in investments; Cloudera's rival, another Hadoop version publisher,--mapr 25 million dollars in investment; NoSQL database manufacturer 10Gen (MongoDB's suppliers) and DataStax (Cassandra's suppliers) received 32 million dollars and 11 million US dollars respectively for financing; This April, just on the Nasdaq IPO, financing 230 million dollars ...　　This list can be set very long. It is clear that, in the eyes of market research institutions and investors, the big data market is now at its zenith, and its heat has even overshadowed the cloud. At the same time, "Big data is big cheat, big lie" of the argument also endless.　　In the chaos of the marketing campaign, how do we know the big data, big data is the big opportunity or a big scam? Ushered in the big data era large data and data explosive growth is closely related. According to IDC Research, the world's new data generated at the end of 2010 has reached 1.2 million PB (or 1.2 ZB, which can be stacked from the Earth to the moon (about 240,000 miles from the earth), if used on CD-ROM storage.　　IDC estimates that the amount of data to be stored by 2020 will reach 35 trillion GB, 42 times times the 2010 data storage. China today is a big country in data production. IDC also provided data, as of June 2012, China has nearly 390 million mobile customers, 530 million of Internet customers.　　In a more typical smart city, 200PB of video data is likely to be produced in each quarter. It should be said that the challenges we face are not just massive data, but a growing array of data formats, particularly unstructured and semi-structured data, far exceeding traditional structured data. Research shows that more than 80% of the new data today are unstructured or semi-structured data, such as logs, pictures, videos, emails, and so on.　　These data are not (or are not) used in traditional methods, need new thinking, new act, this is the big data technology. There is no clear and consistent definition of what large data is.　　There are two different interpretations of big data, one is to think of it as a series of technologies that deal with a large number of structured and unstructured data to get a variety of analysis and prediction results; another, and more people understand, simply called massive datasets called large data, this article uses the latter recognition. Although a clear definition of large data is lacking, there is a consensus on the three "V" features of large data, namely massive data scale (volume), Fast data flow (velocity), and diverse data types (produced).　　Among them, the "massive" is the big data to cause the widespread attention the premise, but the fast and the complex data type is the key which causes the people widespread attention. Big data from where to talk about big data, have to talk about mobile devices.　　Because the big data we face comes from business applications, operational data and supply chains, suppliers producing all kinds of data, and a large part of the social media and mobile apps, and mobile smart devices are one of the biggest drivers behind it. As we all know, the popularization of intelligent mobile devices brings many changes to society. One of them is that people can get information at any time, anywhere, to communicate, collaborate, and publish social content in real-time. This has led to a change in the way the data is produced: In the past, we were only producing data at work, and now we are producing data almost every moment of the day; in the past, the data were traded, typically in the form of a transaction, which is good for traditional databases, and today there are so many data sources, And a lot of data is no longer by people but by machines, all kinds of RFID, sensors are generating data.　　In addition, even man-made, such as social networks, microblogs, the form of data and the form of the past is not the same, the main manifestation of unstructured. "The amount of data has increased far beyondExpected, and the enterprise is currently facing a more complex data environment. In such an environment, enterprises need new methods to acquire traditional financial or financial information analysis ability. This is the big data hot background.　　"Cao Yuchin, a senior analyst at Forrester, said in a recent speech to the Big Data Forum. In the face of such a large, complex data needs to be stored, need to manage more need to analyze, this raises the big data of great concern, and VMware Global Senior vice President Fan See, things become complex there are two other factors. "The Big Data topic today, in addition to the data generation, includes the spread of cloud applications and the changes in data users." "Fan leads a data division within VMware to engage in the development of related products, including support for the rapid deployment of Hadoop in virtualized environments. Fan explained that the popularity of cloud applications has changed the form of a single data in the data center, with more and more data being kept in the public cloud outside the corporate firewall, making data integration a difficult task, and data users from the initial focus on managers, senior managers and gradually becoming popular with general business staff, This requires simpler and more flexible ways of getting the results of the analysis.