Lack of talent constraints large data landing

The rise of E-commerce, social media and mobile internet and the popularity of mobile smart devices have triggered explosive growth in large numbers of data. The real value of large data, so many traditional enterprises also want in the thinking, work mode, technology, etc., get large data wave baptism. However, the lack of relevant talent is still the traditional enterprise to tame large data constraints.

2014 Mid-Autumn Festival to fall, whether it is a gift to friends or relatives or their own consumption, choose what brand of moon cakes have become a lot of people focus on the issue. At the end of August, the China Statistical Information Service Center (CSISC) large data Research laboratory released the "2014 Chinese mooncake brand word-of-mouth research report." The report from the brand awareness, consumer interaction, quality approval, corporate reputation, product acclaim, brand health and so on 6 dimensions of the moon Cakes brand reputation of the pros and cons of this year.

I have to say that Csisc's report, which is based on a large data structure, provides an important reference for people to buy mooncakes, and provides a good reference for their brand reputation management from the perspective of the Mooncake enterprise. It is clear that in previous years the case of "diapers and beers" has come to the report of the brand reputation of the moon cakes, and large data technology is moving from a foreign concept to a more practical local application.

The driving force behind big data technology

A few years ago many people were worried that big data would be another it concept hype. Now, the reporter found that the large data technology HDFs and Mapreduce, represented by Hadoop, as well as its open source components hbase and hive, some of the big data open source technology is gradually being widely learned and applied by developers. The Hadoop software ecosystem Forecast, published by IDC, shows that the market is expanding at a rapid rate of 60% per cent a year. IDC expects the market to grow at a rapid rate of 813 million trillion by 2016.

In fact, large data exist in the process of people consuming, communicating and using mobile Internet. People are constantly making data, then consuming data, getting value from data, and then driving the rapid development of large data technology demand. It can be said that large data technology is data driven, at the same time, the results of large data analysis in turn to continue to produce data.

The large data technology, represented by Hadoop, has gained wide attention mainly because of its advanced technology. Such technologies are a good solution to the scalability, high performance, and high-availability challenges of large-scale systems, a problem that large companies, especially large internet companies, need to address.

From a technical point of view, the main driving force that drives these big data technologies from birth to maturity is the real demand of enterprises. From a business point of view, the speed of knowledge transmission in the Internet age to let more people know these new technologies, open source community development also allows more people to participate in the development of new technologies, while the power of capital is also driving these new technologies fast to maturity and commercialization.

Feng Dazhi, senior consulting manager at Cloud Base Big Data company, put forward two points of view. On the one hand, Hadoop's outstanding distributed storage and computing capabilities have increased the scale and efficiency of data mining for traditional enterprises. Feng Dazhi For example, a well-known domestic insurance company, for nearly 100 million customers to achieve a full volume of customer clustering, customer churn model, the analysis of the relevance of insurance products are based on the analysis of the total data. In addition, compared with the traditional storage, small machine, relational database combination, without considering the maintenance cost, the large data technology represented by Hadoop does have some advantages of performance and price.

According to Feng Dazhi, a domestic mobile company's system is a set of dozens of small organizations into a large data warehouse system with nearly hundreds of nodes, regardless of the cost of the system itself or the operation of the cost is very high. Feng Dazhi that such a large system is a double challenge to the traditional technology system and price system.

The value of large data is more than technology

The significance of large data to the analysis of public opinion, first of all, is from the change of thinking and work mode, and secondly is the technical improvement.

Whether the Government or enterprises should learn from the Internet company on the management of public opinion, the views of netizens analysis, internet thinking to manage public opinion. In the mode of work, using large data analysis technology, government and enterprise can obtain more data and visualize, also can change the management mode of the existing public opinion.

Based on his hands-on experience at work, Bai, chief engineer of the Shanghai Stock Exchange, has pioneered the data-processing solution of "nobility".

The traditional enterprise's IT system often has the "aristocratic" characteristic: the purchase cost is expensive, the maintenance cost is expensive, the platform migration cost is more expensive. In the past, the traditional enterprises under the great pressure of safe operation, only in this "aristocratic" and the kind of "aristocratic" between the choice, with "aristocratic" solution to highlight the value of the program.

The real value of large data technology can not only continuously impact the limit of data processing, but also generally reduce the cost of data processing in the non limit situation.

In contrast, some traditional enterprise IT staff have been used to IoE (IBM, Oracle and EMC) products, and suddenly let them in the open source technology based on the development and operation, often feel unfamiliar, not accustomed to. Moreover, the traditional enterprise data processing system has been in operation for more than more than 10 years, the technical category of IT personnel are still built on the original IT system as the core, the most important is the life cycle of various infrastructure is very long, new technology must be balanced. From this perspective, large data is conducive to the elimination of traditional enterprises in the "aristocratic disease", more conducive to the integration of the organization's business, data and other resources, to mobilize the enthusiasm of the relevant personnel can be to maximize the value of efforts.

Man is the first driver of big data.

If the big data represented by Hadoop is a small elephant, then the enterprise must have a trainer who can tame it. When many companies embrace such large data technology, the talent that is proficient in large data technology becomes a big gap. Wugansha, chief engineer of the Intel China Institute, said in a speech that people are the first driver of big data.

According to Forrester's latest report, most companies have analyzed only 12% of existing data, and the remaining 88% have not been fully exploited. The reason is that the lack of large data analysis ability is the main cause of this situation. Here, traditional companies have a heavier burden of data analysis than some innovative and internet companies, and a lack of large data-related technical personnel.

Yianyang, general manager of China National Securities Information Technology department, said at a salon that the company used the lightweight general-purpose hardware platform in 2008 to build a "go aristocracy" data Warehouse in conjunction with the Open source System (Greenplum), becoming a model for the industry. However, at the same time, Yianyang also issued a "tired" sigh.

Sun Yuanhao said that the current market for skilled use of spark talent is relatively scarce, so the company has to develop its own Scala programmers and spark developers. But Liu also said that SAS needed complex talent: on the one hand, the technology in the field of Hadoop, on the other hand, the company enhanced the expertise in the field of analysis and statistics, so SAS can only insist on training their own talent in the project.

Unlike traditional enterprises, many start-up companies or internet companies, they do not have much historical data, the core team is more technical experts, so the use of large data technology has advantages.

Teradata Company Greater China Large Data division director Kong Yuhua in a number of traditional enterprises to communicate, found that many companies have been using Hadoop to do research and application, but also limited to storage, preprocessing and some basic web analytics.

And, now that Hadoop technology is growing fast, users often encounter new technologies, new problems, and need to go to the Hadoop open source community to solve specific problems, which is a bit slower for traditional business needs.

So now many businesses are beginning to realize that to really do data analysis, data mining applications in the Hadoop platform, there are two options, either a knowledge of the data, understand the analysis, understand programming and skills of the technical team to operate, or choose a commercial company launched a mature large data platform.