The age of Big data development: Seven challenges and eight trends

Large data challenges and opportunities coexist, large data in the next few years of development will be the previous years of the expected expansion stage, speculation stage into the rational development stage, landing application stage, large data in the next few years will gradually enter a rational development period. The future of big data development still has many challenges, but the outlook is still very optimistic. Challenges to large data development there are still many challenges to the development of large data, including the seven major challenges: the business unit does not have clear large data requirements leading to the gradual loss of data assets, enterprise internal data isolated island serious, resulting in data value can not be fully exploited; low availability of data, poor quality of data, resulting in data unavailable Data-related management technology and architecture lag behind, leading to the lack of large data processing capacity, the security capabilities and awareness of the poor, resulting in data leakage, large data talent is not enough to cause large data work difficult to carry out; large data more open and valuable, but lack of large data-related policies and regulations, resulting in data openness and privacy difficult to balance, Also difficult to better open. Challenge one: The business unit does not have clear large data requirements many business units do not understand large data, and do not understand the application of large data scenarios and values, it is difficult to put forward the precise needs of large data. Because the business unit demand is not clear, large data departments and non-profit departments, the enterprise decision makers worry about the cost of investment more, resulting in many enterprises in the construction of large data departments hesitant, or many companies are in the wait-and-see attitude, fundamentally affected the enterprise in the direction of large data development, Also hinders the enterprise to accumulate and excavate own data assets, even because the data does not have the application scene, deletes many valuable historical data, causes the enterprise data assets to lose. As a result, large data practitioners and experts are needed to drive and share large data application scenarios, so that more business people understand the value of large data. Challenge two: Enterprise internal data island serious enterprise start big data The most important challenge is the fragmentation of data. In many enterprises, especially large enterprises, the data are often scattered in different departments, and the data in different data warehouses, different departments of data technology may not be the same, which led to the enterprise's own data can not get through. Without this data, the value of large data is very difficult to dig. Large data needs to be linked and integrated with different data in order to better understand the customer and understand the business advantages. How to get the data of different departments through, and realize the sharing of technology and tools, can better exert the value of large data of enterprise. Challenge three: Low data availability, poor quality of data many medium-sized and large enterprises, every moment is also generating a large number of data, but many enterprises in the preprocessing phase of large data is very important, leading to data processing is not standardized. In the large data preprocessing stage, the data should be extracted and transformed into a convenient data type, and the data should be cleaned and de-noising to extract the effective data. Even many enterprises in the data reported there are a lot of unreasonable unreasonable situation. All these reasons lead to poor availability of data, poor data quality and inaccurate data. The significance of large data is not onlyIt is possible for data analysis and data mining personnel to extract valuable information from large, high availability data by collecting large amounts of data and preprocessing the collected data. Sybase's data show that data applications with high quality data can significantly increase business performance, increase data availability by 10%, and increase corporate performance by at least 10%. Challenge four: The challenges of data-related management technology and architecture technology architecture include the following: (1) Traditional database deployments cannot handle terabytes of data, and fast-growing volumes of data exceed the management capabilities of traditional databases. How to build a distributed data warehouse, and to facilitate the expansion of a large number of servers to become a lot of traditional enterprise challenges; (2) Many enterprises adopt traditional database technology, at the beginning of design did not consider the diversity of data categories, especially for structured data, semi-structured and unstructured data compatible (3) The traditional enterprise database, the data processing time is not high, the statistical results of these statistics often lag one day or two days to be counted out. But large data needs to be processed in real time, in minutes and even seconds. The traditional database architect lacks the ability of real-time data processing, (4) The massive data needs the very good network structure, needs the formidable data center to support, the data center's Operation dimension work also will become the challenge. How to ensure data stability, support high concurrency, reduce the server's low load, become a massive data center operation of a key work. Challenge Five: Data security networked life makes it easier for criminals to get information about people, and there are more criminal methods that are difficult to track and prevent, and there may be more sophisticated scams. How to guarantee the user's information security becomes a very important topic in the large data age. Online data is more and more, the motive of hacker crime is stronger than ever, some well-known website password leak, system flaw causes the personal sensitive information that the user information is stolen and so on to alert us, must strengthen the big Data network security construction. In addition, with the increasing of large data, the physical security requirements of data storage will be higher, so that the multiple copies of data and disaster-tolerant mechanism also have higher requirements. The data security of many traditional enterprises is worrying. Challenge VI: Large data personnel lack of large data construction of each link need to rely on professional completion, therefore, we must train and create a master data technology, understand management, have large data application experience of large data construction professional team. At present, the shortage of large data-related talents will hinder the development of large data market. Gartner predicts that by 2015, there will be 4.4 million new jobs around the world with big data, and 25% of organizations will have a chief data Officer position. Large data related positions need to be complex talent, able to mathematics, statistics, data analysis, machine learning and natural language processing and other aspects of comprehensive control. In the future, the big data will be about 1 million of the talent gap, in all sectors of large data in the high-end talent will become the hottest talent,Covers a large number of data development engineers, large data analysts, data architects, large data background development engineers, algorithmic engineers and other directions. Therefore, universities and enterprises need to work together to cultivate and excavate. The biggest problem at the moment is that many colleges and universities lack big data, so companies with big data should combine with schools to train talent.