How big data technology can create greater value

The term “big data” has been on fire for several years. In the last one or two years, the limelight seems to have been taken away by the concepts of artificial intelligence and deep learning, and has gradually become a “out of gas” technical vocabulary. But in fact, we believe that the "death" process after this hype shows that big data as a cutting-edge technology has begun to be truly applied in various fields.

2016 is a milestone year for big data, not only in many industries, but also to make more meaningful decisions, and to be more user-friendly and easier to operate in terms of availability, backup and recovery. We will also continue to focus on quality projects that make production more efficient, resource allocation more rational, and transactional efficiency faster, thereby increasing the profitability of producers.

In 2017, how will big data technology create greater value for businesses and users?

After several years of development of big data, great progress has been made in infrastructure construction. Some companies have become listed companies (such as HortonWorks and NewRelic), and some companies such as Cloudera and MongoDB have already exceeded one. One hundred million U.S. dollars. In addition to the ability to store and process big data at the infrastructure level, the application of big data to various industries is just beginning.

In the past year, we have seen the deep application of big data in industries such as financial technology, medical, agriculture, and enterprise services. Big data analysis makes devices connect faster, make decisions smarter, and operate more efficiently. So in 2017, what new vitality can big data show?

The combination of the first big data and deep learning will be closer

In 2016, the development of deep learning and the integration of tools for big data platforms and frameworks led to significant advances in big data analytics. In fact, the rise of deep learning in the past two years is largely a credit for big data. The algorithm behind deep learning was born more than a decade ago, but until recently big data could be obtained cheaply enough and processed quickly enough. Played its potential.

At the same time, the application of deep learning in different fields and different scenarios, especially the open source of deep learning framework, will lead to the need for more models and applications to generate larger data. This mutual promotion will bind big data and deep learning more closely, making big data analysis play a more important role in the industry.

The second largest data will encourage more companies to use cloud hosting services

In a recent survey of big data media O’Reilly, most companies will continue to use other big data services once they have gained experience with big data services in the cloud. This shows that enterprises are increasingly accepting cloud-based big data services.

While public cloud services are popular, issues such as enterprise legacy systems, sensitive data, security, compliance, and privacy still make companies more willing to choose a private cloud or a hybrid cloud model. Now a more flexible model is becoming more and more accepted, that is, building a company's proprietary cloud in the public cloud, which is a proprietary hosting service, such as Predix for industrial IoT or CIA cloud based on Amazon AWS.

There are now a variety of proprietary cloud-hosted big data services, including storage, data processing, visualization, analytics, and artificial intelligence. One advantage of this is that data professionals within the enterprise will not need to learn how to maintain cloud data, and cloud hosting service providers will manage it. The other is that data can be deposited and stored in the cloud, making it easier to calculate, process and move.

The third Hadoop's position in big data will be weakened and will eventually be replaced.

In the past few years, we have seen some technologies emerge with the wave of big data, meeting the needs of Hadoop analysis, such as the emergence of Spark. However, organizations with complex, heterogeneous environments no longer want to build separate BI access points for a single data source from Hadoop. In 2017, we will see more companies analyzing data from all sources, and platforms that don't depend on a data source will thrive.

2016 is already the tenth year of Hadoop, and it's not just a storage and computing framework, but a huge ecosystem. But with the emergence of new frameworks such as Spark, Yarn, and Platfora (which have been acquired by Workday) and adoption by more and more companies, Hadoop's role is getting weaker and weaker and will eventually be replaced.

The convergence of the fourth Internet of Things, cloud computing and big data will create new opportunities for self-service analysis

In 2017, more and more sensors will be put into use, IoT will generate a large amount of structured and unstructured data, and more and more data will be deployed in the cloud. Data is usually heterogeneous and exists in multiple relational and non-relational systems. While innovation in storage and management services accelerates the data capture process, accessing and understanding the data itself remains a major challenge. As a result, there is an increasing demand for self-service analytics tools that seamlessly connect to various cloud-hosted data sources.

The self-service analysis platform allows users to build analysis models for analysis and visualization based on existing tables in the platform data warehouse. It can also connect its own data to the platform and build an analysis model on the accessed data. Analysis and visualization. A number of innovations have been seen in this area, such as Alteryx, Trifacta and Paxata, which have lowered the threshold for big data users.

The diversification of the fifth data format and source will become the focus of investors

The 4V characteristics of big data mentioned above are growing at a rapid rate, but diversity will be the single biggest driver of big data investment. According to a recent survey by NewVantagePartners, as companies seek to integrate more sources of data and focus on the “long tail” of big data, how to process and analyze diverse data becomes a core competency. From modeless Json to nested types in other databases, to non-planar data (Avro, Parquet, XML), data formats are growing exponentially. In 2017, analytics platforms that connect diverse data will become investors' attention. direction.

Although we are still in the early stages of development of big data technology, this technology will be more and more widely used in the industry. As big data continues to mature, the vocabulary itself will “dead”, and when the technology is as omnipresent as the air, it is when it really “dies out”.