Enterprise Big Data-machine data

Source of machine data

What is the machine data, on its literal meaning understanding, machine-generated data. What exactly is the machine data? For example: Log data, monitoring camera images transmitted over the data, mobile phone data sent over, the sensor passed, sweep code ..... Almost all over the life of the little bit drops.

Traditional solutions for machine data

For such data, the previous IT operations monitoring, video surveillance software, power environment system has been implemented, mainly to the health threshold of the device to do critical warning, and the video is the image of the transmission of images split screen display, independent environment to the room temperature and humidity monitoring, critical alarm. In the past, there was no progress in the analysis of machine data, and the analysis was to use data to speak, with big data produced new production patterns have been produced.

The workaround now

Now the machine data solution is to rely on big data technology, data Integration index, provide convenient search and query services, based on the analysis of integrated data, to provide customers with data analysis and display.

Architecture

For big Data machine data processing should be such a logic, data acquisition, we based on data from the data to do group management, application Model building, after the construction of data to extract the important fields, extract the field we can based on the existing experience of the data an early warning processing, after the field extraction of data index, Build a query service. After building a timed task to analyze the regularity of these data, summarize some of the occurrence of the law for production and maintenance to provide some constructive advice, I strongly recommend these event data storage learned, as a machine case, ten years of this data is a concept of the gold content, What is the value for users and producers. What kind of technical architecture should be adopted to deal with such a logic? Such as:

in this architecture, the lines of Spark, Hadoop, Zookeeper, Hive, es in the dashed box are built as a platform for operations and storage, both of which are the same, both of which are distributed, and the computation is primarily flow-based. Strom and streaming each have advantages, arbitrary trade-offs, ES and SOLR can store themselves can also be placed in HDFs, both methods also have advantages.

ES node added a long time, automatic migration of data shards, if the data stored in HDFs, storage is a whole block, no migration this said, when the query service volume is large, we only need to set up a query node to do load balancing, do not consider the data migration and so on unchanged, And the data exists in the Distributed file system, regardless of the storage and security requirements, is a good choice.

What's the future?

With the rise of big data technologies, these traditional things are evolving into a new paradigm, and with the help of visualization and storage of Big data analytics, these silent languages emit the sound they deserve, letting people know that they are machines, but they are also dead and gone, They are also sounding in their own way. We all know the story of "beer and diapers", these two seemingly unrelated two things, simply by people to find the law and widely used in the recommendation system, which shows the importance of a nameless connection. For machine data, that is more important, the data itself is not characteristic, but in the occurrence of other things on their own influence this relationship, is the most difficult to be noticed. In fact, all things have a relationship, no water people can not live, no electric machine to run, machine CPU long time peak on the application must have an impact, the CPU is big who in consumption, on the other have no impact on the production line, the middle of this link is broken, what causes, and then this thing happened, What are the associated things, and how do they relate to each other? 4.0 of industry is a revolution, but what kind of subversion will this revolution give to production?

My answer is no one said clearly, if a manufacturer is willing to spend money to engage in a big data development team to do their own set of customized after-sales service of Big data platform, or to ask software companies to do a big data such a platform, the completion is possible, production and consumption of data sharing, chain recommendations, a series of one-stop choice to buy services will become very competitive, that each manufacturer will no longer be a manufacturer, it is a system, he has his own procurement, his own production, his personnel scheduling, his own sales platform. The role of the factory is evolving, all the resources are integrated, it has its own fixed customer base, their customers not only need their own production of goods, as well as other manufacturers of products, so that evolution for advertisers, the best combination of powerful alliance. If the development of the affairs of the extreme, I think it should be these large online shopping platform miniaturization, the factory must be developed to take back the missing ecosystem of the piece.

Industry 4.0 says the data will be the first productive force that the factory can hand over to others? Previously heard that the Internet to later Internet +, and later I think should be the big Data era and big Data + such a production model.

Enterprise Big Data-machine data