Big data is useful, but it doesn't work.

Large data is now no longer the "patent" of the It circle, from last year's gala, to just past the two sessions, can see its figure, but in fact, the Spring Festival Gala and the two sessions of the data can only be called small data, it and the real big data is still far from the difference. Even so, the value generated by the data has been recognized.

As far as large numbers are concerned, its development can be divided into three stages, the first stage is the organization's internal data, these data are usually structured data, we generally classify these data, sorting and other operations, the same type of data for comparison, analysis, mining, in summary, is basically statistical work. In the second phase, the scope of the data was extended to the industry, with a wide range of application data, large increases in data volume, especially the emergence of unstructured data. The typical data such as video and picture, at this stage is characterized by the coexistence of unstructured and structured data, and the large amount of data, the analysis of these data is the state of our current stage.

The third stage is the future of large data development idealized state, first of all it must be cross-industry, and the scope of the data is the whole society. Through the analysis of these data to use, will directly change our way of life, which is now the vision of many enterprises in the future of transport, medical, education and other areas of development direction.

Big data is too big to use

The third phase is what we aspire to, but in the second phase we are faced with more problems. One of the problems is "big". Large data to give people the most intuitive feeling is big, it brings the problem is not only storage, more is a large number of data can not be used to traffic, as an example, from 2001 began in Beijing's main road on the addition of some card equipment, to today basically can see the streets.

The amount of data produced by these devices on a daily basis is staggering, with only 20 million images per day, and the storage of these data is the most basic task, and we need to use that data. For example, the inspection of the set of vehicles, the monitoring of suspected vehicles, when you want to use this data, the traditional database and system architecture, put such a large amount of data, is not moving. The problem has led many companies to balk at big data.

Big data is too hard to use

When it comes to the use of big data, Hadoop,hadoop itself provides two of the most important things in a distributed system: Distributed Storage (HDFS) and distributed Computing (Mapreduce). Both solve the computational and storage problems of dealing with large data, but more importantly, it opens the way for large data applications to be developed.

Hadoop is one of the most popular ways to solve big data problems at the moment, but it is still immature, Jonathangray, a Yahoo cloud computing and Facebook software engineer, says: "Hadoop is difficult and complex to implement, and if it does not address technical complexity, Hadoop will be the end of itself. "It is for this reason that Gray founded his company--continuuity, the company's goal is to create a layer of abstraction based on Hadoop and hbase, shielding the complexity of the underlying technology of Hadoop." This shows that want to use good big data is a big test.

Large data too expensive to use

The feature of Hadoop is that you can use cheap x86 devices to do large data business, but in fact if you really want to use it to accomplish certain business tasks you have to be a "tyrants". In a successful case of foreign use of big data, Amazon has given such a set of figures that NASA needs to pay more than 1 million dollars for 45 days of data storage services. Digital advertising companies like Quantcast are also spending billions of dollars on Hadoop technology to tailor systems to their needs. From the above two cases, the large data for commercial use is still very expensive at this stage, as the large data software environment matures and development tools increase, prices will gradually decrease in the future.

From the list of these three difficulties, in fact, is not to throw cold water on the big data, but to say big data want to gold is not simple, first of all, before the big data, a good inventory of their own resources, not only data resources, but also including knowledge and skills. After you have identified your abilities, choose a project that will maximize the value of your existing resources. If you need help, you should consider the business consultant first and then consider the technical personnel. In order to solve a business confusion spent money, called investment, and put money into a special skills of it talent, it is called sunk costs. When you have these, choose a more flexible and scalable tool to lay the groundwork for future expansion. More importantly, start small.

Perhaps in the near future, large data analysis will become a necessary skill for enterprises. But that day is far from coming. When those suppliers spend millions of dollars on the layout of large data analysis, you can rest assured that you have not missed anything. And your money is spent more in need, more efficiency and more value.