Large data from "small" start--small and medium-sized enterprises to solve the problem of big data

Any era or mode of the rise, can not be separated from the relevant killer app, such as the C/S ERA of SAP ERP, Internet 1.0 era portal, as well as the Internet 2.0 era of search and SNS, then in today's cloud computing this era have those killer app? Of course, the first thought must be virtualization and related IAAS services, represented by VMware and Amazon EC2, and in addition, the big new data is definitely one of the killer apps for cloud computing. Not only similar to Baidu, Ali, and Tencent, such as the Internet giants have relevant application needs, and according to the author's usual contact with customers, found that there are many ordinary small and medium-sized enterprises, especially the medium-sized Internet and things networking companies, in large data on the scene there are many. This article will first introduce you to the author in the eyes of the big data, as well as the significance and characteristics of large data, and then introduce the common processing flow of large data, and then we will share with you how to help some small and medium-sized enterprises to implement large data-related solutions, that is, large data from the "small" start.

What is big data?

In the past, the computer generated data is simpler, basically is a pen business, although the total size, but the overall growth rate is manageable. Traditional financial firms, for example, often use several mainframes to manage all their business data. In recent years, because of the increasing number of smart devices represented by tablets, smartphones and sensors, the data generated by these devices are far more than we could imagine. According to IDC, global digital information is expected to grow at an alarming rate in the next few years, estimated to be 44 times times the total by 2020. According to another data, 90% of the world's data have been generated over the past two years and are growing at a rate of 50% a year. Every day, sensors, mobile devices, online transactions and social networks in every corner of the world produce petabytes of data, and every month, global netizens post 10多亿条 Twitter messages and 300多亿条 Facebook messages. So what are the values and implications of these big data?

The significance of large data

The author and some friends have always felt that big data is like an oil well, because it contains a very rich value, if the enterprise can effectively use its internal storage of massive data, then will improve its own products and services, so as to enhance customer and audience experience, in the large data age to gain competitive advantage. And with the continuous improvement of analysis and mining technology, we can provide a new decision model on the basis of the former, so as to support managers to make quick and accurate decision, so that they can surpass their rivals and seize the market opportunities.

The following is a few industries to illustrate the significance and role of large data?

Internet companies

Some customers are mainly doing online public opinion or online advertising business, they will process and collect TB level of log or web information, structured and unstructured data. They are analyzing the data to provide value to their customers, such as analyzing whether a male skin care advertisement is placed during the World Cup or during the Asian Cup? Ebay is a good example of E-commerce. Its analysis platform handles up to 100PB of data a day, surpassing the daily data processing of the Nasdaq exchange. To accurately analyze the user's shopping behavior, ebay defines more than 500 types of data, tracks customer behavior, and facilitates ebay's own business innovation and profit growth through these analyses.

Smart Grid

We have a partner who is doing smart grid-related solutions. For those grids, failure to accurately estimate the actual use of electricity would require the power grid to emit excess power, although the excess power could be preserved in some mode, but a large amount of power waste was unavoidable. and through their smart grid solutions, every quarter of an hour will collect a province tens of millions of users of electricity data, then they will be based on this data to accurately analyze the user's power model, and finally through this model to optimize power production, so as to effectively reduce the waste of power resources.

Car Networking

In connection with vehicle networking, one client has hundreds of thousands of Android-based terminals in a city, and these terminals send GPS messages to the back-end data clusters every once in a while, and these clusters analyze the vast amount of GPS information and analyze which sections are blocked at any time. These very valuable information is then continuously pushed to the customer to help reduce the time spent on the road.

Medical industry

In the medical industry, there are a lot of big data use cases. First of all, through the analysis of a large number of case information, will be effective to help doctors heal; Secondly, if the probe device is added to multiple nodes of a patient's body, and each probe collects gigabytes of data on the state of human cells and blood every day, then the compute cluster can analyze it based on the data This can be more accurate to determine the cause, so that doctors to patients with more targeted treatment.

Machine learning

In this regard, the most famous example is the recently popular Siri, which has a large hbase cluster to analyze and manage text data such as languages, making Siri an increasingly sophisticated personal assistant, providing a date reminder for iphone 4S users, Services such as weather forecasts and hotel recommendations. In addition, IBM's Watson, through a Uima framework based on the Hadoop cluster to excavate a large amount of text information to achieve a certain degree of artificial intelligence, and in the United States famous quiz show Jeopardy beat a number of outstanding human players.

National security

The most famous example of this is the Federal Intelligence Agency (CIA) in the United States. Over the past 10 years, they have collected a large number of geographic-related video data from the Afghan side through unmanned reconnaissance aircraft, and then analyzed the vast amount of video data to locate the most dangerous groups of terrorist groups.

(Responsible editor: The good of the Legacy)