Data manifesto: Big Data is a big change

Large data is not only a http://www.aliyun.com/zixun/aggregation/8302.html "> Data warehousing Technology or BI (business intelligence) upgrade version, but also a revolution." Big Data Manifesto: No Fear, no doubt, big data is a big change.

When the first world revolution, we were thinking about how to reduce the number of data and how to archive it, but how to collect more data and analyze it. At that time, we were not afraid of being threatened by the volume of data, but looking for additional data to help us develop our business and understand our government and society.

Later, low cost and price declines in disk and storage drive the trend towards decentralized processing of commodity server clusters. Although we have been collecting, processing and analyzing large data for a long time, until now, the methods we use are still inefficient and lack of economics. But this big change has data that we've always wanted to know but haven't been involved in before, so now we don't have to be afraid.

Definition of large data my basic definition of large data is the collection and analysis of very fine event-driven data in the High-tech field. This involves the derivation of data from the Internet, which is much larger than the analytics capabilities of the Web site. It is also related to sensor data, and for most of the sensor data we have until recently discarded. Now, those data with great value will help us to understand the business operation and interaction with customers. For me, this is the standard that can be called Big data.

However, event-driven dataset capacity is so large that the transaction database system does not function efficiently. Large data such as Hadoop, complex event processing (CEP), and large scale parallel processing (MPP) systems are developed specifically for this larger workload. In the future, trading systems will be improved, but they will always have an obstacle that they are not designed and applied.

2012: Large data Age?

Big data is becoming mainstream, transforming specialized technologies from science and technology companies into proprietary technologies for enterprise IT applications. Large data as the mainstream of it tools, has a significant impact on the IT sector, its availability and ease of installation standards are higher than the scientific and High-tech companies in the field of large data requirements. That's why we see companies like Microsoft that can get into the gaming industry by using a Web browser to leverage the cloud-based large data technology.

In order to better integrate large data with enterprises, we should realize the refinement of technology and reduce operating costs. At the moment, many large data tools are either crude or expensive, or are supported by highly specialized technicians who need to be able to perform operations. However, this situation is gradually changing, and it demonstrates the bright future of the big data revolution from the side.

Spreadmarts is not big data, but they also have their own role big data and we used to use the spreadsheet model and the number of operations are different? Spreadsheet technicians have been doing analysis, but certainly not large data, because Excel does not conform to the previous definition of large datasets. Until 2007, Excel can handle more than 16,384 rows of spreadsheets. However, it is still unable to handle the larger volume of business data, which is much inferior to large data.

Of course, the results of large data analysis can be further calculated and discussed with Excel. In fact, Microsoft has developed an attachment to Excel, a data warehouse interface connected to Hadoop, and a large symbolic data technology. Large data work is like post-production after an analysis based on Excel and rough editing.

On the other hand, BI (business Intelligence) and DW (Data Warehouse) are complementary, which is a good thing for large data. Large data allows backward, traditional techniques to provide insights into datasets that cover a wider range of operations and interactions than before. We can continue to use familiar tools in a completely new environment to achieve access to seemingly impossible or random things.

Natural language processing and large data solutions for natural language Processing (NLP) with Hadoop have been created. The solution involves the Python programming language and a set of called NTLK (Natural Language Toolkit). Another example is the application of Apple's Siri technology on the iphone. Users can only talk to Siri and get answers from a large team of experts in the field. Interestingly, large data technologies will help improve natural language technologies, such as processing a larger number of written works and understanding the algorithms. As a result, large data will become easier to use.

(Responsible editor: The good of the Legacy)