Analysis of Business intelligence platform technology in the new era of large data (Bigdata)

Facebook has announced that its users are now over 750 million and that the number of shares per day has reached 4 billion. IDC predicts that from 2009 to 2020, the total amount of data will increase 44 times times to 35ZB (Zettabyte), and that 80% of the data is unstructured data. Bigdata is also a concept without a normative definition, and different applications have different interpretations of large data. Whether the big data has opened a new era, it may be premature to draw a conclusion, but the enormous impact it brings will not allow us to ignore it. Informatica Enterprise Data Integration Product Management Director Zheng, she shared: "The big data has two distinct characteristics, first, the data attribute is unstructured or semi-structured data, second, the data generated frequent interaction, large-scale data analysis, and real-time integration with the business data mining." "At the same time add that" is only the order of magnitude of the constant refresh does not represent the big data.

Technical deconstruction of large data

From these two characteristics, it can be seen that the large data is including the transaction and the interactive data set of all data, whether from the scale or complexity, far beyond the common database or business intelligence technology capture, management and processing capabilities. Some companies already feel the impact of runaway data growth on business, Zheng that big data is made up of three key technology trends:

1. Large transaction data: Traditional relational data and unstructured and semi-structured information continues to grow in online transaction processing (OLTP) and analysis systems from ERP applications to data warehousing applications. This situation becomes more complex as businesses move more data and business processes to public and private clouds.

2. Big Interactive Data: The Forces nouvelles are made up of social media data from Facebook, Twitter, LinkedIn and other sources. It includes call detail records (CDR), device and sensor information, GPS and geo-location mapping data, mass image files transmitted through the managed file transfer (Manage file transmits) protocol, Web text and click Stream data, scientific information, e-mail, and so on.

3. Large data processing: the emergence of large numbers has spawned an architecture designed for data-intensive processing, such as Apache Hadoop, which has open source and runs in the commodity hardware cluster. The challenge for businesses is to quickly and reliably access data from Hadoop in a cost-effective way.

In the United States recently held in the Hadoop forum, 5,500 participants, admission tickets are said to be sold out 8 hours after the opening. Hadoop Distributed File system, MapReduce algorithm, large-scale parallel processing (MPP) database technology was first developed by Google, Facebook and other internet companies. As an Open-source technology, Hadoop now attracts many enterprise-level users to start experimenting. Compared to the previously expensive large-scale parallel processing and massive data analysis technology, Hadoop is able to use a more cost-effective and cost-effective way to deploy large data applications. "By combining traditional transaction data with new interactive data to gain insight and business value," Zheng, for example, says: "Companies can use social media to understand customer preferences and improve customer data to improve target marketing efficiency." ”

Application Platform for large data

From a business point of view, enterprises still need data integration, business intelligence in large data environments, but their scale is at a massive level, while the infrastructure for data integration needs to be more malleable. In June this year Informatica launched the Informatica9.1 for the big data, Zheng that this is the world's first dedicated for large numbers of unified data integration platform. "The development goals of this platform are very clear, it is based on Informatica data integration technology to help enterprise users to give full play to the business potential of large data," Zheng said: "On the data integration platform on the basis of the IT department to maintain control, through self-service to enhance the ability of all users to obtain relevant information, And can adapt the data service, provides the related data and the reliable data which adjusts according to the business demand, obtains the business insight and the consistency. ”

In response to the characteristics of large data, Informatica 9.1 offers innovative solutions in three aspects of data integration. The first is a new Data Warehouse device package that is capable of connecting to large transaction data and storing data through OLTP online analysis to provide access to high volume transaction data. Second, the use of connectors with new social media to connect to large interactive data. Access to data sources such as Facebook, Twitter, LinkedIn, and other media. The scope of data collection is extended to the emerging value dataset of the enterprise industry, including equipment and sensors, CDR, mass image files.

Thirdly, the connection function of the platform is used to support the processing of massive data. Allow IT to input data from different sources into Hadoop, and to explore and mine calculations for data applications, data quality in Hadoop. Better management of the interactive data within and outside the Hadoop system to provide insight into the enterprise.

"One of our clients, a leading professional fashion retailer, provides services to our customers through local department stores, networks and their mail-order catalogues," Zheng a success story for Informatica: "The company wants to provide differentiated services to customers, how to locate the differentiation of the company, By gathering social information from Twitter and Facebook, they learned more about the marketing model of cosmetics, and then realized that they had to keep two types of valuable customers: high consumers and highly influential people. Hope that by accepting the free make-up service, let the user carry on the Word-of-mouth propaganda, this is the transaction data and the interactive Data Perfect union, provides the solution for the business challenge. "Informatica's technology helps the retailer enrich customer master data with data on social platforms, making his business services more targeted," he said.