Application of Big Data platform in Internet industry

This paper is a summary of the application of big data in the Internet industry by Yuzhongyan, senior manager of the 58 City information Systems Department. The practical experience explains the transformation of big data from prototype to development, as well as the conversion of traditional data warehouse to big data platform and visualization application of big data in the process of development.

Big Data is a prototype--development

All companies started out with data that was relatively backward, but faced a big problem as the environment pushed the internet and the company planned to list on the New York Stock Exchange in 2013. Before the IPO, the company's data, whether it is traffic data or financial product data, need to undergo a very rigorous audit. To this end, we began to collate data, do data reports, spend more than a year to do data collation, data specification, slowly big data platform has a prototype.

After the company went public, the focus of data operations shifted from reflecting past and present conditions to directing business, managing performance, and supporting sales.

From 2015 to 2016, the company invested its goals in data intelligence, established a very large system platform, and fully implemented a data-based operation for the entire sales and product operations system. On this platform, we not only manage sales, but also build a sales competency model to predict future performance. Because the stock price in the U.S. market is more dependent on the future development of the company, the company's future expectations determine the company's current share price, so the total performance forecast is very important, can be very accurate forecast sales more important.

Big Data Platform Construction

The company has a lot of business database, there are orders, CRM, contract management, human resources and other databases. In the past, we will integrate the data in these databases, do a simple data warehouse, and then according to the business requirements, on the basis of data integration, through the sail soft report to establish a decision-making system, do data display, provide services. The benefits of this process are low development costs, low input costs, simple technical architecture and the ability to run very quickly within the company.

But such a way with the increase in business systems, gradually began to be inapplicable. One is the diversification of business data, the data is not centralized management, the effective use of data difficult, data storage capacity is limited, can not be traced farther historical data. Second, in many cases, the requirements of different product managers and engineers are different, and these different requirements are intertwined. As a result, different teams make the same indicators, and because of inconsistent data definition, it is difficult for management to make decisions when there are similar repetitive indicators. The third is that the data channels used for analysis and operation rely solely on the monitoring of their own data, which can lead to greater pressure on the IT department.

In addition to the above-mentioned issues, the company presented a new round of status based on the future development situation, and proposed the following requirements:

• Diversified business development, the company's large number of mergers and acquisitions led to the group data volume and data demand burst growth;
• Data mesh flow, basic data, data indicators lack of unified metadata management;
• In order to better explore the value of data, improve data capabilities and build a standardized data system becomes a top priority.

Big Data platform Business architecture

The business structure of the group can be broadly divided into several pieces of content. The first one is the data model master. The main work is to do meta-data management, data model development and the establishment of a unified data standards. The second one is the data developer, which will dock the product and then do some business development. The third is the business people, who are operations, sales, and marketing personnel who have made a lot of visualizations in this area to help them make decisions. Finally, the company's data analysis department, they have a lot of audit, including some of the company's core data analysis needs, they will be independent data query system to achieve multidimensional analysis of data.

Is the technical architecture, the data storage at the bottom, crawl the data, and the storage layer of data to the processing layer, the processing layer to complete business computing, the formation of indicators. Then the front-end through the application of the Sail soft report, the data in the form of business reports graphically display, and in Mobile report app display.

In addition to this, we have also managed to do the data Center Building code. Then there is data quality management, which includes ensuring the accuracy of the data and establishing a code of operation and management to ensure the timeliness of the data. The four blocks are combined to ensure smooth operation of the platform.

Big Data visualization

Data visualization is the last step of the platform and the most widely used step. In this regard, since we

1, the demand source is many: the team needs at the same time to connect the Finance Department, the sales center, the Customer Service center, the management level and so on all levels, and the various departments concerned indicators vary widely.

2, the way of visualization of the report more: the need through the schedule, summary table, Drill table and various types of charts to meet the needs of different management ranks, but also to the demand side to provide offline reports, real-time reports, mail & SMS push data and other channels.

3, Custom development more: In order to butt into the company's internal business system, the need for frequent custom development.

Therefore, a visual report development tool is needed to meet these needs. This piece, whether from the original traditional data Warehouse mode or now the big data platform, we have chosen the sail soft report.

The advantages are as follows:

• class EXCEL Design Style: The operating interface greatly reduces the learning cost.
• Multi-data source support: It is convenient to access various types of databases in various departments.
• Excellent icon display: HTML5 charting technology, supporting a variety of chart types, styles, styles, parameter transfer flexibility, rich interactive effect.
• Friendly interface: Developers use Web scripts, API interface, and so on in- depth development and control, support plug-in development, installation, use and management to meet their individual needs.

Visualize the presentation of:

Application of Big Data platform in Internet industry