Implementation of the optimized application structure of run-dry set calculation report t+0 real-Time report

In the reporting project, customers are increasingly concerned about the real-time nature of the source data, and want to see the latest occurrences of the data reflected in the report. However, the traditional reporting tools + Data Warehouse +etl Way difficult to do this, often can only see yesterday, last week or even last month's situation, that is t+1,t+7,t+30 collectively T+n report. It is difficult to implement t+0 reports, which can reflect real-time information.

Analysis the reason is: 1 If the report's historical data and the latest data are read from the customer's production system, you can implement T+0 etl 3 Although theoretically it is possible to form real-time reports from both the historical library and the production library, the general reporting tools do not have the ability to calculate the cross-Library count. Other cross-Library computing schemes are more complex and difficult to implement.

you may consider using the run-dry set of reports to provide T+0 Report Scheme, using the hybrid data source capability of the aggregate report to achieve low-cost real-time reporting. The realization of the idea is: a large number of historical data will not be changed in the data file storage, a small amount of new data from the production library to read, while guaranteeing the real-time report, while reducing the cost of historical data storage, reduce the reporting system caused by the production database load. The traditional t+n scheme and the t+0 scheme of the aggregate report are compared as follows:

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/5A/1D/wKiom1T2qo2hyaXeAAETfe8y8QE915.jpg "style=" float: none; "title=" report5_structure_t+0_1.jpg "alt=" Wkiom1t2qo2hyaxeaaetfe8y8qe915.jpg "/>

Run dry calculation in the report structure, "Export (not real-time)" refers to a data file that synchronizes new data from the production database to historical data during non-working hours (for example, at night). The concrete implementation is to use the run-dry collector to provide command-line execution, with the operating system Timing task mode, see the Collector tutorial.

here is the "State sales statistics" to explain the run-dry set calculation report T+0 The specific approach to the programme, the report reads as follows:

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/5A/19/wKioL1T2q6LgV1SZAAGh1BgOuUc639.jpg "style=" float: none; "title=" report5_structure_t+0_2.jpg "alt=" Wkiol1t2q6lgv1szaagh1bgouuc639.jpg "/>

The amount of historical sales data in a report is large and comes from a data file. At the same time, in order to guarantee the real-time report, the small amount of data is removed directly from the production database (DB2).

The specific implementation steps are as follows:

The first step is to write the set-up script sales-state.dfx in the collector .

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/5A/1D/wKiom1T2qo6w5UPKAAGS3rQJ1cE614.jpg "style=" float: none; "title=" report5_structure_t+0_3.jpg "alt=" Wkiom1t2qo6w5upkaags3rqj1ce614.jpg "/>

A1: Connect a pre-configured production database (DB2).

A2: Create a database cursor that reads the sales data and the sales person's data with simple SQL. From days in the Where condition (current date) =days (OrderDate) , you can see that the sales data only reads new data for the day.

A3: A cursor thatestablishes a pre-exported data file d:/files/sales/sales.b . File cursors can read data in batches from larger data files to avoid a memory overflow situation. the @b option refers to reading a file according to the binary encoding provided by the collector.

A4: Vertically merges database cursors (new data) and file cursors (historical data).

A5: The groups function is used to complete the group summary of the merged cursors.

A6: In descending order of total sales.

A7: Close the DB2 database connection.

The second step is to establish the DB2 data source and the set data set in the collection report:

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/5A/19/wKioL1T2q6LA3gZ0AADZAgi-PWA275.jpg "style=" float: none; "title=" report5_structure_t+0_4.jpg "alt=" Wkiol1t2q6la3gz0aadzagi-pwa275.jpg "/>

The third step: Design the report as follows:

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M02/5A/1D/wKiom1T2qo6ylc4lAAE-_KOurNY842.jpg "style=" float: none; "title=" report5_structure_t+0_5.jpg "alt=" Wkiom1t2qo6ylc4laae-_kourny842.jpg "/>

The chart specific production method in the report is described in the collection Report tutorial.

It is necessary to note that the collection report also enables data to be stored in other ways, such as MongoDB, HDFs,or a traditional data warehouse. The new data in the production library can be exported with the collector, or it can be exported with other ETL tools.

This article is from the High performance report data calculation blog, so be sure to keep this source http://report5.blog.51cto.com/8028595/1617225

Implementation of the optimized application structure of run-dry set calculation report t+0 real-Time report