Implementation of a lightweight data management and analysis platform

Introduction to Application Background

In many typical industry applications, data that is generated and maintained according to geography, time, or category dimensions is needed. The managed data classification is maintained in a constant manner, while the content (field) of the data itself needs to evolve according to the business needs. On the one hand, the data that is managed must support to carry on the enhancement check operation according to the authority; On the other hand, we need to be able to do the WYSIWYG chart analysis, such as the comparison of data between different regions, the analysis of the rule of time dimension data, the classification of different categories of data, and even the maintenance of statistical relationship between related data.

For example, in the power industry, grid companies are divided according to their geographical location, and their core business data, represented by electricity and load, is measured in time. These managed data have certain generality: the data belongs to the region, the data time divides, the data's statistic caliber and so on data's dimension basically maintains unchanged, but the data category but along with the economic situation development change, the different area industrial structure, even is the local policy and so on often changes. For example, the monthly electricity consumption in the region, although the electricity industry has the use of classification standards for the unification of Electricity classification and category name, but the imbalance of economic development and industrial structure, so that the actual need to pay attention to different areas of electricity classification is similar, which requires the system to provide data management analysis function can There is a strong adaptability to the change of data table columns in the background relational database; On the other hand, regional differences have also led to differences in data sets of different grid companies, which require that the data management and analysis function provided by the system can adapt to the personalized requirements of the database table.

This article needs to realize the objective function called the Data management analysis platform, it needs to have the following characteristics:

This is a function used in the product, versatility and maintainability is the first place.

This is a B/s application, the user uses the browser to access the system.

Users in the interface can be used to increase the data to check.

The computational relationships between different data remain transparent to the user.

The user can analyze the data of the query to the graph.

Cross-database platform.

Analysis and Scheme selection

Because it is expected that this data management analysis platform can be reused in different products, this scenario should be abstracted from the common features of data storage in relational databases, rather than the business logic for a particular product. In other words, the tables and fields of a relational database should be abstracted, rather than based on the business data objects of a particular product. This allows the final implementation to add and reduce managed data categories (extensions of managed tables) with a simple configuration, as well as increase and decrease the number of managed data fields (extensions of managed columns) for specific data categories.

Why use JDBC

If the managed data, according to the database table map to POJO, and then for POJO implementation of subsequent display and persistence operation, even if the DIY out of a Hibernate as complete as the persistence tool to increase the number of managed data categories, still need root According to this data class corresponding relational database table, maps out a POJO class. This process produces new Java code, meaning that the entire software engineering process, such as research and development, testing, publishing, and implementation, has been initiated, and it is clearly inappropriate to call this process configuration.

Of course, b/s architecture applications commonly used in the form of submission to implement CRUD solution is even more inaccessible, in the popular SSH (Struts, Spring, Hibernate) mode For example, add new maintained data, meaning that at least one additional JSP page to generate user interface, A actionformbean is used to receive data from a form, and a Hibernate table mapping POJO object Not only means starting a complete software engineering project, but also increasing the workload linearly with the number of new data categories.

Here, you may have thought of the database client, yes, the database client is a typical WYSIWYG to meet the users of the data to increase the need for the search and deletion of the implementation. If we transform a database client, let it show the results of the query to the user's business perspective, and let it run in the browser, provide the ability to draw analysis of the displayed data, and can maintain the relational data calculation relationship, that is complete.

Unfortunately, we do not like the database client, so that the end user directly facing the physical table of storage business data, in a database table of the direct query results of the increase in the search operation. Since the database design paradigm is met, there are many fields in the datasheet that we call IDs. For example, in a table that stores some kind of data by region, in a field that represents a region, it must be a foreign key association that stores an ID representing the region and another data table that stores the area information, and when querying, you need to do a long table connection to get the locale ID corresponding to the area description that the end user can read.

Therefore, our data management analysis platform should at least be able to provide users with a set of multiple table connection query results that can be modified. If you think of a view, that's fine, because many relational databases support the view's update, and since JDK1.4, the rowset extension of JDBC is officially part of the JDK standard API, allowing data updates to query result sets through JDBC. If our data management analysis platform is bound to a specific database platform, then carefully study the corresponding database platform to view the update support situation, the careful specification of a set of feasible views design scheme may theoretically be possible. However, our data management analysis platform is expected to be able to cross the database platform, because the different databases in the view of each feature, the adoption of this scheme will greatly reduce the database independence of the solution.

That being the case, you might as well hold on to JDBC's thighs and think again. In the Java world, we use JDBC to complete the additions and deletions of a row of data, there are two necessary conditions: first, you can get the name of the data table, and the second is the only condition that locates the data row, usually the primary key value. Therefore, for a result set of a connection query, you can use standard SQL to update the data by using JDBC, as long as you can confirm the table corresponding to the modified data column and the corresponding primary key value for the row in which the column is located.