Research on an incremental processing mechanism of parallel data based on NoSQL

Research on an incremental processing mechanism of parallel data based on NoSQL

Huazhong University of Liu

Cloud computing brings new opportunities and challenges to data processing. The traditional relational database is unable to meet the requirement of large data, and the distributed non relational database can satisfy the high concurrency and high demand of reading and writing because of its high reliability and high availability. However, the superior performance of distributed, relational databases is at the expense of SQL data-processing capabilities. Therefore, it is an important research topic to improve the data processing ability of the non relational database. Non-relational database data processing ability can be improved from off-line data processing and online data processing in two aspects. In the field of off-line data processing, this paper combines the open source MapReduce data processing system Hadoop with the non relational database, realizes the Hadoop job configuration module, the data fragment module, the data input and the output module for the non relational database, Hadoop can directly deal with the massive data stored in the non relational database, which preserves the high reliability and high availability of the relational database, and enhances the system's off-line batch processing capability. In the field of online data processing, a distributed multiple-line transaction algorithm is implemented through a series of storage control processes based on the distributed single transaction function built in the relational database. Secondly, the message notification mechanism of similar triggers is implemented in the non relational database by adding redundant columns to the database table and calling the registration hook function for the database system. According to the multiple-line transaction algorithm and the message notification mechanism, incremental data processing is implemented in the non relational database to meet the needs of online data processing. Tested data sets containing 4,200,000 structured data records.

Research on an incremental processing mechanism of parallel data based on NoSQL