Small-range Chat microservices-data extraction that thing (a)

First, preface

We mentioned in the "Micro Service is a double-edged sword http://blog.csdn.net/u013970991/article/details/73195907" when we put the application service, Many of the data statistics and analysis operations that can be carried out in a single-block system will be greatly affected, and this article will continue with an in-depth analysis of the previous article, as a back-end data statistics and analysis of how to do.

Note: The database in this article is based on the Oracle database second, the status quo analysis after service

Take a simple quick payment system as an example, after the service of the system call diagram is as follows:

Through the above figure, we can see that the single-block system according to the service of the business, each system functions a single, clear responsibility and independent deployment, this is only from the perspective of the system to describe the service of the call relationship, then from the point of view of micro-service, there is also a point to the central, that is, the database is also split The diagram below shows the relationship between each service and its corresponding database.

Above we can see that each service corresponds to a database, so from the top to the bottom has been all split apart, and then combined with the theory of Conway Law, each service by a team responsible for the management, the team to collaborate and communicate with each other. three, the data extraction technology selection

On the background of data statistics requirements, because the database has been split after the service, so the background data statistics caused a certain disturbance, in response to this problem I first think of the use of database synchronization to solve, the different libraries or tables of data unified together. So next, I will be with you to explore and analyze gradually. 1. Using the Oracle Golden Gate (OGG) tool

The implementation of the Ogg principle is to extract the source side of the redo log and archive log, and then post through the TCP/IP protocol to the target side, and finally resolve the restore synchronization to the target side, so that the target side to achieve the source side of the data synchronization, as shown in the following figure:

1.1 Advantages of using OGG:

1, the impact on the production system is small: real-time read transaction log, with low resource consumption to achieve large transaction volume data real-time replication.
2, the transaction as a unit copy, to ensure the consistency of the transaction: only the submitted data synchronization.
3, Flexible topology: support a pair of one or one-to-many, many-to-one, many-to-many and two-way replication.
4, you can customize the filter rules based on the table and row, you can perform flexible mapping and transformation of real-time data. 1.2 points to note when using Ogg:

1. When synchronizing data between two libraries, if we want to add fields to the table, we must stop Ogg to add the field and then start the new field synchronization will take effect.
2, when using Ogg to do data synchronization, the tool is not very stable, often appear suspended animation or exit situation.
3, Ogg occasionally appear in the synchronization process when the data lost. 2. Using Oracle Logminer

Logminer is a tool that Oracle provides from 8i to analyze redo log information, including the DBMS_LOGMNR and Dbms_logmnr_d two package, and the back D is the meaning of the dictionary. It can analyze both the redo log file and the archived archive log file. Logminer enables you to track all DML, DDL, and DCL operations of the Oracle database. 2.1 The framework diagram for data synchronization using Logminer is as follows:

the 2.2 Data synchronization flowchart is as follows:

Synchronization Process Description::

The source-side synchronization data is obtained by specifying configuration information, such as source-side, target-side database information, Logminer synchronization time, and so on.

1. Check whether the data synchronization time is reached through timed polling, and if it is, synchronize the data, or continue polling.
2, the time to load the database archive log files into the dynamic table v$logmnr_contents.
3. Read the specified SQL statement according to the conditions.
4. Execute the SQL statement.

The core code for the data synchronization section of Java-based Logminer is as follows:

try {ResultSet ResultSet = null; Get source database Connection Sourceconn = Database.getsourcedatabase ();
    Statement Statement = Sourceconn.createstatement (); Add all log files, this code only analyzes the online log stringbuffer sbsql = new StringBuffer ();
    Sbsql.append ("BEGIN"); Sbsql.append ("Dbms_logmnr.add_logfile" (logfilename=> ' "+constants.log_path+" \\REDO01. LOG ', OPTIONS=>DBMS_LOGMNR.
    NEW); "); Sbsql.append ("Dbms_logmnr.add_logfile" (logfilename=> ' "+constants.log_path+" \\REDO02. LOG ', OPTIONS=>DBMS_LOGMNR.
    AddFile); "); Sbsql.append ("Dbms_logmnr.add_logfile" (logfilename=> ' "+constants.log_path+" \\REDO03. LOG ', OPTIONS=>DBMS_LOGMNR.
    AddFile); ");
    Sbsql.append ("END;"); CallableStatement callablestatement = Sourceconn.preparecall (sbsql+ "");
    Callablestatement.execute ();

    Print the analyzed log file information ResultSet = Statement.executequery ("Select Db_name, thread_sqn, filename from v$logmnr_logs"); 
   while (Resultset.next ()) {System.out.println ("added log file ==>" +resultset.getobject (3)); } System.out.println ("Start analysis log file, starting SCN number:" +CONSTANTS.LAST_SCN); CallableStatement = Sourceconn.preparecall ("begindbms_logmnr.start_logmnrstartscn=>" "+Constants.LAST_SCN+" ', Dictfilename=> ' "+constants.data_dictionary+" \\dictionary.ora ', OPTIONS =>DBMS_LOGMNR. COMMITTED_DATA_ONLY+DBMS_LOGMNR.NO_ROWID_IN_STMT);
    END; "); 
    Callablestatement.execute ();
    SYSTEM.OUT.PRINTLN ("Complete analysis log File"); 
    Query Get analysis Results System.out.println ("Query analysis results"); ResultSet = Statement.executequery ("Select Scn,operation,timestamp,status,sql_redo from V$logmnr_contents WHERE seg_

    Owner= ' "+constants.source_client_username+" ' and Seg_type_name= ' TABLE ' and operation! = ' select_for_update ');
    Connect to the target database, execute the REDO statement in the target database Targetconn = Database.gettargetdatabase ();
    Statement targetstatement = Targetconn.createstatement (); String LASTSCN = CONSTANTS.LAST_SCN;
    String operation = NULL;
    String sql = null; Boolean iscreatedictionary = false; while (Resultset.next ()) {LASTSCN = Resultset.getoBject (1) + ""; 
    if (Lastscn.equals (CONSTANTS.LAST_SCN)) {continue; 
    } operation = Resultset.getobject (2) + "";
    if ("DDL". Equalsignorecase (Operation)) {iscreatedictionary = true;
    } sql = Resultset.getobject (5) + ""; Replace User sql = Sql.replace ("\" "+constants.source_client_username+" \ ".", "");
    System.out.println ("scn=" +lastscn+ ", automatically executes sql==" +sql+ ");
    try {targetstatement.executeupdate (sql.substring (0, Sql.length ()-1));
    } catch (Exception e) {System.out.println ("tested, already executed");}
    }//Update SCN CONSTANTS.LAST_SCN = (Integer.parseint (LASTSCN)) + "";
        DDL changes, update data dictionary if (iscreatedictionary) {System.out.println ("DDL changed, update data dictionary"); 
        CreateDictionary (Sourceconn); 
        SYSTEM.OUT.PRINTLN ("Complete Update data dictionary");
    Iscreatedictionary = false;
} System.out.println ("Complete a unit of work");
    } finally {if (null! = Sourceconn) {sourceconn.close ();
      } if (null! = Targetconn) {  Targetconn.close ();
    } sourceconn = null; 
    Targetconn = null; }
}

2.3 points to note when using Logminer for data synchronization:

1. Logminer is synchronization at the database level.
2, Logminer tool timeliness is poor, synchronization delay time is very long.
3. The target library must be the same as the source library version, or higher than the source library version, the target library is consistent with the source library character set, or is a superset of the source library character set.
4, the source database and the target library, must be running on the same hardware platform.
5, through the Logminer way to get the log, through the Oracle provides tools to read the redo log information, and then parse into a SQL queue. Some special data types, data changes are not recorded to redo, such as LOB field changes 3, summary

The above two kinds of solutions have advantages and disadvantages, but the actual work requires less synchronization delay, while the stability of excellent and very low data loss rate of the scheme, you can see that these two scenarios are not suitable for a real data extraction tool, to achieve a scenario as follows:

In the next article, I will combine the practical work to introduce an efficient database synchronization tool, and finally solve the micro-service implementation of the data generated by the pain point.

Note: This article references the http://www.cnblogs.com/shishanyuan/p/3142788.html