Intermediate table synchronization scheme (full, incremental)

A big data statistics and analysis system has been developed recently.
Http://www.cnblogs.com/edobnet/archive/2005/04/24/144521.html
Http://edobnet.cnblogs.com/archive/2005/11/11/273947.html

To improve performance, the synchronization of intermediate tables adopts the scheme of full synchronization at the beginning, daily and incremental synchronization.
Let me introduce our incremental solution!
To increment, the incremental log table is required. The design of the incremental log table is as follows.

Key (original business table keyword), create_date (Change Time), flag (data modification and deletion flag), and use_flag (whether the incremental table is used or not, (One bucket is used for filtering ))

Perform the following operations during daily incremental synchronization: -- If the incremental log table has all the flag spaces used, move the incremental log table to the incremental log backup table.
Insert Log_manual_inc_bak
Select   *
From Log_manual_inc
Where   Left (Using_flag, 4 ) = ' 1111 '

Delete Log_manual_inc Where   Left (Using_flag, 4 ) = ' 1111 '

-- Create a temporary intermediate table to facilitate synchronization (the Field content is the same as that of the real table)
Create   Table # Tg_entry (
[ Te_entry_id ]   [ Char ] ( 18 ) Not   Null  
..
)
Go

-- ------------------- Delete unintegrated records from the intermediate table
-- Substring (using_flag,) indicates the flag used for this integration.
  Delete   From Tg_entry Where Tg_entry_id In ( Select Tg_entry_id From Log_manual_inc Where   Substring (Using_flag, 2 , 1 ) <> 1 )

.. Now we can synchronize the data,
... The synchronized data is obtained from the original business table. However, the data range is limited. The incremental table does not perform incremental operations (and the operation type is newly added and modified ).

Add a where condition synchronously,
Where log_manual_inc.arj_mark = 'M' and substring (using_flag, 2, 1) <> 1
The retrieved data is stored in the newly created temporary intermediate table, # tg_entry

After synchronization, do not forget to add more table labels.