DTS development record (9) -- data migration task datatask

In the DTs development record (1)-system overall structure, we can see that a data transfer task is performed based on a connection pair and a data ing. However, we didn't mention the "data migration task" element before. With the evolution of the design, we found that a data task is no longer just a connection pair plus a data ing. Now, it is necessary to abstract the concept of "data migration task", which is called a ask.

With datatask, our data engine can be modified to execute based on a specified datatask. Datatask not only encapsulates the connection pairs and data ing for data migration, but also the control/configuration information of data migration. In addition, the configuration for scheduled data transfer task execution is also encapsulated in datatask. After the datatask is abstracted, We can reuse it in multiple places:
(1) Data Transmission Engine
(2) Data check
(3) timed setting
(4) Detailed task settings

All data migration tasks are serialized and stored in the configuration file of the DTs system.ProgramAt startup, all existing data migration tasks are automatically loaded. The datatask class diagram is as follows:

The usetrigger attribute specifies whether to use a trigger to obtain data increments when the data source is a database (the premise is that we must have the permission to create a trigger and add a scale ). Datatransferoption records all detailed settings about this task. The timerconfiguration attribute is the scheduled execution time of the task.

The following describes the datatransferoption and timerconfiguration in detail. First, let's take a look at datatransferoption.
(1) The dtstype attribute specifies the data export type: all export and incremental export.
(2) For incremental export, the increaseadd, increaseupdate, and increasedelete attributes indicate the increments to be exported.
(3) The localfilelogenabled attribute indicates whether to record local file logs.
(4) The sourcefieldinvalidhandletype attribute specifies the processing method when the data source does not meet the conversion conditions: Ignore, rollback, and log recording.
(5) dblogtype indicates the type of database logs: whether to record logs to the source database or the target database.
(6) sourcepagesize is the page size when the data source is read by page.
(7) The sourcewherestr attribute is a condition statement for filtering source data.
(8) The whereenabled attribute controls whether sourcewherestr is valid.

Timerconfiguration records the detailed settings of the scheduled task. There are several ways to execute the scheduled task:
(1) Hourly
(2) daily
(3) Weekly
(4) Monthly
(5) at specified intervals

Public   Enum Timertype
{
Perday, perhour, permonth, perweek, everyspan
}

Scheduled tasks are usually combined with incremental export.

All data migration tasks in the system are managed by dtconfiguration. Similarly, dtconfiguration manages all connection pairs in the system. Except that all the ing files are stored in a specific directory, rather than managed by dtconfiguration. This is because we often need to copy a ing file to another place. The dtconfiguration class diagram is as follows:

Go to: DTS development records (sequence)