MySQL 39th article ~canal in-depth interpretation 1

A brief introduction: After a period of research, the canal has some insight

Two configuration files:

1 canal.properties (System root configuration file)

main parameter list
Canal.properties (System root configuration file)
canal.destinations= example the instance list deployed on the current server defaults to Example After canal.properties defines the canal.destinations, a file of the same name needs to be established in the Canal.conf.dir corresponding directory

Canal.auto.scan turn on instance automatic scanning if the configuration changes to the instance configuration in the True,canal.conf.dir directory will automatically trigger:
A. Instance directory added: trigger instance configuration load, lazy is true when it starts automatically
B. Instance directory removal: Uninstall the corresponding instance configuration and close if it is started
C. instance.properties file changes: Reload instance configuration, such as auto-overloading started

  canal.auto.scan.interval=5 scan interval default 5S
  canal.instance.global.mode=spring Global configuration load mode,
  canal.zkservers= cluster ZK cluster configuration
  #binlog配置
  Canal.instance.binlog.format = row,statement,mixed Supported Binlog formats
  canal.instance.binlog.image = Full,minimal,noblob supported Binlog
  #sql语句过滤
  CANAL.INSTANCE.FILTER.QUERY.DCL = False permission and stored procedure related statements set to false means that
 canal.instance.filter.query.dml = false is not discarded Delete Add and change related statement set to false means do not discard
 canal.instance.filter.query.ddl = False DDL-related statements set to false means not discarded
  Canal.instance.filter.table.error = False

Three about canal destinations
1 when the server is first started, the Conf directory is automatically scanned, and the file name is instance name, starting the corresponding instance
2 during server operation, the scan is performed according to the frequency defined by the Canal.auto.scan.interval
1. Discovery directory has been added, start a new instance
2. Found the directory has deleted, close the old instance
3. The instance.properties of the corresponding directory is found to be changed and restarted instance
3 You can see from the above that no more instance definitions are added to the configuration file, and the scan automatically discovers

Four Binlog about the canal scan.
1 Canal.instance.master.journal.name + canal.instance.master.position: precise designation of a binlog point for start-up
2 Canal.instance.master.timestamp: Specify a timestamp, the canal will automatically traverse the MySQL binlog, find the corresponding timestamp Binlog point, start
3 do not specify any information: The default is to start from the current database's bit point. (Show master status)
Five examples of canal's meta
1 under each instance if the app reads, there is a meta file that records the details of the synchronization
{"Clientdatas": [{"clientidentity": {"clientId": 1001, "Destination": "dbname", "Filter": "Filter Rule"}, "cursor": {"identity ": {" SlaveID ":-1," sourceaddress ": {" Address ":" IP "," Port ": 3306}," postion ": {" included ": false," Journalname ":" mysql-bin.000085 "," position ": 283121001," timestamp ": 1512620199000}}]," Destination ":" DBName "}
Includes filter rules, instance name, synchronization information (main library IP filename,position)
2 whether the current instance is read by the program can be determined by meta presence or not
Vi. the use of the canal in relation to consumption
1 Separate instance can only be consumed by one client
2 If the first client stops spending, Binlog will inherit the first client to consume when the second client consumes
3 as long as the target database makes transaction changes, the backend consumption will appear transaction display
Seven about the canal and ZK cooperation
1 build two groups of canal services and a set of ZK configurations
2 Canal.zkservers=ip:port
3 Restart two groups of canal in force
4 Enter ZK to view get/otter/canal/destinations/example/running
{"Active": true, "Address": "CannalIP1", "CID": 1}
5 Stop access to the canal and re-enter ZK to view
{"Active": true, "Address": "CannalIP2", "CID": 1}

Eight questions about the canal development
1 about the single transaction filtering problem, there is a lot of transaction, how to filter, this is a problem

MySQL 39th article ~canal in-depth interpretation 1