Environment configuration of otter Based on alibaba Open-Source Distributed Data Synchronization System installation configuration document, alibabaotter

Environment configuration of otter Based on alibaba Open-Source Distributed Data Synchronization System installation configuration document, alibabaotter

Otter Project Open Source Address: Workshop.

Working principle:   Principle description:   1. Obtain incremental Database Log Data Based on Canal open-source products. What is Canal? Click 2. typical management system architecture: manager (web Management) + node (Work node). when the manager is running, push the synchronization configuration to node B. the node sends the synchronization status back to the manager. 3. based on zookeeper, it solves Distributed State scheduling and allows multi-node nodes to work collaboratively.

Component explanation: canal: What is canal? Otter open source before a sub-project, open source link address: http://github.com/alibaba/canal positioning: based on the database incremental Japanese parsing, provide incremental data subscription & consumption, currently mainly supports the working principle of mysql: The principle is relatively simple: similar to the original master-slave replication mechanism of MYSQL. 1. canal simulates the interaction protocol of mysql slave, disguising itself as a mysql slave, and wants the mysql master to send the dump Protocol 2. the mysql master receives the dump Request and starts to push the binary log to slave (canal) 3. canal explains binary log objects (originally byte streams) related documents: See the wiki page for: wiki documentation

Environment configuration: canal and otter depend on jdk environment and node depends on aria2. Start: apt-get update & apt-get install default-jdk aria2-y

Component installation configuration: canal:. the canal principle is based on mysql binlog technology. binlog is required, and the format of log is ROW: [mysqld] log-bin = mysql-bin # Add this line. OK binlog-format = ROW # select row mode server_id = 1 # configure mysql replaction to be defined, it cannot be the same as the canal slaveId B. authorize slave synchronization: create user canal identified by 'canal'; grant select, replication slave, replication client on *. * TO 'canal' @ '%'; or: grant all privileges on *. * TO 'canal' @ '%'; flush privileges; unzip and use it directly. Configuration File: vi conf/example/instance. properties ####################################### ########### mysql serverId canal. instance. mysql. slaveId = 1234 # position info. You need to change it to your database information canal. instance. master. address = 127.0.0.1: 3306 # specify the master's ip address: port canal. instance. master. journal. name = # You do not need to configure binlog file (currently started by default) canal. instance. master. position = # You do not need to configure the postion of binlog (the default value is the current one) canal. instance. master. timestamp = # canal. instance. standby. address = # canal. instance. standby. journal. name = # canal. instance. standby. position = # canal. instance. standby. timestamp = # username/password, which must be changed to your own database information canal. instance. dbUsername = canal # configure as the authorized account canal. instance. dbPassword = canal # configure the authorization password canal. instance. defaultDatabaseName = # You can specify the database canal. instance. connectionCharset = UTF-8 # configure the encoding format # table regex canal. instance. filter. regex =. *\\.. *####################################### ######### start and stop: bin/startup. sh bin/stop. sh zookeeper: Installation configuration: configure the hosts file 192.168.56.1 zk 192.168.56.2 re 192.168.56.3 yt zk configuration file for all zk hosts: Vim conf/zoo. cfg ####################################### ########## The number of milliseconds of each tickTime = 2000 # The number of ticks that the initial # synchronization phase can take initLimit = 10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit = 5 # the directory where the snapshot is stored. # do not use/tmp for storage,/tmp here Is just # example sakes. dataDir =/usr/local/zk_cluster/zookeeper-3.4.6_node1/data # the Directory needs to be created # dataLogDir # the port at which the clients will connect clientPort = 2181 # the ports on each node are different server.1 = zk: 2887: 3892 # whether it is the front port or the back port. Each node is different from server.2 = zk: 2888: 3893 server.3 = zk: 2889: 3894 server.4 = re: 3386: 3388: observer # remote zkobserver role server.5 = re: 3387: 3389: observer # remote zkobserver role server.6 = yt: 2892: 3895: observer # remote zkobserver role # the maximum number of client connections. # increase this if you need to handle more clients observer configuration: # The number of milliseconds of each tick tickTime = 2000 # The number of ticks that the initial # Synchronization phase can take initLimit = 10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit = 5 peerType = observer # specify this node as the observer type # The directory where the snapshot is stored. # do not use/tmp for storage,/tmp here is just # example sakes. dataDir =/usr/local/zk_cluster/zookeeper-3.4.6_node4/data # dataLogDir # the port at which the clients Will connect clientPort = 2184 server.1 = zk: 2887: 3892 server.2 = zk: 2888: 3893 server.3 = zk: 2889: 3894 server.4 = re: 2890: 3890: observer server.5 = re: 2891: 3891: observer server.6 = yt: 2892: 3895: observer # the maximum number of client connections. # increase this if you need to handle more clients start, stop: zookeeper-3.4.6_node4/bin/zkServer. sh started zookeeper-3.4.6_node4/bin/zkServer. sh stop Note: each node has a unique myid. Create a myid file in the data directory and write the server. x id of the local node to myid. The data directory needs to be created. For example, the first node: server.1 = zk: 2887: 3892 only needs to input 1 in the myid file. otter: manager is the web management interface, to support the mysql database, deploy mysql and grant permissions on the manager. You also need the original data, load the original data; download it online: wget https://raw.github.com/alibaba/otter/master/manager/deployer/src/main/resources/ SQL /otter-manager-schema. SQL configuration file: vim conf/otter. properties # otter manager domain name # modify it to the correct ip address and generate a URL using otter. domainName = 192.168.56.4 # configure the access domain name or ip address # otter manager http port otter. port = 8080 # configure the web access port # jetty web config xml otter. jetty = jetty. xml # Modify otter manager database config to the correct database information Otter. database. driver. class. name = com. mysql. jdbc. driver otter. database. driver. url = jdbc: mysql: // 127.0.01: 3306/ottermanager # configure the manager link database otter. database. driver. username = otter # configure the user name otter for database connection. database. driver. password = otter # configure the password for connecting to the database # Otter communication port otter. communication. manager. port = 1099 # configure the node link port # Otter communication pool size Otter. communication. pool. size = 10 # Default zookeeper address, modify it to the correct address, and manually select a zookeeper cluster list otter in the nearest region. zookeeper. cluster. default = 192.168.56.1: 2181,192.168 .56.1: 2182,192.168 .56.1: 2183,192.168 .56.2: 3384,192.168 .56.2: 3385,192.168 .56.3: 2186 # configure a nearby zk cluster address to write one closest to the manager. For example, 192.168.56.1: 2181 ## default zookeeper sesstion timeout = 90 s Otter. zookeeper. sessionTimeout = 90000 # Otter arbitrate connect manager config otter. manager. address =$ {otter. domainName }:$ {otter. communication. manager. port Start and Stop: bin/startup. sh bin/stop. sh access url: http: // 192.168.168.4: 8080 default account password: admin/admin node: vim conf/otter. properties # otter node root dir otter. nodeHome =$ {user. dir }/.. // ## otter node dir otter. htdocs. dir =$ {otter. nodeHome}/htdocs otter. download. dir =$ {otter. nodeHome}/downlo Ad otter. extend. dir =$ {otter. nodeHome}/extend # default zookeeper sesstion timeout = 60 s otter. zookeeper. sessionTimeout = 60000 # otter communication pool size otter. communication. pool. size = 10 ## otter arbitrate & node connect manager config otter. manager. address = 192.168.56.4: 1099 # specify the manager's ip address: port to start and stop. Note: To start a node, configure nid first. Bin/startup. sh bin/stop. sh Note: here we mention an nid. This is the unique ID of node. How can we get the ID of This nid? This is generated by the node added on the manager, which will be mentioned below. This nid file needs to be in the conf directory of node; it is just to add the ID number. Nid is located in the conf directory. Next document: manager Single Instance configuration based on alibaba Open-Source Distributed Data Synchronization System installation and configuration document otter