Zookeeper Application Scenario: configure file synchronization

Question guidance:
1. What is the relationship between the three roles in this document?
2. What are the roles of the three roles?
3. How can the code implement the functions of these three roles?





Application scenarios of zookeeper: zookeeper addresses issues related to distributed cluster configuration file synchronization. This article describes how to apply ZK to configuration file distribution scenarios.
Assume there are three roles.

• Trigger: Release the latest configuration file data, send commands and data to zk_agent, and implement the following trigger. py
• Zk_agent: receives commands and data triggered by trigger. py, updates the data to ZK service, and triggers zk_app to obtain the latest configuration data.
• Zk_app: The znode where the configuration file in ZK is deployed on each worker to obtain the latest configuration file and apply it to the worker. The implementation of zk_app.py is as follows:
  
  <Ignore_js_op>

Configuration file synchronization to: zk_agent implementation logic:
* Initialize the connection to the zk service and first run for the master
* Master creates a znode of the persistent type managed by the configuration file, such as/applications/nginxconf,
* Start a thread and specifically receive the commands sent by the trigger. Once the command is received, create a sequence starting with "conf-" | a persistent node, specify that the znode data content is the data received from the trigger
* The node/applications/nginxconf/conf-0000000000 will be created upon the first update, and the number of configuration files will increase each time a new configuration file is updated.
Vim zk_agent.py

#! /Usr/bin/ENV python2.7 #-*-coding: UTF-8-*-# Author: firefoxbug # E-mail: [email protected] # blog: www. firefoxbug. netimport loggingfrom OS. path import basename, joinfrom zkclient import zkclient, Zookeeper, watchmethodimport osimport sysimport threadingimport signalimport timelogging. basicconfig (Level = logging. debug, format = "[% (asctime) S] % (levelname)-8 S % (Message) s") log = loggingclass gjz Ookeeper (object): zk_host = "localhost: 2181" root = "/roles" workers_path = join (root, "Workers ") masters_num = 1 timeout = 10000 def _ init _ (self, verbose = true): Self. verbose = verbose self. masters = [] self. is_master = false self. path = none self. app_root = "/Applications" self. app_conf = join (self. app_root, "nginxconf") self. zk = zkclient (self. zk_host, timeout = self. timeout) self. say ("Login Zookeeper successfully! ") # Init self. create_roles_znode () # register self. register () def create_roles_znode (Self): "" create the zookeeper node if not exist |-roles |-workers "nodes = (self. root, self. workers_path) for node in nodes: If not self. zk. exists (node): Try: Self. zk. create (node, "") failed T: Pass @ property def is_slave (Self): return not self. is_master def register (Self): "" register a node for this worker, Z Node Type: ephemeral | sequence |-roles |-workers |-worker000000000x ==>> master |-worker000000000x + 1 ==>> worker .... "self. path = self. zk. create (self. workers_path + "/worker", "1", flags = zookeeper. ephemeral | zookeeper. sequence) self. path = basename (self. path) self. say ("I'm % s" % self. path) # check who is the master self. get_master () def get_master (Self): "" Get children, and check who is Smallest child "@ watchmethod def watcher (event): Self. say ("child changed, try to get master again. ") self. get_master () Children = self. zk. get_children (self. workers_path, Watcher) children. sort () self. say ("% s's children: % s" % (self. workers_path, children) # Check if I'm master self. masters = children [: Self. masters_num] If self. path in self. MASTERS: Self. is_master = true self. say ("I 've become Master! ") Self. create_app_znode () else: Self. say ("% s is Masters, I'm slave" % self. masters) def create_app_znode (Self): "create the zookeeper node if not exist |-Applications |-nginxconf" nodes = (self. app_root, self. app_conf) for node in nodes: If not self. zk. exists (node): Try: Self. say ("create znode [% s]... "% (node) self. zk. create (node, "") failed T: Pass def create_conf_znode (self, data): "" create the zookeeper node's children if not exist, contents is conf data |-Applications |-nginxconf |-item-000000000x => data "" self. child_node = join (self. app_conf, "conf-") Path = self. zk. create (self. child_node, Data, flags = zookeeper. sequence | zookeeper. persistent) self. say ("create znode % s" % PATH) def say (self, MSG): "print messages to screen" If self. verbose: If self. path: log.info ("[% s (% s)] % s" % (self. path, "Master" If self. is_master else "slave", MSG) else: log.info (MSG) Class watcher: def _ init _ (Self): "" creates a child thread, which returns. the parent thread waits for a keyboardinterrupt and then kills the child thread. "self. child = OS. fork () If self. child = 0: Return else: Self. watch () def Watch (Self): Try: OS. wait () cipher t keyboardinterrupt: Print 'exit... 'self. kill () sys. exit () def kill (Self): Try: OS. kill (self. child, signal. sigkill) failed t oserror: passdef start_zk_worker (): "connected to zookeeper for initialization" gj_zookeeper = gjzookeeper () Th1 = threading. thread (target = start_agent_worker, name = "thread_1", argS = (gj_zookeeper,) th1.start () th1.join () def start_agent_worker (gj_zookeeper): "listens for configuration file change information, resolution command "" Import socket address = ('', 8877) S = socket. socket (socket. af_inet, socket. sock_stream) # S = socket. socket () s. BIND (Address) s. listen (5) print "listen on port 8877... "While true: SS, ADDR = S. accept () print "receive connetcion from", ADDR content = "" While true: Try: Data = ss. recv (512) if not data: Print "Close connetcion", ADDR ss. close () Break content = content + Data print "Receive message from % s: % s" % (ADDR [0], data) failed t exception, E: print "receive error from % s: % s" % (ADDR [0], STR (E) ss. close () Break parse_trigger_package (content, gj_zookeeper) s. close () def parse_trigger_package (data, gj_zookeeper): Try: cmd = data. split ('|') [0] content = data. split ('|') [1] failed t exception, E: Print "error:", STR (e) return if cmd = "add": gj_zookeeper.create_conf_znode (content) else: passdef main (): Watcher () start_zk_worker () If _ name _ = "_ main _": Main () import time. sleep (20000)

　　

Configuration File receiving application: the logic of zk_appzk_app is as follows, which works on each worker

• Initialize connection to ZK Service
• Get the znode list of all/applications/nginxconf and set watcher
• Find the largest znode node in the list, record its number, and getdata obtains its data. The latest configuration information is obtained here.
• Each time watcher is triggered, the node with the largest number in the list is obtained, and the last saved number is compared with it. Once updated, the data is retrieved again and applied to the worker, if they are the same, you do not need to obtain data on zk.

Vim zk_app.py

#! /Usr/bin/ENV python2.7 #-*-coding: UTF-8-*-# Author: firefoxbug # E-mail: [email protected] # blog: www. firefoxbug. netimport loggingfrom OS. path import basename, joinfrom zkclient import zkclient, Zookeeper, watchmethodimport osimport sysimport threadingimport signalimport timelogging. basicconfig (Level = logging. debug, format = "[% (asctime) S] % (levelname)-8 S % (Message) s") log = loggingclass gjz Ookeeper (object): zk_host = "localhost: 2181" root = "/roles" workers_path = join (root, "Workers ") masters_num = 1 timeout = 10000 def _ init _ (self, verbose = true): Self. verbose = verbose self. masters = [] self. is_master = false self. path = none self. app_root = "/Applications" self. app_conf = join (self. app_root, "nginxconf") self. zk = zkclient (self. zk_host, timeout = self. timeout) self. say ("Login Zookeeper successfully! ") # Init self. create_roles_znode () # register self. register () def create_roles_znode (Self): "" create the zookeeper node if not exist |-roles |-workers "nodes = (self. root, self. workers_path) for node in nodes: If not self. zk. exists (node): Try: Self. zk. create (node, "") failed T: Pass @ property def is_slave (Self): return not self. is_master def register (Self): "" register a node for this worker, Z Node Type: ephemeral | sequence |-roles |-workers |-worker000000000x ==>> master |-worker000000000x + 1 ==>> worker .... "self. path = self. zk. create (self. workers_path + "/worker", "1", flags = zookeeper. ephemeral | zookeeper. sequence) self. path = basename (self. path) self. say ("I'm % s" % self. path) # check who is the master self. get_master () def get_master (Self): "" Get children, and check who is Smallest child "@ watchmethod def watcher (event): Self. say ("child changed, try to get master again. ") self. get_master () Children = self. zk. get_children (self. workers_path, Watcher) children. sort () self. say ("% s's children: % s" % (self. workers_path, children) # Check if I'm master self. masters = children [: Self. masters_num] If self. path in self. MASTERS: Self. is_master = true self. say ("I 've become Master! ") Self. create_app_znode () else: Self. say ("% s is Masters, I'm slave" % self. masters) def create_app_znode (Self): "create the zookeeper node if not exist |-Applications |-nginxconf" nodes = (self. app_root, self. app_conf) for node in nodes: If not self. zk. exists (node): Try: Self. say ("create znode [% s]... "% (node) self. zk. create (node, "") failed T: Pass def create_conf_znode (self, data): "" create the zookeeper node's children if not exist, contents is conf data |-Applications |-nginxconf |-conf-000000000x => data "" self. child_node = join (self. app_conf, "conf-") Path = self. zk. create (self. child_node, Data, flags = zookeeper. sequence | zookeeper. persistent) self. say ("create znode % s" % PATH) def say (self, MSG): "print messages to screen" If self. verbose: If self. path: log.info ("[% s (% s)] % s" % (self. path, "Master" If self. is_master else "slave", MSG) else: log.info (MSG) Class watcher: def _ init _ (Self): "" creates a child thread, which returns. the parent thread waits for a keyboardinterrupt and then kills the child thread. "self. child = OS. fork () If self. child = 0: Return else: Self. watch () def Watch (Self): Try: OS. wait () cipher t keyboardinterrupt: Print 'exit... 'self. kill () sys. exit () def kill (Self): Try: OS. kill (self. child, signal. sigkill) failed t oserror: passdef start_zk_worker (): "connected to zookeeper for initialization" gj_zookeeper = gjzookeeper () Th1 = threading. thread (target = start_agent_worker, name = "thread_1", argS = (gj_zookeeper,) th1.start () th1.join () def start_agent_worker (gj_zookeeper): "listens for configuration file change information, resolution command "" Import socket address = ('', 8877) S = socket. socket (socket. af_inet, socket. sock_stream) # S = socket. socket () s. BIND (Address) s. listen (5) print "listen on port 8877... "While true: SS, ADDR = S. accept () print "receive connetcion from", ADDR content = "" While true: Try: Data = ss. recv (512) if not data: Print "Close connetcion", ADDR ss. close () Break content = content + Data print "Receive message from % s: % s" % (ADDR [0], data) failed t exception, E: print "receive error from % s: % s" % (ADDR [0], STR (E) ss. close () Break parse_trigger_package (content, gj_zookeeper) s. close () def parse_trigger_package (data, gj_zookeeper): Try: cmd = data. split ('|') [0] content = data. split ('|') [1] failed t exception, E: Print "error:", STR (e) return if cmd = "add": gj_zookeeper.create_conf_znode (content) else: passdef main (): Watcher () start_zk_worker () If _ name _ = "_ main _": Main () import time. sleep (20000)

　　

Configuration File Sending: trigger implements simple logic, namely simulating to send data packets to zk_agent. The format is as follows:

1. Command | data
2. Add | helloworld

Vim trigger. py

#!/usr/bin/pythonimport socket  import syssock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)  sock.connect((‘127.0.0.1‘, 8877))str = sys.argv[1] + ‘|‘ + sys.argv[2]sock.send(str)  sock.close()

Run

$ python zk_agent.py$ python zk_app.py$ python trigger.py

　　

Zookeeper Application Scenario: configure file synchronization