This is a creation in Article, where the information may have evolved or changed.
"Editor's note" The ETCD team recently released a beta version of ZETCD, a proxy tool that allows ETCD to be compatible with zookeeper protocols, allowing users to disassociate existing applications from the Zookeeper service.
Distributed systems usually rely on an arbitration system to work together, and generally such a system ensures accurate communication of information through arbitration to avoid brain fissures. Such systems have been given ample room for design by sacrificing versatility, and this has clearly been exemplified by the various concrete implementations that are constantly spreading. There are many such systems, such as CHUBBY,ZOOKEEPER,ETCD and consul. Although the concepts and protocols of these systems differ, they provide similar key-value-based distributed arbitrations. As part of the most anticipated infrastructure component of the distributed system, the ETCD team developed a new agent, ZETCD, which allows ETCD clusters to consume zookeeper service requests without any changes to the ETCD.
Zookeeper is the first open source implementation of the Quorum software, which makes it a backend for many distributed system preferences. In theory, these systems should be compatible with ETCD, but for historical reasons this is not the case; ETCD clusters cannot replace zookeeper, their data models and client protocols are incompatible with zookeeper applications, and vice versa. If the system is already in production, there is little incentive to push it into the new backend, introducing new complexities. Fortunately, the ETCD V3 API is preparing to simulate support for the Zookeeper data model through a standard proxy zetcd, ZETCD is a new open source project developed by the ETCD team, which released the first beta version of Zetcd today, v0.0.1, the goal is to manage and deploy ZETCD systems in production systems.
The ZETCD agent is deployed in front of the ETCD cluster and serves a simulated zookeeper client port, enabling zookeeper applications to invoke ETCD natively in the upper layer. In general,
zetcdAccepts zookeeper customer requests, translates them into ETCD data models and APIs, forwards requests to ETCD, and forwards the returned information back in a way that the client can understand. The performance of ZETCD is comparable to that of zookeeper and simplifies the management and operational complexities between zookeeper clusters and ETCD. This article will reveal how ZETCD,ZETCD works and performance benchmarks are used.
Zetcd Getting Started
ZETCD is required to run a go compiler, get source code from the Internet, and a system that can run ETCD. The following example shows how to run the Zookeeper command from the ZETCD source to the ZETCD. Since both ETCD and ZETCD are built on the basis of the development branch, it is not recommended to do so in a production environment, which is just a simple example of how to use it.
First, obtain the ETCD and ZETCD source code, and compile it into binary codes:
Go get GITHUB.COM/COREOS/ETCD/CMD/ETCD
Go get GITHUB.COM/COREOS/ZETCD/CMD/ZETCD
Next, run the ETCD and connect the ZETCD to the ETCD customer service side:
#etcd uses localhost:2379 by default
ETCD &
Zetcd-zkaddr localhost:2181-endpoints localhost:2379 &
Try zetd by adding a subscription and creating a key:
Go Install Github.com/coreos/zetcd/cmd/zkctl
Zkctl Watch/&
Zkctl create/abc "foo"
Conceptually, the above example is done by adding a layer of ZETCD on a single ETCD instance.
What is ZETCD on this floor?
Support for zookeeper in ETCD3
In depth, ZETCD will translate the zookeeper data model into a ETCD API. For key lookup, ZETCD converts the zookeeper hierarchical directory into ETCD flat binary space (flat binary keyspace). To manage metadata, when writing to the Etcd backend, ZETCD uses memory-level transactions to safely and atomically update the information to zookeeper Znode information.
Zookeeper lists key (GetChildren) as a directory, and Etcd is by interval (Range). Explains how ZETCD can encode key under ETCD to effectively support listing in a directory format. All ZETCD keys in Etcd have a prefix that includes the full directory name (for example: "/" and "/abc" represent depths of 0 and 1 respectively). To list a directory, ZETCD emits a prefixed range request (for example ["/zk/key/002/abc/", "/zk/key/002/abc0") to list key under all/abc/that satisfy the directory depth and path. The depth limit is only for the directory itself, and if the ZETCD uses only the path and not the depth, then ETCD will return all KEY,ZETCD in that directory and the result will be discarded, whereas the key in this directory is returned.
Each zookeeper key has some metadata in the Znode, that is, the key adjustment, version and permissions, and so on. Although ETCD also has metadata for each key, it is much simpler than znode, for example, because there is no sub-version of the directory, because ETCD uses a role-based authentication mechanism and therefore has no ACLs because the actual clock is out of scope, so there is no timestamp. These additional metadata is mapped to the corresponding key, which is used to describe a complete znode. When modifying metadata, ZETCD uses soft transactions at the memory level to automatically update a subset of keys, ensuring that znodes does not require expensive locking mechanisms to remain consistent.
In addition, ZETCD can do dynamic validation with an authorized zookeeper server. To make comparisons, ZETCD can connect to both ETCD and external zookeeper servers at the same time. When the client initiates a request to the ZETCD in this mode, the request is forwarded to both the ZETCD and the zookeeper server. If two servers respond with inconsistent data, ZETCD will identify a warning for this response.
Performance Benchmarks
Because of the data conversion and the additional network overhead, it may be tempting to think that such simulations are impractical. Although ZETCD has additional costs for zookeeper or ETCD clusters, it still has the advantage that some ETCD applications still need to be zookeeper to coordinate scenes after installation. For example, early users reported that TLS encrypted traffic through ETCD in ZETCD was simpler than a similar classic zookeeper configuration. In these scenarios, it is more important to support the simple reliability of the same zookeeper protocol than performance.
Similar to the interface and reporting form of the ETCD Performance tool, the ZETCD command-line tool zkboom can be used to determine whether a ZETCD performance benchmark meets the requirements. Other zookeeper performance tools should also be available for user convenience in Zetcd;zkboom, we might as well try it out to create a test for key:
Go get github.com/coreos/zetcd/cmd/zkboom
Zkboom--conns=50--total=10000--endpoints=localhost:2181 Create
The ZETCD should be able to provide adequate performance assurance for small loads. A delay benchmark for a simple two-node configuration indicates that ZETCD can withstand a large number of requests. Specifically configured for two Linux servers interconnected by a gigabit switch, one of the devices runs the agent and service side on the raid disk configuration, and another device is used to generate customer requests. Zkbook is measured by creating an empty key store and reading the key-value pairs from 128Kbytes. The user request is limited to 2,500 requests per second, and then gradually increases the number of concurrent clients. ZooKeeper 3.4.10 and ETCD results are shown in comparison.
The relationship between the ZETCD client concurrency and the average delay in creating a key is revealed. Because the ETCD has a 5-35ms advantage over zookeeper in latency, ZETCD has ample room to handle delays due to additional load and network jumps. There is always a gap of about 20ms compared to the ZOOKEEPER,ZETCD agent, but there is no queue congestion from the throughput data processing 2500 requests. A slow write to ZETCD is that, unlike reading, there are differences in the data model, so you need to write more than one key for each zookeeper key.
The relationship between the ZETCD client concurrency and the average latency of key values is revealed. Because the value of zookeeper latency is about 2ms faster than ETCD, to ZETCD to provide data faster than zookeeper, I am afraid to rely on the ETCD itself to further improve performance. However, although ZETCD needs to request additional keys from ETCD to emulate zookeeper Znode metadata, the ZETCD hit delay only adds about 1.5ms to the ETCD key extraction data. ZETCD requires only one round trip to the data extraction operation of key, because the read request is packaged into a ETCD transaction.
Future
Zetcd promises the results of the above performance benchmarks to be reasonable, and can easily support thousands of operations per second in acceptable latency scenarios. The above simulations provide an alternative to Mesos,kafka and drill alternative zookeeper. However, there is still a lot of room for improvement in performance for ZETCD itself. Testing more zookeeper applications at the same time will further push ZETCD into a replacement for zookeeper servers.
Zetcd was released from the open source community last October and has recently released its first version: Zetcd v0.0.1. Although it is the first beta release, it has provided stable management and deployment for future production systems. If used in conjunction with ETCD, the system running ZETCD will have a self-driven "ZooKeeper" cluster, which can automate background upgrades, backup and TLS management. For more information, please refer to https://github.com/coreos/zetcd/.
original link: zetcd:running ZooKeeper apps without ZooKeeper (translation: 杨峰 review: Wu Jiaching)
