This is a creation in Article, where the information may have evolved or changed.
The ETCD team has just been excited to announce the newest member--etcd 3.2.0 in the Etcd 3.x series. New features and features such as agent improvements, enhanced backend concurrency, distributed coordination services, lighter go clients, and JWT authentication are emerging in this new version.
In today's article, we'll show you a few of the new features in Etcd 3.2.0, as well as a discussion based on this. The first thing to mention is the multi-tenancy mechanism: The new namespace client and the agent that can provide a fully isolated key space. After that, we're going to talk about scaling--GRPC agents can now support millions of events per second. Finally, in order to better coordinate the various systems, this new version introduces a new concurrent RPC service with built-in distributed locks and corresponding options that can be easily accessed by any GRPC customer.
Name space
For applications that share the same set of ETCD clusters, in order to avoid mutual interference with each other, they need to maintain their own keys only. Although the ETCD authentication mechanism can use permissions to protect individual users ' keys, it is not possible to prevent potential collisions between keys due to the use of the same name. Instead, the application typically takes a prefix parameter and then adds it to the key names of all applications to implement a valid name for the key. It should be emphasized, however, that the correct implementation of this prefix is often tedious and error-prone, so ETCD now delivers both the customer namespace and the proxy namespace as ready-made reusable components to the user.
Each namespace is responsible for organizing the ETCD key into its own completely separate key space. A namespace belongs to a set of views with a full key and a prefix. The key name within a namespace contains the prefix when viewed by the underlying ETCD key space, but does not include a prefix when viewed through a namespace. Similarly, any ETCD client that accesses the ETCD proxy in the namespace (optionally masquerading the proxy as a standalone cluster to avoid exposing the core endpoint) can access only the individual keys under the proxy namespace prefix. For example, in the following Etcd topology diagram, a client accesses ETCD through the proxy "/app-a/", which translates the request for the key "ABC" to "/app-a/abc". Similarly, "App-b" can only access its "ABC" Through "/app-b/abc", where the namespace "/app-a/" and "/app-b/" are isolated from each other.
example diagram of ETCD topology with two sets of namespace proxies
Namespaces are also very simple to configure. We can test the topology in the example above using the following command:
# Start ETCD
$ ETCD &
Launch namespace proxies for/app-a/and/app-b/
$ etcd grpc-proxy start--endpoints=http://localhost:2379 \
--listen-addr=127.0.0.1:23790 \
--namespace=/app-a/&
$ etcd grpc-proxy start--endpoints=http://localhost:2379 \
--listen-addr=127.0.0.1:23791 \
--namespace=/app-b/&
# Write to/app-a/and/app-b/
$ etcdctl_api=3 Etcdctl--endpoints=http://localhost:23790 put ABC a
$ etcdctl_api=3 etcdctl--endpoints=http://localhost:23791 put ABC Z
Confirm the different keys were written
$ etcdctl_api=3 etcdctl--endpoints=http://locahost:2379--prefix/app
Million events per second
Any change to the key will trigger ETCD to broadcast the event to all monitors that are concerned about it. For a set of large-scale ETCD deployment systems, which often contain thousands of concurrent monitoring programs. Unfortunately, the existence of these monitoring procedures also brings corresponding costs.
An overly large number of surveillance programs can often cause ETCD overloading. As shown in the table is a set of small ETCD instances overloaded when a single shared key is continuously updated for a large number of monitoring programs. The ETCD server here is a weaker n1-standard-1 (1vCPU + 3.75 GB memory) machine, while the client uses a more powerful n1-standard-32 (32vCPU + GB memory) machine, The latter is able to achieve 500 writes per second and full server resources through the monitoring program. This means that continuing to add the monitor will eventually result in a drop in the write rate, and the event rate will be lower than the desired level.
overload of the ETCD server that is raised when there are too many monitoring programs pointing to a single key
The ETCD GRPC agent is able to replay events from one server monitor to multiple other client monitors. Each agent is able to merge the related input client monitors into a single ETCD server monitor. Specifically, the agent can provide consolidated monitoring events for multiple clients. These clients will share the same server monitor, while agents will effectively help the core focus away from resource-intensive pressures.
By adding proxies, ETCD can implement up to million events per second, as shown in. Using the same test platform as the previous example, we attached 100 new monitors to each agent that was added to the cluster. Each agent runs its own n1-standard-1 instance as the ETCD instance. When the number of agents reaches 20, the activity intensity of the monitor increases linearly to 1000 events per second, while the ETCD cluster writes less than 500 times per second-that is, there is no overloading phenomenon.
increase event hold throughput by increasing watchdog agent
You can use the ETCD benchmark tool to perform similar experiments locally. The following are the results of the delay changes resulting from testing 100 distributed connections with 3 agents:
$ ETCD & $ etcd grpc-proxy start--endpoints=http://localhost:2379--listen-addr=127.0.0.1:23790 & $ ETCD Grpc-pro XY start--endpoints=http://localhost:2379--listen-addr=127.0.0.1:23791 & $ etcd grpc-proxy start--endpoints=http ://localhost:2379--listen-addr=127.0.0.1:23792 & $ benchmark watch-latency \--clients=100--conns=100--ENDP oints=http://localhost:23790,http://localhost:23791,http://localhost:23792
Distributed Coordination Services
ETCD one of the advantages of this kind of consistent distributed key-value storage scheme is its ability to coordinate and synchronize the distributed systems. This general principle of coordination typically includes distributed shared locks and leader node elections. In ETCD 3.2, distributed shared locks and elections are exported as RPC services, which not only greatly simplifies the distributed coordination process, but also improves performance in high latency environments.
The early development of the ETCD V3 API involves writing a distributed approach for "preliminary inspection". Based on relevant experience, effective coordination algorithms are actually very rare; we can't expect every ETCD language binding to have a good locking protocol. In general, third-party ETCD bindings with distributed locks can only support a single simple custom lock, which is responsible for automatic waiting (that is, using call hibernation to limit the number of requests), thus avoiding the contention of large requests for the transmission path. And because Etcd's GRPC protocol can deliver excellent portability, we naturally have a reason to make a more effective "lock-as-a-service" attempt.
The biggest advantage of using server-side lock is to avoid the problems caused by unstable network conditions. For a client lock, it must open a watchdog and wait for a response to see if the lock is already occupied by another process. On the other hand, lock RPC only needs one transfer round trip, so the impact of network latency will be effectively controlled; If the lock pointed to is already occupied, ETCD allocates the monitor internally and returns it after the RPC acquires the lock. The specific impact of network latency on client locks and RPC locks is shown. The average latency of RPC locks is lower, and the performance of the two is gradually converging with the decrease of network transfer time.
the impact of contention and delay level increases on lock latency (the lower the better).
Client-side coordination code for each RPC multiplexing ETCD for cross-compatibility. To connect the ETCD server to itself as a client, each service can use a new set of embedded clients to map the ETCD server internally to a set of ETCD clients. For servers that are responsible for declaring client sessions, the new version also brings a new session recovery feature for building sessions based on the original lease. Through this code reuse, the server-side RPC lock and the election mechanism can be behaved in accordance with the ETCD client lock and the client election mechanism.
In addition, the ETCDCTL command-line tool can also help users quickly use ETCD locks. The following simple example will continue to increment the file F in the shell, while the locking mechanism ensures that all deltas are actually superimposed:
$ echo 0 >f
$ for ' seq 1 100 '; Do
Etcdctl_api=3 etcdctl Lock Mylock--bash-c ' expr 1 + $ (cat f) > F ' &
pids= "$pids $!"
Done
$ wait $pids
$ cat F
The lock service endpoint receives the JSON request through the ETCD GRPC gateway. The following example shows how to use JSON to get a set of locks and wait for the actual release of the lock:
$ lid=$ (etcdctl_api=3 etcdctl-w fields lease Grant 5 | grep ' "ID" ' | awk ' {print $} ')
Acquire lock by name "Mylock" (base64 encoded) using JSON
$ curl localhost:2379/v3alpha/lock/lock-xpost-d "{\" name\ ": \" bxlsb2nr\ ", \" lease\ ": $lid}" >/dev/null
$ date
Lease expires in five seconds, unlocking "mylock" and letting Etcdctl acquire the lock
$ etcdctl_api=3 etcdctl Lock Mylock Date
Learn More
Interested friends can click here to learn the latest and most important development results of the ETCD project. The project also hosts the 3.2.0 version of the signed binary library and the various historical versions on the ETCD publishing page. This GitHub release also contains the latest documentation related to ETCD cluster operations and ETCD application development.
As always, the ETCD team is committed to creating the best distributed consistency key-value storage solution, and if you find any bugs, questions or suggestions, please click here to visit the ETCD Question tracking page.
The release of the ETCD version will also be included in the future version of tectonic. If you are interested in learning about other distributed computing solutions based on ETCD, Kubernetes and CoreOS, we also invite you to click here for a free tectonic experience.
original link: Etcd 3.2 Now with massive watch scaling and easy locks (translation: Tri Juenwen)
