ETCD of service Discovery VS Consul

Copy it from here.

************************************************************************************************

Online to find are both ZK and ETCD comparison, and consul relatively few, this feeling is reliable, also seen in other places on the consul of The Spit Groove, recorded

************************************************************************************************

Lead

In a distributed micro-service architecture, an application may consist of a set of services that single a group of responsibilities. A mechanism for registering a service is required to register a service or a node is available, and a discovery service mechanism is needed to find which services or which nodes are still serving.

In practical applications, it is often necessary to have a configuration file that tells us some configuration information, such as the address of the data connection, the address of the Redis, and so on. But most of the time, we want to get this information dynamically without modifying the code, and to manage them well.

With these requirements, a series of open source frameworks have been developed, such as ZooKeeper, ETCD, Consul and so on.

These frameworks generally provide such services as:

Service Registration
Host name, port number, version number, or some environment information.

Service discovery
You can get the client to the server address.

A distributed general-purpose K/V Storage System
Based on Paxos algorithm or Raft algorithm

Leader election (Leader election)

Some other examples:
Distributed lock (distributed locking)
Atom Broadcast (Atomic broadcast)
Serial number (Sequence numbers)
...

We all know that CAP is Eric Brewer's three elements of a distributed system:
Strong consistency (consistency)
Availability (availability)
Partition tolerance (Partition tolerance)

Almost all service discovery and registration programs are CP systems, that is, to meet the same data have multiple copies of the situation, for the data update operation to achieve the final effect and operation of a data is the same, but in the presence of network partitions (nodes between partitions can not network communication) such a failure, Some nodes will not be able to provide services (some systems support stale reads in case of node loss) before resuming communication between nodes and synchronizing data.

ZooKeeper, one of the oldest in such frameworks, is written by Java. Java is great in many ways, but it is still a bit heavy for this type of work, and because of its complexity and yearning for new things, we give up choosing it the first time.

This paper compares ETCD and Consul from protocol and application layer, and finally gives the author's preference.

Etcd

ETCD is a distributed k/v storage system written using the go language. Given that it is a CoreOS company project and the number of star on GitHub (9000+), Google's famous container management tool Kuberbetes is based on ETCD. The first thing we think about is it.

Before we introduce ETCD, we need to understand some basic concepts. We know that it is easy to maintain a state in a single server process, and there is no conflict when reading and writing. Even in multi-process or multi-threaded environments, the use of lock mechanisms or co-coroutine can also allow read-write to proceed in an orderly manner. However, in a distributed system, the situation is much more complex, the server may crash, the machine between the nodes may not network, and so on. So the consistency protocol is used to maintain a consistent state across multiple machines in a cluster.

Many distributed systems will adopt the PAXOS protocol, but the Paxos protocol is difficult to understand, and it is very different in actual implementation. So ETCD chose Raft as its consistency protocol. Raft is Diego Ongaro and John ousterhout in the ' in Search for an understandable Consensus algorithm '. It makes great optimizations for Paxos and is much easier to understand than Paxos when it sacrifices little usability and achieves similar functionality.

It focuses on solving two problems:

Leader election (Leader election)
Raft first elected a Leader through the leadership of the election, the subsequent consistency of maintenance is done by the Leader, which simplifies the problem of consistency. Raft will guarantee that there will only be one Leader at a time and will be selected as Leader if more than half of the nodes are voted. When the Leader hangs, the new Leader will be chosen.

Log copy (Replication)
In order to maintain the status, the system will log all operational command logs. Leader is appended to the end of the log after it receives the client action command. The Leader then sends the Appendentries RPC request to all the other nodes in the cluster, each of which replicates the command through a two-phase commit, which guarantees that most of the nodes can be completed.

In the actual application, the general ETCD cluster to 5 or 7 is appropriate, can endure 2 or 3 nodes hang off, why not the more the better? is due to performance considerations, because the replication of logs can lead to bottlenecks in both the CPU and the network because of the number of nodes.

ETCD API is relatively simple, can be a directory or a key to Get,put,delete operation, is based on HTTP. ETCD provides the ability to watch a directory or a key, maintaining a long connection between the client and the ETCD cluster (long polling). Based on this long connection, once the data has changed, the client is immediately notified, and the returned result is a changed value and the value before the change, which is useful in practical applications (this is also one of the Consul slots in the back).

ETCD Watch and under normal circumstances will not miss any changes. Because ETCD not only stores the current key-value pairs, it also stores the most recent change records, so if a watch that is behind the current state can still get all the updates by traversing the history change record. ETCD also supports Compareandswap this atomic operation, first a value comparison of a key, only the result is consistent before the next assignment operation. Using this feature is like implementing a lock with x86 CAS to implement a distributed lock.

In Etcd, there is a concept of a proxy, it is actually a forwarding server, the start of the need to specify the address of the cluster, and then you can forward the client's request to the cluster, which itself does not store data. In general, each service node will start a proxy, so this proxy is also a local proxy, so that the service node does not need to know the specific address of the ETCD cluster, only need to request a local proxy. As mentioned before, a k/v system should also support leader Election,etcd can be implemented via TTL (time to live) key.

Consul

Consul and ETCD also have two kinds of nodes, one called client (and ETCD proxy) only responsible for forwarding requests, and the other is the server, which is the node that really stores and processes transactions.

Consul uses a gossip protocol based on the serf implementation to manage dependencies, failure detection, event broadcasts, and so on. The gossip protocol is a magical consistency protocol that is named gossip because the protocol simulates the spread of rumors in humans. To spread rumors there is a seed node, and the seed node randomly sends its own list of nodes to other nodes every second, as well as messages that need to be propagated. Any newly added node will soon be known to the whole network in this mode of communication. The magic of this protocol is that it does not want information from the beginning of the design to be passed to all nodes, but over time, the whole network will get the same information at some point in the end. Of course, this moment may only exist in theory and never be reached.

Consul uses two different gossip pool, called LAN and WAN, because Consul native supports multiple data centers. Inside a data center, the LAN gossip pool contains all the nodes in the data center, including proxies and servers. The WAN pool is globally unique, and all data center servers are in this pool. The difference between the two pool is that LAN handles failure detection within the data center, event broadcasts, and so on, while the WAN is concerned with cross-datacenter. In addition to the gossip protocol, Consul also uses the Raft protocol for leader election, and the process of copying logs after leader is basically consistent with ETCD.

Back to the application level, Consul is more like a full stack solution that not only provides consistent k/v storage, but also encapsulates service discovery, health checks, built-in DNS server, and more. It looks very nice and can be used out of the box. As a result, we initially selected Consul as the framework for our service discovery and dynamic configuration. But the reality is often cold, after deep study of its API found a lot of pits, perhaps the designer has his own consideration.

When you use the Get all services API, only the name and tag of the service are returned, and if you want specific information for each service, you need to go to the request. This is very inconvenient on the client side, because in my opinion, getting a list of all the services and specific information should be a very common requirement, and the less the number of requests the better.

If the watch service is changed and the value changes, the returned result is the equivalent of re-reading all services, which does not reflect the change in service information, which makes it difficult for the client to update the operation.

Health check is a built-in feature of Consul, and it sounds good to add a custom health check when registering a service. But if you forget to give your health check to a service, it becomes unpredictable in the results of the various API returns.
...

Conclusion

After a few days of tossing, we finally decided to return to ETCD, which proved to be the right decision. We achieve a stable service discovery and dynamic configuration based on ETCD, but there are some more details that are not elaborated in this article, welcome to explore together.

ETCD of service Discovery VS Consul