Reprint: http://www.csdn.net/article/2015-01-22/2823659
Summary: in the "Walk the Cloud: CoreOS Practice Guide" series of the first few, respectively, how to set up CoreOS cluster, system services Butler SYSTEMD and Cluster command post fleet, This article will introduce the module ETCD that connects each node in CoreOS ecosystem and supports the cooperative operation of Cluster service.
Note: This article is starting in csdn, please indicate the source of the reprint.
"Editor 's note" As an operating system, CoreOS uses a highly streamlined system core and peripheral customization to implement many features that would otherwise require complex manual operations or third-party software support at the operating system level, while rejecting other non-core software for server systems. such as GUI and package manager. Csdn invited ThoughtWorks software engineer Linfan to bring the "Walk the Cloud: CoreOS Practice Guide" series of articles, with you to learn about the essence of CoreOS and recommended practice methods. This article is based on the fifth article: Distributed data storage Etcd. ETCD is a module in the CoreOS ecosystem that is at the core of connecting each node's communication and supporting the collaborative operation of Cluster services, and this article will mainly introduce the operation of ETCD and the use of APIs from the perspective of the system OPS workers.
Author Profile:
Linfan, an IT siege lion born in Gen Y, thoughtworks Chengdu Office Cloudops team member, usually likes to study devops-related applications in his spare time and is currently preparing for AWS Certification and promotion of Docker-related technologies.
The storage of distributed data has always been the core focus of solving the synchronization and coordination operation of Cluster service messages. In the previous section of this series, the fleet service for cluster management was introduced, while the implementation of Fleet's functionality depended on the dbus extension of the SYSTEMD service, and its distributed control was largely due to the reliable and efficient coreos of distributed data Services provided by ETCD. In fact, ETCD is a module in the CoreOS ecosystem that is at the core of connecting each node's communication and supporting the collaborative operation of Cluster services, and this article will mainly introduce the operation of ETCD and the use of APIs from the perspective of the system OPS worker.
What is distributed data storage? From a result point of view, it is to spread the data on multiple separate devices, so as to improve the reliability of data or read and write performance methods. From the perspective of implementation, the current mainstream NoSQL database, such as MongoDB, foundationdb and so on can be easily implemented distributed storage. While ETCD is inherently a bit like a NoSQL database system, it is more accurate to say that it is a highly available key-value storage System. Unlike the general NoSQL database, ETCD was designed primarily for shared configuration and service discovery, and it was inspired by zookeeper and Doozer. A comparison of the three and other similar service discovery frameworks can be found in this article, which simply means that the ETCD support for TTL and the HTTP Restful API are a great highlight. The language-and platform-agnostic restful APIs make it easier to develop two of times based on it, and can provide friendly support for programs that run in the application container, such as Docker.
In terms of data consistency, ETCD handles log replication through the raft consistency algorithm to ensure strong consistency. This is not intended to be an in-depth discussion of the raft algorithm, the specific principles of the raft algorithm can be see Guiyang this article.
The development of ETCD
Etcd was the oldest in CoreOS's core service module, and its first line of code was submitted in June 2013, earlier than fleet (October 2013) and other CoreOS modules, and it was the foundation of many other service modules.
It is worth mentioning that Etcd's main contributor is the two Chinese developer Qin Yicheng and Li rang. The original ETCD used a set of independent Goraft libraries, which began in mid-2014, etcd a massive code refactoring, removed reliance on the Goraft library, and re-implemented a more streamlined and stable raft algorithm. The code for the ETCDCTL command is also merged from a separate project into the ETCD repository. From GitHub's code submissions, ETCD has been the most active module in several of CoreOS's main modules.
By the time this article was released, CoreOS's latest blog has announced the Release candidate version of Etcd v2.0, the original v0.5 version (Release Candidate), which will be an important milestone in ETCD's move to production ready.
It should be noted that the v2.0 version contains a very important change, the ETCD for the client data Operations API, Port 4001 is modified to 2379, and port 7001 for communication between ETCD nodes is modified to 2380. Although the new version of the API calls and interfaces are still compatible with the content described in this series of articles, and the 4001/7001 port will continue to be able to be used for a foreseeable period of time, but please future you, in the use of ETCD It is important to note this distinction when v2.0 and later versions, replacing 4001/7001 ports in the series with 2379/2380 ports to avoid misleading. ETCD services can be mined more content, the ETCD API section will be placed in the next series of detailed details.
From the cluster node self-discovery speaking
From the ETCD development process, it is designed to be a configuration sharing service with zookeeper similar to the subscription/notification (subscript/public) feature. When the cluster starts, we only configure the URL address of a cluster identity so that each node knows each other, and can get the information of other nodes through fleet, there is etcd of the silent credit.
The CoreOS official Discovery Cluster Identification Service address is https://discovery.etcd.io. In the process of building the CoreOS cluster in the second part of the series, we get an identity address by visiting the/new page under this website, if you have forgotten this address, you can view it in any node in the cluster/run/systemd/system/etcd.service.d/ The 20-cloudinit.conf file, which records some information about when the node was started.
$ cat/run/systemd/system/etcd.service.d/20-cloudinit.conf [service] environment= "etcd_addr=172.17.8.101:4001" Environment= "etcd_discovery=https://discovery.etcd.io/09363c5fcdfcbd42ed60b8931263fda1" Environment= "ETCD_PEER_ addr=172.17.8.101:7001 "
Access this identity address directly to get the basic node information of the current cluster.
[email protected] ~ $ curl HTTPS://DISCOVERY.ETCD.IO/09363C5FCDFCBD42ED60B8931263FDA1 {"Action": "Get", "node": { ... }}
When the new CoreOS node starts, the ETCD service is first started through Cloud-init (which is also previously configured through the User-data file), and during the ETCD boot process, a list of information about the nodes already in the cluster is obtained through this identity address. Then add your own information to this list as well. Discovery services that provide this node information we will be in separate articles in another article.
After Etcd gets to the state of the cluster, it does a series of initialization work, one of which is to update the contents of the ETCD data about the cluster members. As a result, other services in the future can be directly updated with real-time cluster member information through ETCD data.
The ETCDCTL command allows you to view this piece of information to the ETCD service store. For example, the following command can list all nodes stored by ETCD.
[email protected] ~ $ etcdctl ls/_etcd/machines--recursive/_etcd/machines/f260afd8224c4854bdf8427d8451da23/_etc D/MACHINES/0ACDD9BF38194EA5AD1611FF9A4236F1/_etcd/machines/f2558aaa231044f3abbe01510ac2b1d8
View information for one of the specific nodes.
[email protected] ~ $ etcdctl Get/_etcd/machines/f260afd8224c4854bdf8427d8451da23 etcd=http%3a%2f%2f172.17.8.102 %3a4001&raft=http%3a%2f%2f172.17.8.102%3a7001
The ETCDCTL command used here is a command-line tool provided by ETCD for querying and manipulating its stored content, and the following is a look at its other uses.
Basic use of Etcdctl
ETCD itself provides a RESTful API based on HTTP, but in order to facilitate the daily use of operations, ETCDCTL implements many of the functions of this API, skilled use of it can simplify the workload of many operations.
- View the contents of a directory or key
The Etcdctl LS and etcdctl get commands that have been used in the previous section are the two basic commands most frequently used. ETCD's key values can be layered and nested, directories in the ETCD can hold multiple keys and other directories, and each specific directory and key has its own "access path", which is similar to common files and directories in file management. Etcdctl ls The function is to view a list of keys or directories under a specific path. For example, list the contents below the root directory:
[email protected] ~ $ etcdctl ls//coreos.com
All content under the specified directory and subdirectories can be listed once by using the--recursive parameter.
[email protected] ~ $ etcdctl ls/--recursive/coreos.com/coreos.com/updateengine/coreos.com/updateengine/rebootl Ock/coreos.com/updateengine/rebootlock/semaphore
The Etcdctl get command is used to obtain the value stored by the specified key, for example:
[email protected] ~ $ etcdctl Get/coreos.com/updateengine/rebootlock/semaphore {"semaphore": 1, "Max": 1, "holders": []}
Using the Etcdctl get command for a directory will give you a hint that tells the user that this is a directory.
[email protected] ~ $ etcdctl get/coreos.com/coreos.com:is a directory
You can specify the format of the output through the-o parameter of Etcdctl, such as the-O extended parameter to print more detailed information about the key.
[email protected] ~ $ Etcdctl-o Extended get/coreos.com/updateengine/rebootlock/semaphore Key:/coreos.com/updatee Ngine/rebootlock/semaphore created-index:6 modified-index:76156 ttl:0 etcd-index:104439 raft-index:413950 Raft-Term: {"Semaphore": 1, "Max": 1, "holders": []}
- Creating, modifying the contents of directories and keys
Create a new directory and key using the Etcdctl mkdir and Etcdctl MK commands respectively.
[email protected] ~ $ etcdctl mkdir/demo [email protected] ~ $ etcdctl get/demo/demo:is a directory [email prote CTED] ~ $ etcdctl Mk/demo/hello "Hello Etcd" # in reality this will echo output "Hello Etcd", omit [email protected] ~ $ etcdctl Get/demo/hello Hello Etcd
In practice, after each use of MK, the screen echoes the contents of the value written. Since we will again use get to remove this value, to verify that the content is actually written to the ETCD record, so the unified omit echo output, followed by the update and set example is also. Note In the example above, we created a new key and directory on the CORE-01 node and then checked the contents of the ETCD record on the core-02 node. This is done to show that the data obtained on each node is synchronized in real time in a cluster composed of ETCD services. Of course, the same results will be obtained if the records are checked on the core-01.
When using the Etcdctl MK command, if the path to the creation of the key does not exist, the corresponding directory structure is created automatically, for example:
[email protected] ~ $ etcdctl mk/path/to/the/new/key "Text Message" [email protected] ~ $ etcdctl get/path/to/the/ New/key Text Message
Attempting to create an existing key repeatedly will result in an error.
[email protected] ~ $ etcdctl Mk/demo/hello "Something Else" Error:105:key already exists (/demo/hello) [130187]
The record contents of the update key can be completed by the Etcdctl update (or etcdctl set) command.
[email protected] ~ $ etcdctl Update/demo/hello "Hello CoreOS" [email protected] ~ $ etcdctl Get/demo/hello Hello CoreOS
The difference between using Etcdctl update and Etcdctl set is that an error occurs when trying to update a nonexistent key.
[email protected] ~ $ etcdctl update/demo/xx "Message" Error:100:key not Found (/DEMO/XX) [129853]
The table below compares the similarities and differences between Etcdctl MK, Etcdctl Update, and Etcdctl set. It can be seen from the results that Etcdctl set is equivalent to merging the functions of the former two. In the actual situation, the corresponding command should be chosen according to the actual application scenario.
Command/action |
ETCDCTL MK |
ETCDCTL Update |
Etcdctl Set |
Target key does not exist |
Create this key and assign a value |
Error: Key not found |
Create this key and assign a value |
Target key already exists |
Error: Key already exists |
Update the contents of a key |
Update the contents of a key |
- Delete keys and directories
The commands to delete keys and directories are ETCDCTL RM and Etcdctl rmdir, respectively:
[email protected] ~ $ etcdctl Rm/demo/hello [email protected] ~ $ etcdctl Get/demo/hello Error:100:key not Foun D (/demo/hello) [131166] [email protected] ~ $ etcdctl rmdir/demo [email protected] ~ $ etcdctl Get/demo Error:100:ke Y not found (/demo) [131189]
Note that Etcdctl rmdir can only delete empty directories, which is similar to the Linux rmdir command. Attempting to delete a directory that has other keys or subdirectories produces an error. In this case, you can recursively delete all subdirectories and keys under the directory using the Etcdctl RM mate--recursive parameter.
[
Email protected] ~ $ etcdctl rmdir/path error:108:directory not Empty (/path) [131209] [email protected] ~ $ et Cdctl Rm/path--recursive
ETCD's key-value system has a feature that is useful for application configuration, which allows each key or directory to be given a live time limit (TTL, which is the hour to life). The TTL is in seconds, and when the specified number of seconds passes, the corresponding key or directory is automatically removed from the ETCD record if it has not been updated.
The parameter used to add a TTL to a key or directory is--ttl, which applies to several created and updated commands.
[email protected] ~ $ etcdctl Mkdir/path/to/demo--ttl 120 # Add TTL time to directory [email protected] ~ $ etcdctl mk/path/to/ Demo/title "Message title"--ttl 120 # Add TTL time to key
The--ttl parameter of update and UPDATEDIR enables you to reset the remaining lifetime of the key and directory to the specified new value. This function is somewhat like a "watchdog" program for ensuring that the instrument is running correctly, and once the program has not updated the corresponding ETCD record for a certain period of time, the record is considered to be out of date and is removed directly. After you set the TTL, you can use the Ectdctl get command with the-O extended parameter to check the surviving time of the data key or directory remaining.
[email protected] ~ $ etcdctl updatedir/path/to/demo--ttl 500 # Update the TTL time of the catalog [email protected] ~ $ etcdctl Update/ Path/to/demo/titl "Message Title" E--ttl 400 # Update the TTL time of the key
This function makes the Local System node or service failure, the rest of the system can find this situation in time, and make adjustments. Specific application scenarios We will give an example in the next step of the content.
Monitor changes
If ETCD's functionality is limited to storing and distributing data, it has no particular difference from the normal NoSQL database. In fact, ETCD does much more than that, as a service for cluster configuration sharing, in addition to the typical TTL feature, another important feature is the subscription/notification of data changes (subscript/public).
In order to maintain the correct behavior, applications in the cluster need to be adjusted according to the required configuration information at all times, usually in two ways. One is to periodically check the contents of the cluster configuration, that is, timed polling (Polling). The other is to subscribe to specific events that are directly notified to the application for processing by the cluster Configuration service (ETCD) when the corresponding event occurs. Clearly the latter is more applicable from the timeliness of response and the impact on application efficiency.
In fact, ETCD itself does not provide a set of direct subscription/notification service mechanism, but through it provides monitoring Change API and HTTP long-polling approach, can achieve the same functionality. The associated command is Etcdctl Watch and Etcdctl Exec-watch, which is used to wait for the specified key to change, which on the basis of the former provides the ability to automatically trigger another user-specified command after the change occurs.
[email protected] ~ $ etcdctl watch/path/to/demo [email protected] ~ $ etcdctl Update/path/to/demo "New-value"
The above example uses the CORE-01 node listening path/path/to/demo, and when Etcdctl watch executes, the program begins to enter the wait state. The keys on this path are then updated on the CORE-02 node, and the ETCDCTL process waiting on the CORE-01 node receives the change and exits.
This command can receive a parameter--recursive for recursive listener changes to all sub-paths under the specified path.
[email protected] ~ $ etcdctl Watch--recursive/path
Using the Etcdctl Watch command alone doesn't make much sense, because it quits every time it listens to a specified change and does nothing. Commands that are required to execute after a record change are generally written at the back of this line of command, so that when the event occurs, a specific action is taken immediately. In some background scripts, this kind of monitoring is useful. The following script implements the automatic listener/path/to/demo path change, and once a change occurs, the value of this key is preceded by a "Hello".
Key=/path/to/demo while True, do etcdctl watch ${key} sleep 1s etcdctl update ${key} "Hello $ (Etcdctl get ${key} ) "
Done
Be aware of the "sleep 1s" line in this script, if you remove this line, use Etcdctl get immediately later to get the contents of the key before the change occurs. That is, when the ETCDCTL receives the change signal, it needs to wait a little while to get the changed content. This issue has already been submitted to GitHub, and interested students can follow up with the following reply.
In addition to putting the commands that need to be executed into the Etcdctl Watch command, ETCDCTL also provides a command to one step: Exec-watch. However, there are many problems in the actual use of this command, when monitoring the same key, the content of the key is modified, the use of the Watch command can always return immediately, but the Exec-watch command is often not received for no reason In addition, there is an issue where the command does not exit correctly after the specified command has been executed Exec-watch. Given that the work it has done using etcdctl Watch has been able to do, there is a temporary omission to introduce this command.
into the background
In this article, we mainly from the role of the ETCD service and its data manipulation tools Etcdctl introduced the CoreOS core Distributed Data Service overview and usage. Behind these tools, however, is a set of API systems that support the expansion of the ETCD business. In the next article in the series, you will continue to explore the RESTful API usage of the ETCD service, ETCD configuration parameters, and other ETCD features that some users need to know about. Please look forward to it. (Author/linfan Zebian/Zhou Xiaolu)
[CoreOS Reprint] CoreOS Practice Guide (v): Distributed data storage ETCD (top)