Distributed System Coordination using C # and consul

With the advent of the Big Data era, distributed is a major solution to big data problems, and with more and more distributed services, how to coordinate these services in a distributed system becomes a tricky problem. Today we'll look at how to use C # to coordinate distributed services with open source.

In the coordination of distributed applications, the main encounter the following scenarios:

• Business Discovery (Service discovery)

Find the available services and nodes in the distributed system

• Name Service

Know the corresponding resource by the given name

• Configuration management (config management)

How to share a configuration file in a distributed node to ensure consistency.

• Fault discovery and failover (failure detection and failover)

When a node fails, how to detect and notify other nodes, or transfer the services you want to use to other available nodes

• Leadership election (leader election)

How to elect a leader in a multitude of nodes to coordinate all nodes

• Distributed locks (distributed exclusive lock)

How to synchronize in a distributed service with locks

• Messaging and Notification Services (message queue and notification)

How to deliver messages in a distributed service that proactively responds to events in the form of notifications

Consul

Consul is a distributed service coordination management tool developed with Go, which provides features such as service discovery, health check, Key/value storage, and supports cross-datacenter functionality. Consul HTTP-based APIs can be easily bound to a variety of languages, and C # language binding Https://github.com/PlayFab/consuldotnet

Consul runs an agent on each node of the cluster, which can use either server or client mode. The client is responsible for efficient communication to the server, relative to stateless. The server is responsible for electing the leader node, maintaining the status of the cluster, responding to all queries, communicating across data centers, and so on.

Consul website already has the compiled binary package, supports various platforms: win, Linux and so on, downloads the package which conforms to your platform: here, downloads the package: 0.5.2_windows_386.zip. Only one consul file is finished after decompression.

D:\github\consuldotnet\consul.test>consul

Usage:consul [--version] [--help] <command> [<args>]

Available Commands is:

Agent Runs a Consul agent

Configtest Validate config file

Event fire a new event

EXEC executes a command on Consul nodes

Force-leave forces a member of the cluster to enter the ' left ' state

Info provides debugging information for operators

Join tell Consul agent to join cluster

Keygen generates a new encryption key

Keyring manages gossip layer encryption keys

Leave Gracefully leaves the Consul cluster and shuts down

Lock Execute a command holding a lock

Maint Controls node or service maintenance mode

Members Lists the members of a Consul cluster

Monitor Stream logs from a Consul agent

Reload Triggers the agent to reload configuration files

Version Prints the Consul version

Watch watch for changes in Consul

Consul after installation, the agent can be started, the agent can run in server or client mode, each data center has at least one agent running in server mode, the general recommendation is 3 or 5 server. Deploying a single server is very bad because data loss in a failed scenario is unavoidable. This article covers the creation of a new datacenter, all other agents are running in client mode, which is a very lightweight service registration process that runs health monitoring and forwards query results to the service. The agent must be running on each node in the cluster.

Consul.exe Agent-config-file Test_config.json

Let's run an agent in server mode first:

d:\github\consuldotnet\consul.test> Consul.exe Agent-config-file Test_config.json

==> warning:bootstrap Mode enabled! Do not enable unless necessary

==> Warning:windows is not recommended as a Consul server. Do not use the in production.

==> Warning:it is highly recommended to set gomaxprocs higher than 1

==> starting Consul Agent ...

==> starting Consul Agent RPC ...

==> Consul Agent running!

Node name: ' GEFFZHANG-NB '

Datacenter: ' DC1 '

Server:true (Bootstrap:true)

Client addr:127.0.0.1 (http:8500, HTTPS:-1, dns:8600, rpc:8400)

Cluster addr:192.168.1.4 (lan:8301, wan:8302)

Gossip Encrypt:false, Rpc-tls:false, Tls-incoming:false

Atlas: <disabled>

==> Log data would now stream in as it occurs:

2015/08/09 09:14:48 [INFO] SERF:EVENTMEMBERJOIN:GEFFZHANG-NB 192.168.1.4

2015/08/09 09:14:48 [INFO] SERF:EVENTMEMBERJOIN:GEFFZHANG-NB.DC1 192.168.1.4

2015/08/09 09:14:48 [INFO] raft:node at 192.168.1.4:8300 [Follower] entering Follower state

2015/08/09 09:14:48 [INFO] consul:adding server GEFFZHANG-NB (addr:192.168.1.4:8300) (DC:DC1)

2015/08/09 09:14:48 [INFO] consul:adding server GEFFZHANG-NB.DC1 (addr:192.168.1.4:8300) (DC:DC1)

2015/08/09 09:14:48 [ERR] agent:failed to sync remote state:no cluster leader

2015/08/09 09:14:50 [WARN] raft:heartbeat timeout reached, starting election

2015/08/09 09:14:50 [INFO] raft:node at 192.168.1.4:8300 [candidate] entering candidate state

2015/08/09 09:14:50 [DEBUG] raft:votes needed:1

2015/08/09 09:14:50 [DEBUG] raft:vote granted. Tally:1

2015/08/09 09:14:50 [INFO] Raft:election won. Tally:1

2015/08/09 09:14:50 [INFO] raft:node at 192.168.1.4:8300 [Leader] entering Leader state

2015/08/09 09:14:50 [INFO] consul:cluster leadership acquired

2015/08/09 09:14:50 [INFO] consul:new leader ELECTED:GEFFZHANG-NB

2015/08/09 09:14:50 [INFO] raft:disabling Enablesinglenode (Bootstrap)

2015/08/09 09:14:50 [DEBUG] Raft:node 192.168.1.4:8300 updated peer set (2): [192.168.1.4:8300]

2015/08/09 09:14:50 [DEBUG] consul:reset tombstone GC to index 2

2015/08/09 09:14:50 [INFO] consul:member ' GEFFZHANG-NB ' joined, marking health alive

2015/08/09 09:14:51 [INFO] agent:synced service ' consul '

2015/08/09 09:16:03 [DEBUG] Agent:service ' Consul ' in sync

2015/08/09 09:17:30 [DEBUG] Agent:service ' Consul ' in sync

2015/08/09 09:18:38 [DEBUG] Agent:service ' Consul ' in sync

2015/08/09 09:19:47 [DEBUG] http:request/v1/status/peers (0)

2015/08/09 09:19:52 [DEBUG] Agent:service ' Consul ' in sync

As you can see, the consul agent has been started, and some logs have been printed to the terminal, and it is seen from the log that our agent is already running in server mode and is already the leader node of the entire cluster. Also, local members have been marked as healthy members of the cluster. This is when you run Consul members in another terminal to see the entire cluster. At this point you can only see yourself, because our cluster has not yet joined the other members.

The output already shows your own node information, has address information, health status, roles in the cluster, and some version information, and if you want to see some metadata, you can add the-detailed tag

The information produced by the Consul members command is based on the gossip protocol and is ultimately consistent.

KV Basic Operation

Consul provides a simple k/v storage system that can be used to dynamically acquire configurations, perform service coordination, master node elections, build processes that other developers can think of, and so on.

var client = new Client ();

var kv = client. KV;

var key = Generatetestkeyname ();

var value = Encoding.UTF8.GetBytes ("test");

var getrequest = kv. Get (key);

Assert.isnull (Getrequest.response);

var pair = new Kvpair (key)

{

Flags = 42,

Value = value

};

var putrequest = kv. Put (pair);

Assert.istrue (Putrequest.response);

Getrequest = kv. Get (key);

var res = Getrequest.response;

Assert.isnotnull (RES);

Assert.istrue (StructuralComparisons.StructuralEqualityComparer.Equals (value, Res. Value));

Assert.AreEqual (pair. Flags, Res. Flags);

Assert.istrue (Getrequest.lastindex > 0);

var del = kv. Delete (key);

Assert.istrue (Del. Response);

Getrequest = kv. Get (key);

Assert.isnull (Getrequest.response);

Services Discovery (Service Discovery) and health check

Another major feature of consul is the ability to manage distributed services, where users can register for a service and also provide health detection capabilities for the service.

Service definition: A service can dynamically add, delete, and modify services by providing a service definition configuration file or by invoking the HTTP API.

Service query: Once the agent is started and the service is synchronized, we can use the DNS or HTTP API to query.

Service upgrade: The upgrade of a service definition can be done by modifying the service definition configuration file and then sending an SIGHUP signal to the agent, allowing you to upgrade the service without generating agent downtime or service unavailability. or through the HTTP API interface to dynamically add, delete, modify the service.

The consul supports three types of check modes:

• Call an external script, in which the consul timer invokes an external script that obtains the health status of the corresponding service through the return content of the script.
• Call HTTP, in which the Consul timer invokes an HTTP request, returns 2XX, and is healthy; 429 (Too many request) is a warning. All others are unhealthy
• Proactively escalate, in this mode, the service needs to actively invoke an HTTP put request provided by a consul to escalate the health status.

C # API provides the corresponding interface

• Client. Agent.service
• Client. Agent.check

Consul's health check, which checks the status of the service by invoking scripts, HTTP, or unsolicited escalation, is more flexible and can get more information, but more work is needed.

Fault detection (Failure Detection)

Consul provides the session concept, and the session can be used to check if the service is alive. For each of the services we can create a session object, note that here we set the Ttl,consul will be the value of the TTL is the interval time, continuous check the survival of the session. corresponding to the service, we need to continue the renew session, to ensure that the session is legal.

var client = new Client ();

var sessionrequest = client. Session.create (New Sessionentry () {TTL = Timespan.fromseconds (10)});

var id = sessionrequest.response;

Assert.istrue (sessionRequest.RequestTime.TotalMilliseconds > 0);

Assert.isfalse (String. IsNullOrEmpty (Sessionrequest.response));

var tokensource = new CancellationTokenSource ();

var ct = Tokensource.token;

Client. Session.renewperiodic (Timespan.fromseconds (1), ID, writeoptions.empty, CT);

Tokensource.cancelafter (3000);

Task.delay (CTS, CT). Wait (CT);

var inforequest = client. Session.info (ID);

Assert.istrue (Inforequest.lastindex > 0);

Assert.isnotnull (Inforequest.knownleader);

Assert.AreEqual (ID, InfoRequest.Response.ID);

Assert.istrue (client. Session.destroy (ID). Response);

Note here that, because it is based on TTL (minimum 10 seconds) detection, from business interruption to detected, at least 10 seconds of latency, corresponding to the need for real-time response scenarios, does not apply.

Leadership of elections and distributed locks

This document describes how to use consul KV storage to achieve leader election, using the KV function of consul, can easily achieve the function of leading the election and lock.

WEB UI

Consul also supports the Web interface, which can be used to view all the services and nodes, all of the health detections and their current state, reading the values of the set k/v system. The UI automatically supports multiple datacenter by default. These UIs are static HTML you don't need to run a Web server alone, and the consul agent itself can configure a Web service.

Download UI components: WEB UI

After the download is completed is a 0.5.2_web_ui.zip compressed file, after decompression is a dist directory. Then add the-ui-dir parameter and the-client parameter to restart the agent.

d:\github\consuldotnet\consul.test> Consul.exe Agent-config-file test_config.json-ui-dir=d:\github\ Consuldotnet\consul.test\0.5.2_web_ui\dist

You can access the UI by entering http://127.0.0.1:8500 in the browser

Have services, nodes, k/v, ACLs, Datacenter Management, very perfect a system.

Summarize

Consul as a distributed orchestration using the Go language, it provides good support for business Discovery Management, and his HTTP API is also well-bound to different languages and supports applications across data centers. The disadvantage is relatively new and suitable for users who like to try new things.

Distributed System Coordination using C # and consul