Elasticsearch internals:networking Introduction An overview of the Network topology

This article introduces, the networking part of Elasticsearch. We look at the network topology of an Elasticsearch cluster, which connections is established between which nodes and how The different Java clients works. Finally, we look a bit closer on the communication channels between the nodes.

Node topology

Elasticsearch nodes in a cluster form what's known as a full mesh topology, which means this each Elasticsearch node main Tains a connection to all of the other nodes.

Full mesh topology with 6 cluster nodes

In order to simplify the code base, the connections is used as one-way connections, so the connection topology actually E NDS up looking like this:

6 cluster nodes with connections, each line represents a connectionadding a Node

When a node starts up, it reads a list of seed nodes from its configuration using unicast, and sends Multicastmessages loo King for other nodes in the same cluster.

As the node discovers more instances on the same cluster, it connects to them one by one, asks them for information about All the nodes they know and then attempts to connect to them all and officially join the cluster. In this ", previously running instances assist in quickly getting fresh instances Cluster.

Node clients

Even "Client" Elasticsearch nodes (i.e nodes configured with node.client: true or build with NodeBuilder.client(true) ) connect to the cluster this.

This implies, the client node becomes a full participant on the full mesh network. In other words, once it starts joining the cluster, all the existing nodes in the cluster would connect back to the Instanc E. This means, both the client and server firewalls and publish hosts must be properly configured in order to allow th Is. Additionally, whenever a node client joins, leaves or experiences networking issues, it causes extra work for all the othe R nodes in the cluster, such as opening/closing network connections and updating the cluster state with the information About the node.

Node client connecting to a Clustertransport clients

On the other hand, "transport" clients work differently.

When a Transport client connects to one or more instances in a cluster, the instances does not connect back, nor is the Exis Tence of the transport client part of the cluster state. This means, a joining/leaving transport client causes minimum extra work for the other nodes in the cluster.

Transport client connecting to a clusterconnections and Channels

What does we mean when we talk about the-to- connection a Node in Elasticsearch? In the sections above, we refrained from being specific on it and only used the term as a logical connection s communication between and nodes. Let's go into more detail.

Usually, when we talk about connections in the context of networks, we refer to tcp-connections, which provide a reliable Line of communication between, nodes.

Elasticsearch uses (by default) TCP for communication between nodes, but in order to prevent important traffic such as FAU Lt-detection and cluster state changes from being affected by far less important, slower moving traffic such as Query/inde Xing requests, it creates multiple TCP connections between each node. For each of these connections, Elasticsearch uses the term channel , which encapsulates the serialization/deserialization O F messages over a particular connection.

In earlier Elasticsearch versions there used to be three different classes of channels: low , med and high . After a while, is ping added, and a recent change renamed these channels such that they is more descriptive about their Intention. At the time of writing, the following channel classes exist:

• recovery: 2 channels for the recovery of indexes
• bulk: 3 channels for bulk based operations such as bulk indexing. Previously called low .
• reg: 6 channels for medium priority regular operations such as queries. Previous called med .
• state: 1 channel dedicated to state based operations such as changes to the cluster state. Previously called high .
• ping: 1 channel dedicated to pings between the instances for fault detection.

The default number of channels in each of the these class is configured with the configuration prefix of transport.connections_per_node .

All channels in a three-node cluster. Each line represents a single channel

Elasticsearch have support for TCP keepalive which are not explicitly set by default. This prevents unused or idle channels from being closed, which would otherwise cascade to a complete Disconnect-reconnec T cycle as explained above.

Adding an Instance

A consequence of the above is then adding a new instance to an existing cluster causes it to establish from connections to E Ach node, and each node establishes-connections back to the new instance.

Linked Channels

If one of the channels to a particular node was closed due to intermittent network errors for example, all the other Cha Nnels to the same node was closed and the connections to the node is reconnected. This is do in order to maintain some internal invariants and to ensure the internal consistency of the communication bet Ween the nodes.

Ending Remarks

The fact, all the channels between, nodes in a cluster is linked makes it extra vulnerable to network issues, and This was one of the reasons why people be discouraged from trying to create a cluster between data centers that was far AP Art (and thus adding more sources of failure).

In this article we ' ve shown the basic network topology of an Elasticsearch cluster and introduced the concept of channels tha T is used for communication between nodes. In a later article we'll take a closer look at what the goes on the inside each of the these channels.