Action Cassandra (2)-Add, replace, move, and delete nodes

boot (Bootstrap)

Adding a new node is called "booting." The Num_tokens parameter defines the number of virtual nodes (tokens) that the load node will be allocated during boot. The token defines the part of the ring (token range) that the node will be responsible for. token Assignment (token allocation)

With the default token allocation algorithm, the new node is responsible for selecting Num_tokens random tokens. Because tokens are randomly distributed, the load distribution improves with a larger number of virtual nodes, but also increases the token management overhead. The default of 256 virtual nodes should provide a reasonable load balancing and acceptable overhead.

On 3.0+, a new token allocation algorithm was introduced to allocate tokens based on the load of existing virtual nodes in a given key space, and therefore to produce an improved load distribution with fewer tokens. To use this method, the new node must use the JVM option-dcassandra.allocate_tokens_for_keyspace = <keyspace> start, where <keyspace> is the key space in which the algorithm can find payload information to optimize token allocation. Manual token Assignment (Manual token Assignment)

You can manually specify a comma-delimited list of tokens using the Initial_token cassandra.yaml parameter, and Cassandra skips the token allocation process if the token is specified. This can be useful when you use external tools to perform token allocations or when you use their previous tokens to restore nodes. range flow (range streaming)

After the token is allocated, the join node picks up the current copy of the token range responsible for streaming data. By default, it flows from the primary replica of each token range to ensure that the data in the new node will be consistent with the current state.

Under any unusable replicas, a consistent boot process will fail. To override this behavior and the data that the unavailable replica may lose, set the JVM flag-dcassandra.consistent.rangemovement = False. Recovery failed/Pending bootstrapper (Resuming failed/hanged Bootstrap)

On 2.2+, if the boot process fails, you can resume booting from the previously saved state by calling Nodetool Bootstrap resume. If for some reason the bootstrapper hangs or stops, you can also recover by simply restarting the node. In order to clear the boot state and restart, you can set the JVM boot flag-dcassandra.reset_bootstrap_progress = True.

On lower versions, when the boot process fails, it is recommended that you erase the node (delete all data) and then restart the boot process again. Manual boot (Manual bootstrapping)

You can skip the boot process completely and join the ring immediately by setting the hidden parameters auto_bootstrap:false. This can be useful when you restore a node from a backup or create a new data center. Delete node (removing nodes)

You can use Nodetool decommission to remove a node from a cluster to an active node or to delete a dead node using Nodetool removenode. This allocates the old node to be responsible for the scope to the other nodes and copies the appropriate data on the old node. If you use decommission, the data will flow out of the decommission node. If Removenode is used, the data will flow out of the remaining copies.

No data is automatically removed from the node that is being deactivated, so if you want to restore the node to a service with a different token on the ring, you should manually delete the node. Mobile node (moving nodes)

When num_tokens:1, you can use Nodetool move to move the node position in the ring. Movement is more efficient than stopping + booting, both before and after moving the nodes. After the node is moved, nodetool cleanup should delete any unnecessary data. Replace Dead node (replacing a dead node)

In order to replace the dead node, use the JVM boot flag cassandra-dcassandra.replace_address_first_boot = <dead_node_ip> boot. Once this property is enabled, the node will start in hibernation, during which all other nodes will see this node shut down.

The replacement node will immediately start booting data from the rest of the nodes in the cluster. The main difference between the normal boot of the new node is that this new node will not accept any writes at this stage.

Once the boot is complete, the node is marked as "up" and we rely on the insinuate switch to make the node inconsistent (because we do not accept writing since the boot started).

Note: If the replacement process takes longer than Max_hint_window_in_ms, you must run repair so that the placed node remains consistent again, because it misses an in-progress write during boot. Monitoring Progress (monitoring progress)

You can use Nodetool netstats to monitor boot, replace, move, and delete progress, which shows the progress of the flow operation. clean up data after range move (Cleanup after range movements)

As a security measure, Cassandra does not automatically delete data from a node that has "lost" a portion of its token scope because of a range move operation (boot, move, replace). When you are satisfied that the new node is starting and working, run Nodetool cleanup on the node that is out of range. If you do not do this, the old data will still be counted on the load on that node.