There could be different reasons for removing a node from a running ScyllaDB cluster, including the node having hardware problems, lower demand for the application, and a desire to reduce cost. This covers the steps for removing a node when it is up and running and also when it is down for whatever reason.
Transcript
Now let’s talk about how to remove a node
why do we need to remove a node? it could be hardware problems
or we want to reduce the cost due to lower demand, so no reason to be over-provisioned
before we start removing a node, we need to check what is the status of the nodes in the cluster
verify the other nodes in the cluster have sufficient disk space for the additional
data they will need to hold, once the data is streamed from the node you are about to remove
you can take a node out of the cluster with ‘nodetool decommission’ if it’s a live node or
‘nodetool removenode’ from any other machine to remove a dead node, this will assign the ranges the old node
was responsible for to other nodes and replicate the appropriate data there, if decommission is
used, the data will stream from the decommission node, if remove node is used, the data will stream
from the remaining replicas, do note that ‘nodetool decommission’ is an irreversible action and must be
used with caution, you can watch the data being streamed by using the ‘nodetool netstats’ command
if the node is down temporarily and can be booted up again, there is no need to do remove node
operation, this operation like decommission, removes the node token
and does streaming and rebalancing of the data, no data is removed
automatically from the node being decommissioned so if you want to put that node back into service
at a different token ranges or even in a different cluster, the data needs to be removed manually, so
make sure you delete the data, commit log, and hints folder after the node was removed
