Bootstrap a New Node into a Cluster

Let’s start with the first topic, when you want to bootstrap a new node into the cluster, there
are a few parameters you need to set in ScyllaDB.yaml, the cluster_name, which needs to be identical as
the other nodes, the listen_address, which is the IP, that ScyllaDB uses to connect to other
ScyllaDB nodes in the cluster, the seeds which are used during startup to discover the cluster
rpc_address, which is the address for client connection using CQL or Thrift
or a bootstrap, by default this parameter is set to ‘true’ and most likely you won’t have to touch it
it allows new nodes to migrate data to themselves automatically, the endpoint_snitch which you need
to set as well, the snitch determines which data centers and rack nodes belong to
snitches inform ScyllaDB about the network topology so that requests are routed efficiently and
allows ScyllaDB to distribute replicas by grouping machines into data centers and rack, now the
ScyllaDB.yaml comes with the default of simple snitch, this should not be used in production
for production you should be using only. Ec2MultiRegionSnitch, for AWS Cloud deployments
across one or more regions, the GoogleCloudSnitch for GCP deployments across one or more regions
or the gossiping property files snitch, GPFS for bare metal and other Cloud vendors
other snitches are limited and will make it harder for you to add a data center later on
so the cluster name must be identical to the cluster you’re going to add the node to
otherwise that node won’t be able to join, don’t use generic cluster name that might conflict
with other clusters with accessible network, the seeds act as a place for new nodes to announce
themselves to a cluster, so without at least one live seed node, no new node can join the cluster
so for that reason, we recommend having more than one seed node
for redundancy purposes but on the other hand do not add all the nodes as seeds
so not to make all nodes a gossip hotspot, usually two to three seeds per data center are sufficient
another important thing is to prepare your nodes in terms of
opening the relevant ports so nothing will be blocked and
the node could join and communicate with other third-party utilities as well
if the new node started before it joined the cluster, you need to stop the ScyllaDB service and
delete the content of data, commit log and hints folders, after you’ve done that, you can start the
node again and observe the nodetool status, watch the node move to up joining status while data
is being streamed to it and when the streaming is done, you can now see it in up, normal state
then you need to use the ‘nodetool cleanup’ on all the veteran nodes in the cluster
the cleanup will remove keys that no longer belong to the nodes, you need to run this
command one node at a time and it is possible to postpone this step to low demand hours