ScyllaDB is built to work with multiple data centers, for performance reasons as well as for extra availability. Adding and removing a data center requires different steps performed on the existing nodes, on new nodes, configuration changes, and possibly changes to keyspaces. The details are covered here.
Let’s continue to the next topic, adding and removing data centers from or to the cluster
there are steps to be done on the existing nodes in the existing data centers and steps to be done
on the new nodes in the new data center, that you want to add, you need to configure the
scylla.yaml on the existing nodes data center and make sure the endpoint_snitch is on
Ec2MultiRegionSnitch, for AWS Cloud multi DC or GPFS for bare metal or other non-AWS Cloud
make sure all your key spaces are using network topology strategy and not simple strategy
if it’s not the case make sure to change that, if you are going to use GPFS snitch
make sure to set the DC and Rack Names in the cassandra.-rackdc.properties file
after that, you can restart the nodes one by one for the configuration file changes to take effect
install ScyllaDB on the new nodes in the new data center and configure scylla.yaml on each of the
nodes with the following values, the endpoint snitch, again Ec2MultiRegionSnitch or GPFS
and the seeds, the parameters of the seeds, cluster name and endpoint snitch must match
the existing cluster, if the new nodes in the new data center started before joining the cluster
go ahead and stop the service and delete the content of the data commit log and hints folders
after that you can start the nodes again one by one and they should join
the cluster, if you already set the scylla.yaml and the cassandra-rackdc.properties
accordingly, don’t forget to alter system key spaces, the system of, system distributed and
system traces, after the nodes already joined the cluster, you can see that with nodetool status, you
can now start altering the key spaces to replicate into the new data center and you can do that on
the system key spaces, which is very important, the system of system distributed and system traces
after that, you can you should also change your data or user key spaces to add the replication
to the new data centers, you should also add the. IPs of the nodes that will act as seeds in the new
data center to the existing nodes, after you’ve done that you need to run the rebuild command
make sure to specify the existing DC name in the rebuild command, if the existing DC
wasn’t specified in the command, the new nodes will seem to rebuild successfully but will not
contain any data, if you are also using ScyllaDB. Monitoring and ScyllaDB Manager, make sure to update
those stacks accordingly to be able to connect to them and to monitor or to manage them accordingly
after everything was set up, before doing that you should run a full cluster repair
so after the rebuild completes, you should run the full cluster repair and after that
you should verify that the monitoring stack is able to monitor the new nodes and ScyllaDB manager
is able to connect to the new data center, that will require you to install ScyllaDB manager agents
on the new nodes, after you’ve done all of that don’t forget to adjust your clients’ connections
to connect to the local data center, as well as their consistency level to use the ‘local_quorum’
now let’s talk how to remove a data center from a cluster, first of all, verify there are no client
writes to nodes in the data center that is about to be decommissioned, run a repair on each node
in that data center, modify the schema and alter each key space and remove that data center from
the replication, after that you can use the ‘nodetool removenode’ to remove all the nodes from the
data center and finally you can check in nodetool status, that the data center was indeed removed
