Covers an introduction to repairing, why it’s needed and what type of repair operations exist.
ScyllaDB node may go out of sync over time – increasing the Entropy, due to Network issues, Node issues, and Rolling upgrades. This might create Inconsistent data and Data Resurrection. Some Anti-Entropy tools are Repair, Read Repair, and Hinted Handoff. This covers examples of how data can become inconsistent.
Transcript
Let’s move on to the next topic repair operations so why do we need to run repair? as you know ScyllaDB
is an eventual consistent database with tunable consistency, this means you can work with ScyllaDB
using consistency level ‘quorum’ or ‘one’ to get better availability and performance but in return
you have nodes that are slowly going out of sync, this can be a result of node restart
network issues or planned upgrade, having out of sync data can propagate to your application
and there are three anti-entropy tools to fix that repair, which is an offline process to sync
nodes in the background, read repair which is the read path functionality to sync between nodes
only for the read value and hinted handoffs which is queuing and resending messages
mutation when the node restarts, in this session we will cover only the repair option
let’s see how you can get into an inconsistent data situation, we have a 5
node cluster with replication factor three, all replicas are aligned, now we lost replica 2
while that node is down a new transaction is made, our lost node is back to life but
we have inconsistent data replicas 1 and 3 have a more recent data than replica 2, in
this case you should run nodetool repair on all nodes sequentially, one node at a time
and use the ‘pr’ flag to repair only the data that node serves as a replica for
the repair compares the data from all replicas and concludes that data reconciliation is
required for replica 2, once that’s done the data is now consistent on all replicas
another problem is the resurrection problem, so it could be that you’re doing a delete operation
the delete failed on one of the nodes, it can be for any reason, node is down, network is down, so what
I’m using consistency level ‘quorum’, so all is fine but no it’s not… let’s see how, once the tombstone
is removed after the gc_grace_seconds value, which the default is 10 days, the dead data can
resurrect as a result of a repair, read repair or even one of the node is down
when consistency level is greater than ‘one’, updates of the latest value which currently exists only on
the node that didn’t get the deleted data, will be propagated to the other two nodes
