So what happens when data is read or written at the cluster level? Note that what happens at the node level will be explained in another lesson.

Since each node is equal in ScyllaDB, any node can receive a read/write request. These are the main steps in the process:

1. A client connects to a ScyllaDB node using the CQL shell and performs a CQL request
2. The node the client is connected to is now designated as the Coordinator Node. The Coordinator Node, based on hashing the data, using the partition key and on the Replication Strategy, sends the request to the applicable nodes. Internode messages are sent through a messaging queue asynchronously.
3. The Consistency Level determines the number of nodes the coordinator needs to hear back from for the request to be successful.
4. The client is notified if the request is successful.

Hands-on: read and write in a multi-dc cluster

Let’s see how write and read operations works with our multi-dc cluster. First, let’s open a CQL Shell to our node:

docker exec -it scylla-node2 cqlsh

use scyllaU;

Set the consistency to EACH_QUORUM, that is, each DC must have a LOCAL_QUORUM.

consistency EACH_QUORUM;

Now let’s create a table and insert some data:

CREATE TABLE users ( user_id int, fname text, lname text, PRIMARY KEY((user_id))); 
insert into users(user_id, fname, lname) values (1, 'rick', 'sanchez'); 
insert into users(user_id, fname, lname) values (4, 'rust', 'cohle'); 

These inserts are successful. What happens if one of our datacenters is down?

Open a new terminal window and pause DC2:

docker-compose -f docker-compose-dc2.yml pause

Make sure that DC2 is actually down. This might take a few seconds:

docker exec -it scylla-node2 nodetool status

Now let’s try to write data again (in the CQL terminal window):

insert into users(user_id, fname, lname) values (8, 'lorne', 'malvo'); 

The write does not succeed. We get a NoHostAvailable: message. Since we set the CL to EACH_QUORUM, and one DC is down, it fails.
Now restart DC2 (again from a second terminal):

docker-compose -f docker-compose-dc2.yml unpause 

After about 60 seconds, recheck the cluster status and make sure it’s up:

docker exec -it scylla-node2 nodetool status

Now,  return to the CQL terminal and try to write the data again:

insert into users(user_id, fname, lname) values (8, 'lorne', 'malvo');

This time the write is successful. If we try to read the data:

select * from users;

We get an error “InvalidRequest: Error from server: code=2200 [Invalid query] message= “EACH_QUORUM ConsistencyLevel is only supported for writes” as Reads are NOT supported for consistency EACH_QUORUM. You can read more about this here.

Set the consistency level to LOCAL_QUORUM:

consistency LOCAL_QUORUM;

And try to read the data again:

select * from users;

Now we can see that the data is read and that our previous insert was written correctly this time.