Cluster Level Read/Write Interaction

7 min to complete

Cluster Level Read/Write Interaction

So what happens when data is read or written at the cluster level? Note that what happens at the node level will be explained in another lesson.

Since each node is equal in Scylla, any node can receive a read/write request. These are the main steps in the process:

  1. A client connects to a Scylla node using CQL shell and performs a CQL request
  2. The node the client connected to is now designated as the Coordinator Node. The Coordinator Node, based on hashing the data, using the partition key and on the Replication Strategy, sends the request to the applicable nodes. Internode messages are sent through a messaging queue in an asynchronous way.
  3. The Consistency Level determines the number of nodes the coordinator needs to hear back from, in order for the request to be successful.
  4. The client is notified if the request was successful.

Hands-on: read and write in a multi-dc cluster

Let’s see how write and read operations works with our multi-dc cluster. First let’s open a CQL Shell to our node:

docker exec -it mms_scylla-node2_1 cqlsh
use scyllaU;

Set the consistency to EACH_QUORUM, that is, each DC must have a LOCAL_QUORUM.

consistency EACH_QUORUM;

Now let’s create a table and insert some data:

CREATE TABLE users ( user_id int, fname text, lname text, PRIMARY KEY((user_id))); 
insert into users(user_id, fname, lname) values (1, 'rick', 'sanchez'); 
insert into users(user_id, fname, lname) values (4, 'rust', 'cohle'); 

These inserts are successful. What happens if one of our datacenters is down? Pause DC2 from another terminal window:

docker-compose -f docker-compose-dc2.yml pause

Make sure that DC2 is actually down:

docker exec -it mms_scylla-node2_1 nodetool status

Now let’s try to write data again (in the CQL terminal window):

insert into users(user_id, fname, lname) values (8, 'lorne', 'malvo'); 

The write does not succeed, we get a NoHostAvailable: message. Since we set the CL to EACH_QUORUM, and one DC is down, it fails.
Now restart DC2 (again from a second terminal):

docker-compose -f docker-compose-dc2.yml unpause 

After about 60 seconds recheck the cluster status and make sure it’s up:

docker exec -it mms_scylla-node2_1 nodetool status

Now,  return to the CQL terminal and try to write the data again:

insert into users(user_id, fname, lname) values (8, 'lorne', 'malvo');

This time the write is successful. If we try to read the data:

select * from users;

We get an error “InvalidRequest: Error from server: code=2200 [Invalid query] message=”EACH_QUORUM ConsistencyLevel is only supported for writes” as Reads are NOT supported for consistency EACH_QUORUM. You can read more about this here.

Set the consistency level to LOCAL_QUORUM:

consistency LOCAL_QUORUM;

And try to read the data again:

select * from users;

Now we can see that the data is read and that our previous insert was written correctly this time.