Kafka – ScyllaDB Sink Connector Demo

So the entire premise of Connectors I think is a very simple one, so let’s get straight into a demo.
So in this demo, we’ll write messages to Kafka
so messages that have the same schema as ScyllaDB table schema
and the Connector will write them into ScyllaDB
So first of all, how would you download the Sink Connector?
The Sink Connector repository is an open-source repository you can browse the
source code of Sink Connector and you can download it and build it yourself
but I think the best way, and the easiest way is to use the Confluent Hub
Confluent Hub is essentially an app store for Connectors
for example you can see there are many. Connectors for Azure and other databases
and if you search for ScyllaDB Sink Connector, there it is and it’s as easy as invoking this command
so let’s go into it, let’s install Confluent Hub
and then install the Sink. Connector using Confluent Hub
First, we will download the binary of Confluent Hub
Then it’s as easy as extracting it
and afterwards you can just use the. Confluent Hub command to install Connectors
So we want to install the Sink Connector
and we will use the command that is specified here on the Confluent Hub website
we have to specify the paths where it installs to and the defaults are usually okay
and just like this, the Connector is installed
As you can remember we stopped Confluent platform, so let’s start it back
and while it’s starting let’s start ScyllaDB
during this demo, I will use ScyllaDB in Docker container, as it’s the easiest way to run ScyllaDB
I will start
I’ll start this container
I’ve already started one before, so now it’s working
and it’s starting up
in the meantime, the Kafka. Confluent platform is starting
I think it’s a good time to connect to the Docker container with ScyllaDB
we’ll use the Docker exec command
and use cqlsh and as you can see, the ScyllaDB has started and we will create the tables and
the keyspaces that will write the data from Kafka using the Sink Connector
So first we create the keyspace
Next, we’ll create the table which will gather the data from Kafka
and it will gather data from. Sink topic one topic from Kafka
As you can see the Confluent platform has started
So if we go back to the control center
and if you tab connect
it’s the Confluent Kafka connect deployment now the ScyllaDB Sink Connector is visible
and you can use it
First of all, you have to specify which topic to gather data from
and this is what we’ll do, we’ll create a new topic, we’ll name it sinktopic1
below you can specify the number of partitions so
if you wanted to have a higher throughput you specify a larger number of partitions
so the topic is now created
and if we add the Connector, we specify that we want to read the data from sinktopic1
and after we do it a plethora of options appears but hopefully, and not hopefully but
thankfully most of the options have reasonable defaults, so you only need to
change the options of a few settings, so first of all, I will specify the key converter class
This is the setting that allows you to specify the format of
generated messages in this case we’ll use the JSON format, as it’s easy to use
and to make these messages a little bit leaner
we’ll add a new property called
key.converter.schemas.enable
and if we set it to false, the generated messages will not have a schema
which will make it easier to present in this presentation,
but of course, you can leave it as default to also present schemas
Next
under the connection tab, you have to specify the IP address of ScyllaDB. So as you can remember
we have connected to ScyllaDB in Docker, so it’s at this address, at this IP address
and finally
and finally, we have to specify the keyspace
we’ll write the data to, in this case, we’ll use back the “ks” keyspace we have just created before
and now we have started the Connector and it started to listen
for messages on there, on this topic we have created
So let’s go back to ScyllaDB
and as you can see
the table we have created is empty
But if we create a Kafka message, as the key it will have the primary key of the ScyllaDB table
and as a value, it will have the entire row
If we produce the message
It was sent as you can see below, and if we go back to ScyllaDB
now this message has been successfully inserted to the
ScyllaDB table and the Connector is still listening for new messages
and you can use the Connector to insert many millions of rows
So that was a quick demo of ScyllaDB Sink Connector
so to recap we have downloaded and installed. ScyllaDB Sink Connector, we have started ScyllaDB
in a Docker container, and we have configured and started the Sink Connector