Nodetool is a command-line interface that enables managing and collecting information on ScyllaDB clusters. It has two types of commands: informative commands and management commands. On systems with systemd, ScyllaDB logs can be retrieved with the journalctl command.
The goal here is to show you the most common tools the
ScyllaDB administrators should use and to point to you where to find the resources
the documentation the manual pages help you name it
give a man a fish you feed him for a day you know the rest right so let’s start
with nodetool and logs so nodetool is the common line interface for managing a node so
there are two types of nodetool commands the ones that are going to give you
information about that node or the cluster sometimes and the ones that are
going to actually do something so I’ll show you some examples for nodetool status
informative command it’s giving the status of all the nodes you know if it’s
up/normal down/normal leaving joining moving I’m not going to go into
specifics of nodetool commands first because there’s a gazillion nodetool commands
second each one of them have sub-options so sometimes you can see by
the node but you can drill down to a key space sometimes to a table so nodetool info
nodetool cfhistograms this is actually interesting even though the
screenshot is not very interesting but if it was a normal table you would see
on these histograms you would see the distribution of writes reads the
partition sizes very nice you find a you know a big partition that kind of stuff
table stats as you can see only this command for getting statistics of a
table it can be that big so it would depend on what you’re looking for I
always tell people nodetool is okay sometimes we need to go into that your
monitoring should have told you before you needed nodetool so again several
options of nodetool like this one might be familiar if you want to run a major
compaction you’re on your cluster if you’re cleaning up after a new node
join and so on and so forth so in logs because ScyllaDB runs on Linux and after
a long battle some of you might be familiar or not systemd won the war and
now we are using journalctl just because it’s what comes with systemd
the nice thing about journalctl is that you can filter the log messages so for
example on the first one we are filtering by ScyllaDB server which is the
name of the service that ScyllaDB runs but you can you know
drill down you can filter by priority by date you know there’s million tricks you
can do with journalctl by all means go to the journalctl man page there’s like a
gazillion options and whenever we ask you for logs because you’re working with the
ScyllaDB team let’s say you’re customer you’re user doesn’t matter you have a problem
let’s say you’re your cluster is running for
one year and you had a problem yesterday please don’t send me one year of one
year worth of logs just send me you know like two last days something like that
so it’s easier to pinpoint where we are finding you know those problems or those
errors so these are the takeaways for nodetool and journalctl be familiar with
the documentation in the man pages after you have this presentation you see
there the links are there so it’s easy to follow always check your overall
cluster health before you do anything experiment with nodetool on a controlled
environment before running things on production so my main advice for you
guys and you’re going to see that on the hands-on exercises please get a docker
container please have you know one little cluster on development try things
there before you start typing commands in production I’ve seen some bad stuff
happened just because people are you know not careful so try to limit the
log output to meaningful periods use all the filters also look at the
monitoring and use CQLSH to test your hypothesis
