Tracing

tracing enables analyzing of internal data flows in a ScyllaDB cluster that’s not
necessarily true because well okay useful for observing behavior of
specific queries to check you know network issues data transfers
replication factor problems CQL’ed tracing so there’s two types of tracing
that’s the thing that you I want you guys to understand this is client-side
tracing so CQL is doing the tracing and it’s every step of the way
when it’s communicating with ScyllaDB it’s
it’s bringing you some information about that you can do the same if you’re using
any programming languages all of them they have tracing there you can enable
it never by default right because that would be bad because it’s going to hurt
your performance, CQL tracing it’s store on the system_traces this is an
example so remember CQLSH is interactive
so I just type tracing ON and after that everything I do on CQL it’s
going to bring me the result but also the entire tracing session so
if you’re having problems you, know exactly what is the query or the
insert that is causing you problems you have the partition key you can
experiment here okay we also have probabilistic tracing so, probabilistic
tracing is something that you enable with nodetool and of course one it’s 100
it’s a hundred percent of whatever you’re doing so be very careful with
this number here remember that one it’s a hundred percent so this is dot zero
one percent okay and again “but Moreno” what number should I set right so
you should have in mind your own workload so if you’re doing a hundred
I/O per second then you know that that’s more than enough if if you’re doing 1
million I/O per second maybe you want a even smaller value because you want to
sample you know very gently so it’s not going to impact you and by the way with
all tracing you should turn it on and it should be you know a very small window
you don’t want to leave it on forever it’s going to hurt you right we have
this low query tracing in the way to enable with low carry tracing is through
the REST API we have the documentation here again by
all means do your hands-on exercises but it’s basically you go to the API you see
enable slow tracing and you use you also specify for how long so usually you
know a couple of seconds 10/30 seconds no more than that because it’s going to
get a lot of stuff and it’s the type of tracing that you want to use
when you’re clueless about what’s going on in your database
maybe because you’re the database administrator and you don’t
know what the developers are doing they’re just complaining about
performance sometimes you just grab a couple of slow queries and you show to
them and say hey you’re passing you know 16 million values in an in clause I
saw that in a customer and well then you have a problem right don’t do that
this is another example of CQL tracing we have two DCs and this one is a query
just to the local DC so we’re probably using something like consistency level
local quorum and when we are using quorum it’s using both DC so this is what
a cross DC tracing would look like so tracing costly operations don’t enable
it by default use tracing for small periods of time when in doubt ask the
ScyllaDB team tracing can be tricky so by all means we are always there slack you
know email whatever if you don’t know if you should use some type of tracing for
you know a particular troubleshooting that you are trying to do just come to
us we advise you CQL tracing as any other client side tracing is great for a
specific queries but might not give you the full picture slow query tracing is a
great resource if you are clueless about which queries are impacting your cluster
we do have other types of tracing but it’s reserved for advanced sessions you
usually requested by a developer so you have a problem we are working
with our engineering team sometimes they will ask you for more specific stuff
very rarely used so no point in going here