K8 Operator – ScyllaDB + Operator Best Practices

And then, there are the ScyllaDB best practices right, so the first thing is, we say
that in pretty much all of our training sessions, please have ScyllaDB monitoring
the reason for that is plain and simple visibility and transparency
you want to know what’s going on inside your cluster, inside your nodes and you want to be
able to drill down in a particular node and look at things like number of transactions per CPU
latencies, how your cache is being used if you have a bottleneck on the I/O side or on
the network side and on our documentation on on the ScyllaDB Operator GitHub, you see that we’re
using helm charts to deploy Prometheus and Grafana because our monitoring solution is basically that
and annotations to collect the metrics from the ScyllaDB nodes inside Prometheus and then
Grafana will show you all those beautiful dashboards, that are very easy to understand
this is going to give you full visibility into the cluster and it’s going to make your life and
our lives, when we’re trying to help you much easier, also have ScyllaDB Manager, for those
maintenance operations that the Operator is not doing by itself, because we’re trying to separate
duties here, so the Operator is taking care of you know deploying resources and doing operations on
the Kubernetes cluster, but for example repairs and backups they’re just regular maintenance
operations, we don’t need to re-implement that on the ScyllaDB Operator, we can just leverage
ScyllaDB Manager with the ScyllaDB Operator, right to run those operations, as I mentioned the
ScyllaDB pods already come with the ScyllaDB Manager agent container, so each ScyllaDB pod is comprised of two
containers, one for ScyllaDB and one for the ScyllaDB. Manager agent, that we also call the ‘Sidecar’
on the manifest file, you remember that I showed you how to deploy a cluster using the Operator
it’s basically a scyllacluster.yaml manifest file this is a snippet from that manifest, we want you
to use the cpuset as ‘true’ and the hostNetworking as ‘true’, so if you did your homework and
you’re using the correct CPU Manager, a static policy on the Kubernetes side and those nodes
are prepared to receive a ScyllaDB pod, this is going to ensure that ScyllaDB is doing the CPU pinning that is
very beneficial for ScyllaDB’s performance and we have an article on our blog
called the ‘cost of containerization’ and you can see there that we we run all the benchmarks and
we prove that cpuset is very important, also hosting network, it’s basically telling that
you don’t want to use the virtualized Kubernetes network for your pod, you just want your pod to use
the hostNetwork, so the network from the Kubernetes node where your pod resides
this is going to bring back a lot of performance, even though you’re running on top of Kubernetes
after you deployed your Operator, you deployed
your ScyllaDB cluster on Kubernetes, these are the things that I advise you to take a quick look, just
make sure everything’s same, so run nodetool status in one of the pods, so you can see that you have
the correct datacenter, that your pods are mapped to the correct racks or availability zones
you can check that CPU pinning, that I mentioned before see if you know all the
CPUs are there and of course the I/O settings, as. I mentioned one of the things that ScyllaDB does is
to benchmark your disks, so here on the io_properties.yaml, you will be able to see
number of IOPS and bandwidth for both reads and writes, for
random and incremental reads and writes
and you know what to expect, in terms of capacity for each node running on Kubernetes