And then, there are the Scylla best practices right, so the first thing is, we say
that in pretty much all of our training sessions, please have Scylla monitoring
the reason for that is plain and simple visibility and transparency
you want to know what’s going on inside your cluster, inside your nodes and you want to be
able to drill down in a particular node and look at things like number of transactions per CPU
latencies, how your cache is being used if you have a bottleneck on the I/O side or on
the network side and on our documentation on on the Scylla Operator GitHub, you see that we’re
using helm charts to deploy Prometheus and Grafana because our monitoring solution is basically that
and annotations to collect the metrics from the Scylla nodes inside Prometheus and then
Grafana will show you all those beautiful dashboards, that are very easy to understand
this is going to give you full visibility into the cluster and it’s going to make your life and
our lives, when we’re trying to help you much easier, also have Scylla Manager, for those
maintenance operations that the Operator is not doing by itself, because we’re trying to separate
duties here, so the Operator is taking care of you know deploying resources and doing operations on
the Kubernetes cluster, but for example repairs and backups they’re just regular maintenance
operations, we don’t need to re-implement that on the Scylla Operator, we can just leverage
Scylla Manager with the Scylla Operator, right to run those operations, as I mentioned the
Scylla pods already come with the Scylla Manager agent container, so each Scylla pod is comprised of two
containers, one for Scylla and one for the Scylla. Manager agent, that we also call the ‘Sidecar’
on the manifest file, you remember that I showed you how to deploy a cluster using the Operator
it’s basically a scyllacluster.yaml manifest file this is a snippet from that manifest, we want you
to use the cpuset as ‘true’ and the hostNetworking as ‘true’, so if you did your homework and
you’re using the correct CPU Manager, a static policy on the Kubernetes side and those nodes
are prepared to receive a Scylla pod, this is going to ensure that Scylla is doing the CPU pinning that is
very beneficial for Scylla’s performance and we have an article on our blog
called the ‘cost of containerization’ and you can see there that we we run all the benchmarks and
we prove that cpuset is very important, also hosting network, it’s basically telling that
you don’t want to use the virtualized Kubernetes network for your pod, you just want your pod to use
the hostNetwork, so the network from the Kubernetes node where your pod resides
this is going to bring back a lot of performance, even though you’re running on top of Kubernetes
after you deployed your Operator, you deployed
your Scylla cluster on Kubernetes, these are the things that I advise you to take a quick look, just
make sure everything’s same, so run nodetool status in one of the pods, so you can see that you have
the correct datacenter, that your pods are mapped to the correct racks or availability zones
you can check that CPU pinning, that I mentioned before see if you know all the
CPUs are there and of course the I/O settings, as. I mentioned one of the things that Scylla does is
to benchmark your disks, so here on the io_properties.yaml, you will be able to see
number of IOPS and bandwidth for both reads and writes, for
random and incremental reads and writes
and you know what to expect, in terms of capacity for each node running on Kubernetes
