Covers topics such as hardware selection, number of cores, storage layout and types, and number of nodes.
Where to run it?
You can make the choice of I should I run it in this cloud and that cloud should I
run it Bare Metal should I run in here, essentially as I said, you can run it
anywhere but the choice of hardware can make a lot of difference it can matter a lot.
One of the things for you to keep in mind is that where are your
bottlenecks coming from, I mean, are you bottlenecked by the CPU or are you
bottlenecked by the storage both in terms of sizing or in terms of speed if
you are bottlenecked by the storage, specialist storage size try to go with a
larger instance, we have for instance, AWS just announced last year or earlier this
year instances with 60 terabytes of data they have fewer cores
per terabyte but if that is what you need that is a better choice than trying
to squeeze in into an instance of a different type. HDDs is in general I mean
I usually don’t recommend some people do have them and we try to make the best
out of it but SSDs are getting cheaper and cheaper and cheaper by the day
so it’s a much much better bet for you to go on SSDs and if you need the
latencies then NVMes are essentially a better option. For throughput you can
still put a lot of SSDs together on on a. RAID array and you gonna get
good throughput but the latencies coming from an NVMe is much better so
our recommendation is always like – if you can go with it.
Networking in general just forget gigabit networks they’re definitely not
fast enough, we see a lot of problems from people still insisting maybe one
gigabit a second is enough.. It’s not, you’re gonna
start having issues, obviously if you’re on the cloud that’s not that important
but if you manage your own on-prem deployments pay attention to that as
well. Storage layout how to best organize many disks
there are various values cases in which you don’t want to follow this advice but
the advice is still – try to use RAID0 and not a higher RAID set up and
the reason we have this advice I just want to talk about reasoning so it
doesn’t feel like just an empty advice is that ScyllaDB is already replicated and
it is designed in a way that if you lose a node you can replace that node pretty
fast so why would you have, I said, there are valid use cases but in the
general case why would you have a replicated array of disks that if you
lose a disk and just replace the entire node it should be fast enough and
that gives you speed back so you actually get your day-to-day
faster because your disks are faster your array is faster and the rare event of
losing of losing a disk you can just replace the entire node. Splitting commitlog
and data it’s also a question that we we hear a lot, I mean
should I buy a new disk fastest just for commitlog? My answer is usually – no. You
don’t have to go and spend the money or whatever like attach a
different disk just to do that. We have the schedulers in ScyllaDB. It’s part of our
foundation, it’s a part of what we do that are very good at isolating the
workload but if you happen to have, because you bought the
hardware this way and doesn’t change the pricing or you got a promo or
whatever, you happen to have a fast disk for commitlog then you can
dedicate that volume just for commitlog, but I wouldn’t go out of my
way to actually get something just for this purpose
In Terms of hardware sizing I’m not going to spend too much time on how to
actually size it because we have a presentation from our VP of solutions Eyal
just about that so attend the presentation if you’re interested but I
will highlight one fact which is a lot of times we see people sizing the
hardware as if you are going on a trip in which you plan to spend $1000 and then
you take with you $1000 like if you’ve ever done it, you know, you are never
doing it again because what happens is that you had an expense that you weren’t
really counting on and then where is your money? I ran out of money so when
you go on a trip you always take extra because you know that something is going
to happen, you don’t know what, but you know that something is going to happen
so you prepare and leave some spare, so my advice in terms of sizing is to do
that too. Take into account that you’re gonna lose a node there you’re gonna
maybe have to do some manual operation and transfer data, copy data so leaving
some spare on the disk is good as well so you’re gonna have more leeway to deal
with issues as they happen. It’s rare that they happen but when they do happen
you want to have the spare to actually make sure that you can handle it without
affecting your production, you do that by having spare. How you combine this
hardware? So again we hear this question again and again should I use a lot of
small nodes or should I use fewer bigger nodes, so in which situations should I
use more smaller nodes – never. And again we have a white paper
about that we’ve done webinars in the past we had experiments that
show that, the usual concern is but if I lose a node then it’s
more expensive to replace it it really isn’t we have data to back it up I mean
if you have more data in the same node sure it gets more expensive to replace
that node but if you double the data set but also double the power of the node
this is essentially constant it’s what we see in the graph so I mean if
you plan to double your data set consider doubling the size of your nodes
first and try to run it because it’s, operationally, it’s much easier
to handle things that are just fewer entities if you have to do like a kernel
upgrade or etc it’s just easier to do it on fewer
machines. However, just don’t kill me, I have to tell people like how
it really is. There is one situation in which I’m okay if you’re gonna run
smaller nodes and what is that situation when your cluster is small enough
so that when you are expanding you don’t necessarily want to double so in our
example here you have 32 cores and then. I want to expand it so that I have like
32 cores total and I want to expand it so for me to expand it because I want my
nodes to be the same size I am forced to go to 64 cores total unless I go
heterogeneous which isn’t great so that’s why in this situation it’s
actually okay to try to run smaller nodes because when I expand for instance
if I’m packing this on 16 cores I can add a new machine with 16 cores and
that takes me to a total of 48 cores but as soon as you get big enough I mean as
soon as you start having like I’m having three nodes of the of the biggest
possible and I’m okay which is ,actually again recommended, I’m okay
doubling the size then try to pack it in as fewer machines as possible but if you
have this concern what about my first expansion? That is a use case in which
it will be okay for you to try, don’t run don’t go like a bunch of four cores
but maybe take it down a notch it’s a valid concern you have
