This lesson takes a deep dive into ScyllaDB Compaction Strategies. It covers: Bloom Filters, the Read and Write Path, Storage – Log-Structured Merge Tree, Compaction Efficiency, and provides a Compaction Strategies Overview.
One of the things that is
very important so you’ll understand some of the concepts we’re going to talk about our bloom filters so bloom filters are probabilistic filters they can be used to query whether or not a key exists in a file now as I said they are probabilistic so they’re not going to give you a yes-or-no answer if they tell you that the key is not in the file you can be 100% sure that that’s true so you can skip it if the bloom filter tells you this key is not present it’s not present you don’t need to bother you don’t need to read that file but if they tell you yes this could still be a false positive so that’s it in this
way they’re probabilistic you look into the file but if they tell you ‘no’ for sure it’s not there you can skip that file so this is very important as a read path optimization some of you are usually concerned Oh what happens if I have a thousand files that’s not necessarily a concern because the bloom filters will reduce your need for access the fact that you have a thousand files doesn’t mean that you need to access a thousand files only if it is a if at all possible that your key could be there so when we talk about read pathing and read amplification bloom filters are one of the concepts that we are going to need right again the write path we’re not going to cover we write very quickly we write your memtables those mem tables are written to sstables sstables are immutable they never change which means we need to compact them whenever every now and then depending on the structure so this is how it happens you all told me you know what compactions are now what is important when we talk about compaction and what are the things that we’re going to be using today to classify and compare those compaction strategies we’re going to be essentially looking at three things we’re going to be looking at write amplification what is write amplification is the number of times you have to rewrite the same data so I wrote data into disk now I have to rewrite that data again why do I have to rewrite it again is
interesting but why do I have to write that data again because it was an on one sstable file and I’m compacting it to another sstable file them being
immutable and this is specially
important if I didn’t really have to write this data again it just happens to be on a file that I’m compacting so that’s that’s write amplification we also have read amplification which is how many files that I have to read that I would have been better if I had to read from only one so if my data is now split into two files I have a read amplification factor of two because I would be better off if my data was all back into a single file and there is also space amplification so again we write things are immutable so we if you write something and then you write the same thing again you’re going to end up with two copies of that on disk until you compact it to bring it back to one and then space amplification is exactly that in this scenario you have a space amplification of two all right so sstable merge is efficient again one of the things that we do I would like to highlight we believe of course and a lot of you are using ScyllaDB experience with that you’ve seen this to be true we have the schedulers we have the controllers so usually compactions are they have their own time slice the scheduler is controlling whether or not you’re compacting or for how long you’re compacting so the goal one of the goals we have of compaction is to have keep having low tail latencies even during compactions again because we have the schedulers and and the and and the controllers isolating that that doesn’t mean that’s also common is
conception that your latency is not going to go up if you’re not doing anything and now you’re doing something your latencies going to go up the goal is that it’s not going to go up wildly and by too much right we saw the compaction is essentially you grab a bunch of files and you’re going to combine those files into fewer files it doesn’t have to be one by the way you can you can still generate as there is an output of a compaction more than one file but the goal is usually to reduce our state so you have you have lower and therefore better read amplification in the future but how do we choose those files because if you just always compact all the files into one we’re also increasing a write amplification by too much right so imagine every time you have a new SStable you compact this sstable against your entire set your write amplification is going to be terrible so the goal is essentially reduce your read amplification as much as you can
and trying to keep the write
amplification under control that means that every workload will benefit from a different strategy on how to pick those files and that’s where the compaction strategies come into play so ScyllaDB supports a fair amount of compaction strategies and this is
essentially the goal of the presentation today understand what they do so you can understand and those are the criterias with which we will judge those
compaction strategies once you
understand what they do you’ll
understand what’s the best one for my use case we’re going to be talking about the size tiered compaction strategy the time window compaction strategy and then the leveled compaction strategy and finally one compaction strategy that is soon to debut in ScyllaDB enterprise which is the incremental compaction strategy this is essentially code ready for a while now but we
want to make sure that some of those properties in testing weren’t holding in some scenarios but we’re almost there so
