Spring Boot and ScyllaDB Hands-on Lab

In this lab, you will use the popular Framework Spring Boot to build a  sample stock application that uses ScyllaDB to store time series data (stock price). The application has several APIs that support the create, get, and delete operations on stocks with the date and price. Additionally, an HTTP API tool, Postman, is used to interact with our application and to test the API functionality. 

By the end of the lab, you’ll have a  running Spring Boot app that serves as an HTTP API server with ScyllaDB as the underlying data storage. You’ll be able to create stocks with prices and dates and get a list of stock items. You’ll learn how ScyllaDB can be used to store time series data.

The supplementary videos at the beginning of the lesson accompany this hands-on lab. It’s recommended that you watch them before running the lab.

What’s Spring Boot

Spring Boot is an open-source micro framework maintained by a company called Pivotal. It provides Java developers with a platform to get started with an auto-configurable production-grade Spring application. With it, developers can get started quickly without wasting time preparing and configuring their Spring application.

Spring Boot is built on top of the Spring framework and has many dependencies that can be plugged into the Spring application. Some examples are Spring Kafka, Spring LDAP, Spring Web Services, and Spring Security. However, developers have to configure each building brick using many XML configuration files or annotations.

What’s Postman 

Postman is an API tool for building and using APIs. It simplifies each step of the API lifecycle and streamlines collaboration so you can create better APIs. It enables you to easily explore, debug, and test your APIs while also enabling you to define complex API requests for HTTP, REST, SOAP, GraphQL, and WebSockets.

The API client automatically detects the language of the response, links, and format text inside the Body to make inspection easy. The client also includes built-in support for authentication protocols like OAuth 1.2/2.0, AWS Signature, Hawk, and many more.

Through the API client, you can organize requests into Postman Collections to help you organize your requests for reuse so you don’t waste time building everything from scratch. Your collections can also contain JavaScript code to tie requests together or automate common workflows, and you can use scripting to visualize your API responses as charts and graphs.

Service Setup Prerequisites:

You can run this lab with a cluster created using ScyllaDB Cloud or Docker. The following steps use Docker. If you choose ScyllaDB Cloud, you can skip the docker part.

1. Docker for Linux, Mac, or Windows. Please note that running ScyllaDB in Docker is only recommended to evaluate and try ScyllaDB. A regular install (or using ScyllaDB Cloud) is recommended for the best performance in production.
2. 8GB of RAM or greater for the sample application and ScyllaDB services.
3. Docker-compose used to launch the ScyllaDB cluster.

• OpenJDK 17, required by the Spring Boot application

Setup a ScyllaDB Cluster

If you choose to run the cluster on ScyllaDB Cloud, skip the docker-compose section part below.

Get the sample project code from the GitHub repo and start a three-node ScyllaDB cluster using docker-compose:

git clone https://github.com/scylladb/scylla-code-samples

cd scylla-code-samples/spring/springdemo-custom

docker-compose -f docker-compose-spring.yml up -d

Wait one minute or so and check that the cluster is up and running:

docker exec -it scylla-node1 nodetool status

Time series data model

You can learn more about compaction strategies here and specifically about  TWCS in this lesson. 

Once the cluster status is ready, create the data model using cqlsh for the stock application. You can also see the data schema in the sample repo.

docker exec -it scylla-node1 cqlsh

CREATE KEYSPACE IF NOT EXISTS springdemo WITH replication = {'class':'NetworkTopologyStrategy', 'replication_factor':3} AND durable_writes = false;

CREATE TABLE IF NOT EXISTS springdemo.stocks (symbol text, date timestamp, value decimal, PRIMARY KEY (symbol, date)) WITH CLUSTERING ORDER BY (date DESC);

ALTER TABLE springdemo.stocks WITH compaction = { 'class' : 'TimeWindowCompactionStrategy', 'compaction_window_unit' : 'DAYS',  'compaction_window_size' : 31 };

ALTER TABLE springdemo.stocks WITH default_time_to_live = 94608000;

exit

The above creates a keyspace called springdemo and sets the replication to 3, equal to the cluster size. The table stocks contains the columns symbol, date, and value to store the time series stock data. It also sets the table stocks compaction policy to TWCS  with 31 days as the compaction window.

Also, since it is assumed you are only interested in the last three years of quotes, the TTL is set (in seconds) for the table. 

Time-Window Compaction Strategy compacts SSTables within each time window using the Size-tiered Compaction Strategy (STCS). SSTables from different time windows are never compacted together. You set the TimeWindowCompactionStrategy parameters when you create a table using a CQL command.

Time-Window Compaction Strategy (TWCS)

It works as follows:

• A time window is configured. The window is determined by the compaction window size compaction_window_size and the time unit (compaction_window_unit).
• SSTables created within the time window are compacted using Size-tiered Compaction Strategy (STCS).
• Once a time window ends, take all SSTables created during the time window and compact the data into one SSTable.
• The final resulting SSTable is never compacted with other time-windows SSTables.
• With this explanation, if the time window were for one day, at the end of the day, the SSTables accumulated for that day only would be compacted into one SSTable.

Build the App With the Example Code

The spring boot application has a default configuration file in the /spring/springdemo-custom/src/main/resources/ directory named application.yml. In it, you can add custom config settings such as the ScyllaDB nodes IP address and port, credentials, and the name of your data center (DC1 by default).

Edit application.yml with your system settings. 

Then, build the application:

./gradlew build

Launch the spring boot application:

java -jar ./build/libs/springdemo-custom-0.0.1-SNAPSHOT.jar

The application runs as an HTTP server and, by default, listens to port 8082. 

Test the Application’s API

Download the Postman app from https://www.postman.com/downloads/, and install it to your dev environment. You can import an existing API definition into Postman. API definitions can be imported from a local file or directory, a URL, raw text, a code repository, or an API gateway.

This is the API definition file named /spring/springdemo-custom/Springdemo-custom.postman_collection.json:

Launch the app and import it by clicking Collections, Import, File, and Upload Files. This file contains the stock application API definition.

After importing the JSON file, you will see a list of collections used to interact with the stock application’s API with predefined data requests as an example. The first two are used to send HTTP POST requests to create the AAPL and MSFT stocks with a stock value and date. After creating the stocks, you can then use the list API to get a list of stocks. All the stock data is stored in the created ScyllaDB table stocks.

Click Body to see the Post data to be sent through the HTTP API to the Java sample application. You can see that the JSON includes the symbol, date, and value items. Note that if your Java sample application isn’t running in localhost, you need to replace localhost with the actual IP on which your application is running.   Click Send to use Postman to send the HTTP POST request to the application as below.

This is the response after you send the request to create the APPL stock:

Next, select create stock MSFT and click the send button to create the stock.

Data in ScyllaDB

You can now see the data in the ScyllaDB table. Create a new terminal tab and:

docker exec -ti scylla-node1 cqlsh

select * from springdemo.stocks;

You can also delete a stock and get the stocks via Postman. 

While this is not the focus of this lesson, note that the driver uses Prepared Statements for the queries. Doing this improves performance by making sure the preprocessing of the statements is done only once. Additionally, since the driver used is Token Aware, the meta information returned with the identifier allows the driver to direct the executed statement to a replica node saving an extra hop. 

This is the code of the findStockBySymbol method, for example:

Additional Resources

• Time series data https://www.scylladb.com/glossary/time-series-data/
• ScyllaDB data compaction https://docs.scylladb.com/stable/cql/compaction.html
• Compaction strategies https://docs.scylladb.com/stable/architecture/compaction/compaction-strategies.html
• Scylladb expiring data with TTL https://docs.scylladb.com/stable/cql/time-to-live.html

Summary

In this lab, you built an application with the Spring Boot framework that stores time series stock data to ScyllaDB. It uses the TimeWindowCompactionStrategy. 

Choosing this compaction strategy reduces read amplification, write amplification, and space amplification and is recommended for time series data stored in ScyllaDB.