Monitoring and Manager Lab

This lab is based on Manager version 2.6. In this lab, you’ll:

• Start a three-node ScyllaDB cluster (with a ScyllaDB Manager Agent)
• Start a MinIO instance used for backups
• Start ScyllaDB Manager and connect it to the ScyllaDB cluster and to the MinIO instance
• Set up ScyllaDB Monitoring
• Simulate different events (node down, database queries) and see what happens in the Monitoring stack
• Generate an Alert
• Perform operations such as backup and repair using ScyllaDB Manager

Setting up ScyllaDB Manager

If you haven’t done so yet in previous lessons, download the example from git:

git clone https://github.com/scylladb/scylla-code-samples.git

Go to the directory of the manager example:

cd scylla-code-samples/manager

Next, you’ll use docker-compose to create:

• A three-node ScyllaDB cluster
• A single ScyllaDB node used as the ScyllaDB Manager backend. ScyllaDB Manager uses this database to save its data. 
• ScyllaDB Manager (with the Agent installed).
• A MinIO instance used for backups.

Before starting the cluster, make sure the aio-max-nr value is high enough (1048576 or more). 

This parameter determines the maximum number of allowable Asynchronous non-blocking I/O (AIO) concurrent requests by the Linux Kernel, and it helps ScyllaDB perform in a heavy I/O workload environment.

Check the value: 

cat /proc/sys/fs/aio-max-nr

If it needs to be changed:

echo "fs.aio-max-nr = 1048576" >> /etc/sysctl.conf

sysctl -p /etc/sysctl.conf

Keep in mind that this example is not a production setup. Make sure to read the documentation before running it in production. 

docker-compose build

docker-compose up -d

This might download and update some packages. Wait until the ScyllaDB cluster is up and running (UN status). It may take several minutes. Check the status with:

docker-compose exec scylla-node1 nodetool status

Add the cluster to ScyllaDB Manager:

docker-compose exec scylla-manager sctool cluster add --name test_university --host=scylla-node1 --auth-token=token

Check the cluster status in ScyllaDB Manager:

docker-compose exec scylla-manager sctool status -c test_university

To run a backup to MinIO:

docker-compose exec scylla-manager sctool backup -c test_university --location s3:docker

Check the backup status:

docker-compose exec scylla-manager sctool task list

Setup ScyllaDB Monitoring

For more information about setting up Monitoring, complete this lesson first.

Next, we’ll set up ScyllaDB Monitoring and connect it to the Manager.

cd ../..

git clone https://github.com/scylladb/scylla-monitoring.git

cd scylla-monitoring

Extract the IP address of ScyllaDB and Manager for the next step

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' scylla-node1 scylla-node2 scylla-node3

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' manager-scylla-manager-1

Rename the file prometheus/scylla_servers.example.yml to prometheus/scylla_servers.yml and edit it according to your node IP addresses as follows:

Rename the file prometheus/scylla_manager_servers.example.yml to prometheus/scylla_manager_servers.yml and modify it according to your node IP addresses as follows:

Generate the dashboards:

./generate-dashboards.sh -F -v 2021.1

Finally, start the stack. Replace manager_public below with your docker network.

./start-all.sh -v 2021.1 -M 2.0 -D "--net=manager_public"

Next, browse to http://localhost:3000 to view the Monitoring homepage.

Simulate Different Events

Stop the ScyllaDB node and restart it. Check the Cluster and Manager dashboard. Check cluster status with nodetool as you did before.

You can also load data using cassandra-stress, and watch the Monitoring Dashboard.

Generate an Alert

Generate an Alert, and watch it in the dashboard (for example storage alert)

Backup and Repair

Run an ad-hoc repair from ScyllaDB Manager, watch the progress with sctool and Manager dashboard.