Join us for Scylla University LIVE, 28th of July
Learn More >

Monitoring and Manager Lab

18 min to complete

Monitoring and Manager Lab

This lab is based on Manager version 2.6. In this lab, you’ll:

  • Start a three-node Scylla cluster (with a Scylla Manager Agent)
  • Start a MinIO instance used for backups
  • Start Scylla Manager and connect it to the Scylla cluster and to the MinIO instance
  • Set up Scylla Monitoring
  • Simulate different events (node down, database queries) and see what happens in the Monitoring stack
  • Generate an Alert
  • Perform operations such as backup and repair using Scylla Manager

Setting up Scylla Manager

If you haven’t done so yet in previous lessons, download the example from git:

git clone https://github.com/scylladb/scylla-code-samples.git

Go to the directory of the manager example:

cd scylla-code-samples/manager

Next, you’ll use docker-compose to create:

  • A three-node Scylla cluster
  • A single Scylla node used as the Scylla Manager backend. Scylla Manager uses this database to save its data. 
  • Scylla Manager (with the Agent installed).
  • A MinIO instance used for backups.

Keep in mind that this example is not a production setup. Make sure to read the documentation before running it in production. 

docker-compose build
docker-compose up -d

This might download and update some packages. Wait until the Scylla cluster is up and running (UN status). It may take several minutes. Check the status with:

docker-compose exec scylla-node1 nodetool status

Add the cluster to Scylla Manager:

docker-compose exec scylla-manager sctool cluster add --name test --host=scylla-node1 --auth-token=token

Check the cluster status in Scylla Manager:

docker-compose exec scylla-manager sctool status -c test

To run a backup to MinIO:

docker-compose exec scylla-manager sctool backup -c test --location s3:docker

Check the backup status:

docker-compose exec scylla-manager sctool task list

Setup Scylla Monitoring

For more information about setting up Monitoring, complete this lesson first.

Next, we’ll set up Scylla Monitoring and connect it to the Manager.

cd ../..
git clone https://github.com/scylladb/scylla-grafana-monitoring.git
cd scylla-grafana-monitoring

Extract the IP address of Scylla and Manager for the next step

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' scylla-node1 scylla-node2 scylla-node3
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' manager_scylla-manager_1

Rename the file prometheus/scylla_servers.example.yml to prometheus/scylla_servers.yml and edit it according to your node IP addresses as follows:

Rename the file prometheus/scylla_manager_servers.example.yml to prometheus/scylla_manager_servers.yml and modify it according to your node IP addresses as follows:

Generate the dashboards:

./generate-dashboards.sh -F -v 2021.1

Finally, start the stack. Replace manager_public below with your docker network.

./start-all.sh -v 2021.1 -M 2.0 -D "--net=manager_public"

Next, browse to http://localhost:3000 to view the Monitoring homepage.

Simulate Different Events

Stop the Scylla node and restart it. Check the Cluster and Manager dashboard. Check cluster status with nodetool as you did before.

You can also load data using cassandra-stress, and watch the Monitoring Dashboard.

Generate an Alert

Generate an Alert, and watch it in the dashboard (for example storage alert)

Backup and Repair

Run an ad-hoc repair from Scylla Manager, watch the progress with sctool and Manager dashboard.

fa-angle-up