Monitoring and Manager (Version 2.0) Lab

18 min to complete

Monitoring and Manager (Version 2.0) Lab

This lab is based on Manager version 2.0. In this lab, we’ll:

  • Start a three-node Scylla cluster (with a Scylla Manager Agent)
  • Start a MinIO instance used for backups
  • Start Scylla Manager and connect it to the Scylla cluster and to the MinIO instance
  • Set up Scylla Monitoring
  • Simulate different events (node down, database queries) and see what happens in the Monitoring stack
  • Generate an Alert
  • Perform operations such as backup and repair using Scylla Manager

Setting up Scylla Manager

If you haven’t done so yet in previous lessons, download the example from git:

git clone https://github.com/scylladb/scylla-code-samples.git

Go to the directory of the manager example:

cd scylla-code-samples/manager

In this example, we will use a MinIO instance for backups. Keep in mind that this example is not a production setup. Make sure to read the documentation before running in production. 

Start MinIO in a container:

docker run -d -p 9000:9000 --name minio1 \
-e "MINIO_ACCESS_KEY=minio" \
-e "MINIO_SECRET_KEY=minio123" \
-v /mnt/data:/data minio/minio:RELEASE.2020-03-14T02-21-58Z server /data

This might download some files if you haven’t used MinIO before.

We’ll create two containers, a scylla-manager container that runs the Scylla Manager Server and scylla-manager-db, a container that runs the Scylla database. Scylla Manager uses this database to save its data. 

The Dockerfile in this directory patches a regular ScyllaDB image with a Scylla Manager Agent to allow for proper communication between ScyllaDB and Scylla Manager.

Execute the build:

docker build -t scylladb/scylla-with-agent .

Again this might download and update some packages.

Create a new ScyllaDB three-node cluster using the image you just built. This instance will hold the data of your application. We need to link it with the MinIO instance to allow the Scylla Manager Agent to access the MinIO instance.

docker run -d --name some-scylla --link minio1 --mount type=volume,source=scylla_db_data,target=/var/lib/scylla scylladb/scylla-with-agent --smp 1 --memory=750M --overprovisioned 1 --api-address 0.0.0.0
SEED=$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' some-scylla)
docker run -d --name some-scylla2 --link minio1 scylladb/scylla-with-agent --smp 1 --memory=750M --overprovisioned 1 --api-address 0.0.0.0 --seeds="$SEED"
docker run -d --name some-scylla3 --link minio1 scylladb/scylla-with-agent --smp 1 --memory=750M --overprovisioned 1 --api-address 0.0.0.0 --seeds="$SEED"

Wait a few seconds and make sure the cluster is up and running:

docker exec -it some-scylla nodetool status

Now you can start a regular ScyllaDB instance that Scylla Manager will use to store its internal data in with the following command.

docker run -d --name scylla-manager-db --mount type=volume,source=scylla_manager_db_data,target=/var/lib/scylla scylladb/scylla --smp 1 --memory=1G

Finally, it’s time to start the Scylla manager using the following command. We need to link this instance to both of the ScyllaDB instances.

docker run -d --name scylla-manager --link scylla-manager-db --link some-scylla scylladb/scylla-manager:2.0.1

Using sctool

Use docker exec to invoke bash in the scylla-manager container to add the one node cluster you created above to Scylla Manager.

docker exec -it scylla-manager sctool cluster add -c cluster --host=some-scylla --auth-token=token

Set the SCYLLA_MANAGER_CLUSTER variable according to the given ID above, for example:

export SCYLLA_MANAGER_CLUSTER=f36e476b-1fdf-4129-8fa2-00a257241249

Check the status:

docker exec -it scylla-manager  sctool status -c $SCYLLA_MANAGER_CLUSTER

Setup Scylla Monitoring

For more information about setting up Monitoring, complete this lesson first.

Next, we’ll set up Scylla Monitoring and connect it to the Manager.

cd ../..
git clone https://github.com/scylladb/scylla-grafana-monitoring.git
cd scylla-grafana-monitoring

Extract the IP address of Scylla and Manager for the next step

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' some-scylla some-scylla2 some-scylla3
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' scylla-manager

Rename the file prometheus/scylla_servers.example.yml to prometheus/scylla_servers.yml and edit it according to your node IP addresses as follows:

Rename the file prometheus/scylla_manager_servers.example.yml to prometheus/scylla_manager_servers.yml and modify it according to your node IP addresses as follows:


Finally, start the stack

./start-all.sh -v 3.2 -M 2.0

Next, browse to http://localhost:3000 to view the Monitoring homepage.

Simulate Different Events

Stop the Scylla node and restart it. Check the Cluster and Manager dashboard. Check cluster status with nodetool.

Load data using cassandra-stress, watch the Monitoring Dashboard.

Generate an Alert

Generate an Alert, and watch it in the dashboard (for example storage alert)

Backup and Repair

Run an ad-hoc repair from Scylla Manager, watch the progress with sctool and Manager dashboard