K8s Operator Lab

22 Min to complete

K8s Operator Lab

Introduction

In this lab, you will create a Kubernetes cluster using the Google Kubernetes Engine, and you will deploy the following Scylla products using the Scylla Operator:

  • A three-node Scylla cluster
  • The Scylla Manager
  • The Scylla Monitoring stack
  • And finally, two simple applications using CQL and the DynamoDB API.

Sounds hard? Not when using Kubernetes, so let’s get to it.

prerequisites

  1. Make sure you have the Google Cloud SDK installed. It includes the gcloud command-line tool. To do so, follow the instructions.
  2. Make sure you have the account and project properties set.
  3. Make sure you have kubectl installed.
  4. Make sure Helm is installed.
  5. Ensure that you have enabled the Google Kubernetes Engine API.

Walkthrough

If you haven’t done so yet, start by downloading the example from git:

git clone https://github.com/scylladb/scylla-code-samples.git

Next, spin up a GKE cluster.

Set the GCP user and project environment variables based on your current gcloud configuration:

export GCP_USER=$(gcloud config list account --format "value(core.account)")   
export GCP_PROJECT=$(gcloud config list project --format "value(core.project)") 

Next, start the script to create a GKE cluster which you’ll use throughout this demo. This script creates the K8s cluster in two availability zones: us-west-1b and us-west-1c.
Each consists of:

  • Three nodes (n1-standard-4) dedicated to Scylla
  • A single node (n1-standard-8) dedicated for utilities like Grafana, Prometheus, Scylla Operator, etc.

Make sure you are in the directory: scylla-code-samples/kubernetes-university-live then:

./gke.sh --gcp-user "$GCP_USER" -p "$GCP_PROJECT" -c "university-live-demo"

Next, install Cert Manager, which is a Scylla Operator dependency.

As the name suggests, the Cert Manager is responsible for issuing TLS certificates. It’s a Kubernetes add-on that automates the management and issuance of certificates. The Operator Validating Webhook validates User-provided ScyllaCluster resources. On every creation and update of a ScyllaCluster resource, the Operator Validating Webhook is contacted by the K8s api-server to check whether it passes validation. Communication is done using the TLS protocol, and that’s why the Operator requires a certificate that the Cert Manager provides.

kubectl apply -f cert-manager.yaml 

To wait and see if the installation is complete, check whether Cert Manager Custom Resource Definitions (CRDs) enter an established state, which means every K8s api-server accepted them in the K8s cluster, and until all required Cert Manager Pods are ready to use:

kubectl wait --for condition=established crd/certificates.cert-manager.io crd/issuers.cert-manager.io
kubectl -n cert-manager rollout status --timeout=5m deployment.apps/cert-manager-cainjector
kubectl -n cert-manager rollout status --timeout=5m deployment.apps/cert-manager-webhook

Now, install the Scylla Operator:

 kubectl apply -f operator.yaml

Again, to make sure the installation is complete and ready to go,  wait for the CRDs, and make sure all the Scylla Operator Pods are ready:

kubectl wait --for condition=established crd/scyllaclusters.scylla.scylladb.com
kubectl wait --for condition=established crd/nodeconfigs.scylla.scylladb.com
kubectl wait --for condition=established crd/scyllaoperatorconfigs.scylla.scylladb.com
kubectl -n scylla-operator rollout status deployment.apps/scylla-operator
kubectl -n scylla-operator rollout status deployment.apps/webhook-server

Now, you’ll install the monitoring stack. It will give you observability into what is going on within the Scylla cluster, whether nodes are healthy, and provide you with query statistics. Start by adding the Prometheus Community Helm repository to the local registry:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Then, update the local registry:

helm repo update

And install the monitoring stack (Grafana, Prometheus):

helm install monitoring prometheus-community/kube-prometheus-stack --values monitoring/values.yaml --values monitoring.yaml --create-namespace --namespace scylla-monitoring

Next, you’ll install the Scylla Monitoring dashboards. Download and extract them:

wget https://github.com/scylladb/scylla-monitoring/archive/scylla-monitoring-3.6.0.tar.gz
tar -xvf scylla-monitoring-3.6.0.tar.gz

The Scylla dashboards provide several views of the node, as well as query statistics. These range from simple ones like node health and workflow latency to more detailed ones. Import the Scylla dashboards to Grafana:

kubectl -n scylla-monitoring create configmap scylla-dashboards --from-file=scylla-monitoring-scylla-monitoring-3.6.0/grafana/build/ver_4.3
kubectl -n scylla-monitoring patch configmap scylla-dashboards  -p '{"metadata":{"labels":{"grafana_dashboard": "1"}}}'

The Scylla Manager dashboards provide an overview of past and ongoing tasks like repairs and backups as well as different statistics. To import the Scylla Manager dashboard to Grafana:

kubectl -n scylla-monitoring create configmap scylla-manager-dashboards --from-file=scylla-monitoring-scylla-monitoring-3.6.0/grafana/build/manager_2.2
kubectl -n scylla-monitoring patch configmap scylla-manager-dashboards -p '{"metadata":{"labels":{"grafana_dashboard": "1"}}}'

Now, install the Scylla Manager, which can be used for repairs and backups:

kubectl apply -f manager.yaml

Wait until the installation is complete by waiting until all required Pods are ready:

kubectl -n scylla-manager rollout status statefulset.apps/scylla-manager-cluster-manager-dc-manager-rack
kubectl -n scylla-manager rollout status deployment.apps/scylla-manager-controller

This lab does not take a deep dive into Scylla Manager, you can learn more about it in this lesson.

Now, install Scylla and the Scylla Manager ServiceMonitors. ServiceMonitors are used to tell Prometheus from where it should scrape the metrics. Both Scylla and Scylla Manager expose metrics under a specific port, and these ServiceMonitors contain this information.
Create the `scylla` namespace where you’ll install the Scylla ServiceMonitor as well as the ScyllaCluster,

kubectl apply -f namespace.yaml

Install the Scylla and the Scylla Manager Service Monitors.

kubectl apply -f monitoring/scylla-service-monitor.yaml
kubectl apply -f monitoring/scylla-manager-service-monitor.yaml

Next, set up access to Grafana. Create a tunnel to port 3000 inside the Grafana pod. This command will block until the tunnel connection is established:

kubectl -n scylla-monitoring port-forward deployment.apps/monitoring-grafana 3000

To access Grafana browse to http://127.0.0.1:3000/.

Use the default user/password: admin/admin.

Next,  in a new terminal tab, you’ll deploy a small cluster to benchmark the disks. You will use the results later to speed up the bootstrap.

Deploy our example Scylla cluster:

kubectl apply -f cluster.yaml

Wait until Scylla boots up and reports readiness to serve traffic.

kubectl -n scylla rollout status statefulset.apps/scylla-cluster-us-west1-us-west1-b

Next, save the IO benchmark results by saving the result file content from Scylla Pod to ConfigMap.

Start by saving the benchmark result to the `io_properties.yaml` file:

kubectl -n scylla exec -ti scylla-cluster-us-west1-us-west1-b-0 -c scylla -- cat /etc/scylla.d/io_properties.yaml > io_properties.yaml

Then, create a ConfigMap from the `io_properties.yaml` file:

kubectl -n scylla create configmap ioproperties --from-file io_properties.yaml

Remove the previous Scylla cluster and clear the PersistentVolumeClaims (PVC) to release and erase the disk:

kubectl -n scylla delete ScyllaCluster scylla-cluster
kubectl -n scylla delete pvc --all

Now, you’ll deploy the IO-tuned Scylla cluster. Take a look at the definition in the file. IO Setup was disabled via additional ScyllaArgs, and the previously created ConfigMap is attached as a Volume.

Start by deploying the cluster and waiting until both racks are ready. This time Scylla is using a precomputed disk benchmark, so each Pod should be ready faster:

kubectl apply -f cluster_tuned.yaml
kubectl -n scylla rollout status statefulset.apps/scylla-cluster-us-west1-us-west1-b
kubectl -n scylla rollout status statefulset.apps/scylla-cluster-us-west1-us-west1-c

Next, you are going to deploy an application that sends CQL traffic to Scylla and observe what changes in the Monitoring dashboards. Create a K8s Job to execute cassandra-stress and generate CQL traffic:

kubectl apply -f cassandra-stress.yaml

Browse to Grafana (http://127.0.0.1:3000) to see metrics from the workload. The default login is admin/admin.

Click on Home, then select the Overview dashboard. You should see an increase in the load and the number of requests. You can find a summary of valuable statistics at the top of the screen. These include node health, latencies, load, requests, timeouts, etc. For more advanced users, more detailed dashboards are available in the left panel.

 

Once you are done, clean up the Scylla cluster, clear the disks and remove the cassnadra-stress jobs:

kubectl -n scylla delete ScyllaCluster scylla-cluster
kubectl -n scylla delete pvc --all
kubectl delete -f cassandra-stress.yaml

Next, deploy the Alternator cluster. The difference between this and the previous one is an additional field in the ScyllaCluster spec, called `alternator.` You can set which port will expose the DynamoDB API (also called Alternator) as well as the default consistency level. You can learn more about Alternator here

kubectl apply -f alternator.yaml
kubectl -n scylla rollout status statefulset.apps/scylla-cluster-us-west1-us-west1-b
kubectl -n scylla rollout status statefulset.apps/scylla-cluster-us-west1-us-west1-c

Now, you will run the TicTacToe game.

Amazon wrote this game as an example application for DynamoDB. Because the Alternator API is compatible with DynamoDB, you can run the same application without any changes, just pointing it to a different database. It won’t even notice that it’s not talking to DynamoDB:

kubectl apply -f tictactoe.yaml
kubectl -n tictactoe get svc

Check the external IP address of the game. It may take some time before it is assigned. If it isn’t assigned yet, rerun the above command.

Once ready, access the game using the external IP address.

To play the game, enter your name and click Login. Then, invite a friend (or play against yourself). Once that friend logs in, you can start playing.

Enjoy the game!

Summary

In this lesson, you learned how to install several Scylla products on Kubernetes, including installing the Scylla Operator for Kubernetes and its dependencies. You deployed a highly available three-node Scylla cluster with two supported APIs – CQL and Alternator. You also learned how to deploy applications using both of these APIs, and you saw how to run a distributed TicTacToe game.

 

fa-angle-up