Search…
Optimize TVK Backups with StormForge
This article helps users to install and configure StormForge. It can be used find the optimized resource configuration of TVK to run the backup

Introduction

Today many enterprises running Kubernetes applications are not aware of a way to monitor and optimize the resources used by those applications.
StormForge is a platform that helps users to automate resource tuning and deliver the best Kubernetes application performance at the lowest possible cost. StormForge allows users to run an experiment with a user specified number of trials. The trial runs on an application to find the optimized resource configuration required to perform certain operations. It uses machine learning algorithms to pick the combination of CPU, RAM and other resources to run a trial.

Install and configure StormForge with TVK

Follow below instructions to install and configure the StormForge with TrilioVault for Kubernetes and run the experiment to find the optimized resource configuration of TVK

Install and Configure StormForge redsky-controller-manager with K8s Cluster

  1. 1.
    Download and install redskyctl CLI
1
curl -L https://app.stormforge.io/downloads/redskyctl-linux-amd64.tar.gz | tar x
2
sudo mv redskyctl /usr/local/bin/
Copied!
2. Authorize the linux server from where redskyctl will be evoked to run the experiment
1
redskyctl login --force
Copied!
Enter the URL generated by above command into the browser to authorize the Linux server
1
https://auth.carbonrelay.io/authorize?audience=https%3A%2F%2Fapi.carbonrelay.io%2Fv1%2F&client_id=pE3kMKdrMTdW4DOxQHesyAuFGNOWaEke&code_challenge=nUaqSbjv0grweFAMrv_Zk9KBg_8o-4AQAnwxpijJlGk&code_challenge_method=S256&redirect_uri=http%3A%2F%2F127.0.0.1%3A8085%2F&response_type=code&scope=register%3Aclients+offline_access&state=nHGW89zTYO39loEALGTsig
Copied!
3. Verify if you are connected to the server where you want to run the experiment
1
kubectl get nodes
Copied!
1
NAME STATUS ROLES AGE VERSION
2
ip-mytesthost-w1.us-east-2.compute.internal Ready worker 4d5h v1.19.9
3
ip-mytesthost-w2.us-east-2.compute.internal Ready worker 4d5h v1.19.9
4
ip-mytesthost-m1.us-east-2.compute.internal Ready controlplane,etcd 4d5h v1.19.9
5
ip-mytesthost-m2.us-east-2.compute.internal Ready controlplane,etcd 4d5h v1.19.9
6
ip-mytesthost-w3.us-east-2.compute.internal Ready worker 4d5h v1.19.9
7
ip-mytesthost-m3.us-east-2.compute.internal Ready controlplane,etcd 4d5h v1.19.9
Copied!
4. Initiate the redsky controller manager pod on the kubernetes cluster
1
redskyctl init
Copied!
1
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
2
customresourcedefinition.apiextensions.k8s.io/experiments.redskyops.dev configured
3
customresourcedefinition.apiextensions.k8s.io/trials.redskyops.dev configured
4
clusterrole.rbac.authorization.k8s.io/redsky-manager-role configured
5
clusterrolebinding.rbac.authorization.k8s.io/redsky-manager-rolebinding unchanged
6
namespace/redsky-system unchanged
7
deployment.apps/redsky-controller-manager unchanged
8
clusterrole.rbac.authorization.k8s.io/redsky-patching-role unchanged
9
clusterrolebinding.rbac.authorization.k8s.io/redsky-patching-rolebinding unchanged
10
secret/redsky-manager configured
Copied!
5. Verify if the redsky-controller-manager is running on k8s cluster
1
kubectl get pod -n redsky-system
Copied!
1
NAME READY STATUS RESTARTS AGE
2
redsky-controller-manager-6cbb796b79-5wnz8 1/1 Running 1 4d4h
Copied!
6. Authorize the kubernetes cluster, where user want to run the experiment and application is present
1
redskyctl authorize-cluster
Copied!
1
secret/redsky-manager configured
2
deployment.apps/redsky-controller-manager patched
Copied!

Install Demo Application and Configure TVK Resources

1. There must be an application running in the namespace which will be used to perform the backup by StormForge experiment
1
kubectl get pods | grep k8s-demo-app
Copied!
1
k8s-demo-app-frontend-7c4bdbf9b-bbz2z 1/1 Running 1 4d4h
2
k8s-demo-app-frontend-7c4bdbf9b-c84m7 1/1 Running 1 4d4h
3
k8s-demo-app-frontend-7c4bdbf9b-z8jhx 1/1 Running 1 4d4h
4
k8s-demo-app-mysql-754f46dbd7-v85mh 1/1 Running 2 4d4h
Copied!
2. It is expected that TrilioVault for Kubernetes product is installed on the K8s cluster
1
kubectl get pods | grep k8s-triliovault
Copied!
1
k8s-triliovault-admission-webhook-7c5b454ff4-85t92 1/1 Running 1 4d3h
2
k8s-triliovault-control-plane-54594f5796-vpz4r 2/2 Running 2 4d3h
3
k8s-triliovault-exporter-86d94c9967-bx56s 1/1 Running 1 4d3h
4
k8s-triliovault-ingress-gateway-6bdfc75c9b-l5qvb 1/1 Running 1 4d3h
5
k8s-triliovault-web-76d84fdcc5-lj52v 1/1 Running 1 4d3h
6
k8s-triliovault-web-backend-7fb8784588-8v865 1/1 Running 1 4d3h
7
triliovault-operator-k8s-triliovault-operator-66b6d9d96f-k8fjs 1/1 Running 1 4d4h
Copied!
3. Verify that TVK License is applied to initiate the backup
1
kubectl get license
Copied!
1
NAME STATUS MESSAGE CURRENT NODE COUNT GRACE PERIOD END TIME EDITION CAPACITY EXPIRATION TIME
2
test-license-1 Active Cluster License Activated successfully. 9 Basic 10 2022-04-30T00:0
Copied!
4. Make sure that the Target to store the backup is configured and is in Available state
1
kubectl get target
Copied!
1
NAME TYPE THRESHOLD CAPACITY VENDOR STATUS BROWSING ENABLED
2
demo-s3-target ObjectStore 100Gi AWS Available
Copied!
5. Create a Backup plan for the application or namespace
1
kubectl apply -f app/mysql-sample-backupplan.yaml
2
kubectl apply -f app/mysql-sample-backupplan-1.yaml
Copied!
6. Verify the backup plan is created correctly
1
kubectl get backupplan
Copied!
1
NAME TARGET RETENTION POLICY INCREMENTAL SCHEDULE FULL BACKUP SCHEDULE STATUS
2
mysql-label-backupplan demo-s3-target Available
3
mysql-label-backupplan-1 demo-s3-target Available
Copied!

Configure and Monitor StormForge Experiment

  1. 1.
    Users need to create a backup, bash script to monitor the backup and tvk-manager configmap. Users can also follow the readme.md file present at path
    https://github.com/trilioData/AppTesting/tree/master/stormforge/backup-delete-backup-delete-parllel
1
kubectl create configmap bashscript --from-file=configmaps/bash-script.sh
2
kubectl create configmap bashscript1 --from-file=configmaps/bash-script-1.sh
3
4
kubectl create configmap backup --from-file=configmaps/mysql-sample-backup.yaml
5
kubectl create configmap backup1 --from-file=configmaps/mysql-sample-backup-1.yaml
6
7
kubectl create configmap tvkmanager --from-file=configmaps/tvk-manager.yaml
Copied!
2. User has all required entities in place to start the experiment
1
kubectl apply -f experiment/tvk-scale-2-backup-delete-parallel-prod.yaml
Copied!
3. User can monitor the running experiment being performed for each backup.
1
kubectl get experiment
Copied!
1
NAME STATUS
2
tvk-scale-400-backup-delete-parallel-prod Running
Copied!
4. Check the new Trial running by the experiment
1
kubectl get trial
Copied!
1
NAME STATUS ASSIGNMENTS VALUES
2
tvk-scale-400-backup-delete-parallel-prod-000 Setting up deploymentMemory=512, deploymentCpu=250, metaCpu=500, metaMemory=512, dataMemory=1536, dataCpu=1200
Copied!
5. Verify the jobs
1
kubectl get jobs
Copied!
1
NAME COMPLETIONS DURATION AGE
2
tvk-scale-2-backup-delete-parallel-prod-001 0/1 4d3h 4d3h
3
tvk-scale-2-backup-delete-parallel-prod-001-create 1/1 51s 4d3h
4
tvk-scale-2-backup-delete-parallel-prod-001-delete 1/1 6s 4d3h
Copied!

Monitor the Experiment from StormForge UI

  1. 1.
    After the experiment is in running state, user can view the progress at the StormForge UI using the login they have configured.
StormForge Experiment Trial Run Graph
2. Once the experiment run is complete, StormForge experiment will show the recommended configuration with resource details.
StormForge Recommended Resource Configuration

Delete / Remove StormForge Experiment

  1. 1.
    After the experiment run is complete, user can remove the StormForge redsky-controller-manager with simple commands
  • Delete the experiment from kubernetes cluster along with all other resources
1
kubectl apply -f experiment/tvk-scale-2-backup-delete-parallel-prod.yaml
Copied!
  • Delete the experiment from redsky tenant
1
redskyctl delete exp tvk-scale-2-backup-delete-parallel-prod
Copied!
Note: If you delete the experiment from redsky tenant, it will get deleted from the StormForge UI as well.
2. If user is facing any issues while running the experiment, they can check logs of redsky-controller-manager
1
kubectl logs redsky-controller-manager-6cbb796b79-5wnz8 -n redsky-system
Copied!

Conclusion

Using StormForge, users can create desired experiments. Users run those experiments on Kubernetes cluster with applications running and perform operations like Backup/Restore. StormForge UI will provide the analysis through a well curated graph. It will show the best to worst combination of resources used to perform the operation in a particular trial. These optimized resource combinations can be applied in the production clusters to achieve the desired RPO/RTO.
Last modified 3d ago