Getting Started with Trilio on VMware Tanzu Kubernetes Grid (TKG)

Learn to install, license, and test Trilio on VMware Tanzu Kubernetes Grid (TKG)

What is Trilio for Kubernetes?

Trilio for Kubernetes is a cloud-native backup and restore application. Being a cloud-native application for Kubernetes, all operations are managed with CRDs (Customer Resource Definitions).

Trilio utilizes Control Plane and Data Plane controllers to carry out the backup and restore operations defined by the associated CRDs. When a CRD is created or modified the controller reconciles the definitions to the cluster.

Trilio gives you the power and flexibility to backup your entire cluster or select a specific namespace(s), label, Helm chart, or Operator as the scope for your backup operations.

In this tutorial, we'll show you how to install and test operation of Trilio for Kubernetes on your VMware Tanzu Kubernetes Grid deployment.

Prerequisites

Confirm Compatibility

Before installing Trilio for Kubernetes, please review the compatibility matrix to ensure Trilio can function smoothly in your Kubernetes environment.

Verify that a CSI Driver which provides Snapshot functionality

Trilio for Kubernetes requires a compatible Container Storage Interface (CSI) driver that provides the Snapshot feature.

Check the Kubernetes CSI Developer Documentation to select a driver appropriate for your backend storage solution. See the selected CSI driver's documentation for details on the installation of the driver in your cluster.

Trilio will assume that the selected storage driver is a supported CSI driver when the volumesnapshotclass and storageclassare utilized.

Verify that the Required Custom Resource Definitions (CRD) are installed

Trilio for Kubernetes requires the following Custom Resource Definitions (CRD) to be installed on your cluster:VolumeSnapshot, VolumeSnapshotContent, and VolumeSnapshotClass.

Installing the Required VolumeSnapshot CRDs

Before attempting to install the VolumeSnapshot CRDs, it is important to confirm that the CRDs are not already present on the system.

To do this, run the following command:

kubectl api-resources | grep volumesnapshot

If CRDs are already present, the output should be similar to the output displayed below. The second column displays the version of the CRD installed (v1beta1 in this case). Ensure that it is the correct version required by the CSI driver being used.

volumesnapshotclasses                 vsclass,vsclasses   snapshot.storage.k8s.io/v1                    false        VolumeSnapshotClass
volumesnapshotcontents                vsc,vscs            snapshot.storage.k8s.io/v1                    false        VolumeSnapshotContent
volumesnapshots                       vs                  snapshot.storage.k8s.io/v1                    true         VolumeSnapshot

Installing CRDs

Be sure to only install one version of VolumeSnapshot CRDs

Read the external-snapshotter GitHub project documentation. This is compatible with both v1.19 and v1.20+.
Run the following commands to install directly, check the repo for the latest version:

RELEASE_VERSION=6.0

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-${RELEASE_VERSION}/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-${RELEASE_VERSION}/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-${RELEASE_VERSION}/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml

Network Access Requirements

For non-air-gapped environments, the following URLs must be accessed from your Kubernetes cluster.

Charts.bitnami.com
Docker.io
charts.helm.sh/stable/
gcr.io
kubernetes.io
quay.io
github.com
raw.githubusercontent.com
Access to the S3 endpoint if the backup target happens to be S3
Access to application artifacts registry for image backup/restore

Network Port Requirements

If the Kubernetes cluster's control plane and worker nodes are separated by a firewall, then the firewall must allow traffic on the following port(s)

9443

Verify Prerequisites with the Trilio Preflight Check

Make sure your cluster is ready to Install Trilio for Kubernetes by installing the Preflight Check Plugin and running the Trilio Preflight Check.

Trilio provides a preflight check tool that allows customers to validate their environment for Trilio installation.

The tool generates a report detailing all the requirements and whether they are met or not.

If you encounter any failures, please send the Preflight Check output to your Trilio Professional Services and Solutions Architect so we may assist you in satisfying any missing requirements before proceeding with the installation.

Two options for installation method:

Trilio for Kubernetes is available as a certified Backup and Recovery solution in the VMware Marketplace for VMware Tanzu Kubernetes Grid (TKG) environments, however currently there is no option to install T4K onto the Tanzu clusters directly from the VMware Marketplace.

Use an existing helm repository
Use the standard helm install command as per the upstream Kubernetes method

Helm Installation (Method 1)

Users can download the T4K helm chart .tgz file, add it to their existing helm repository and then install the T4K chart from the same repository. To install using this method, perform the following steps:

Log in to VMware Marketplace.
From the page displayed, in the version dropdown, select the version of T4K that you wish to install. If you are free to choose, it is recommended to use the latest version.
Click Deploy.
In the Deploy wizard displayed, confirm that the version is correct.
Click Next.
After reading the Trilio End User License Agreement, place a check/tick in the box to indicate your agreement with the terms.
Click Next.
Click Download to download the .tgz file.
Click Close to Exit the Deploy wizard.
Now add the downloaded .tgz helm chart file to your existing helm repository.
Then use the helm install command to pull the image from your existing helm repository.
Perform steps to prepare environment for Trilio installation

Configure Antrea Networking

Set Antrea NetworkPolicyStats totrue. To accomplish this, change the cluster context to management and run the following command:

antrea_config_values=$(kubectl get secret/antrea-data-values \
                      --namespace tkg-system \
                      -o json | \
                      jq -r '.data["values.yaml"]' | \
                      base64 --decode | \
                      yq e '.antrea.config.featureGates.NetworkPolicyStats = true' - | \
                      base64 -w 0)
                      
kubectl patch secret/antrea-data-values \
                      --dry-run=client \
                      --namespace tkg-system  \
                      --type merge \
                      -p "{\"data\":{\"values.yaml\": \"$antrea_config_values\"}}" \
                      -o yaml > /tmp/antrea-config.yaml
                      
kapp deploy --yes -n tkg-system -a antrea -f /tmp/antrea-config.yaml

On the workload cluster, restart the Antrea controller and agent with these commands:

kubectl rollout restart deployment antrea-controller -n kube-system
kubectl rollout restart daemonset antrea-agent -n kube-system

Install VMware CSI Driver

First, disable automatic updates for the CSI driver by changing the kubeconfig context to the management cluster and running the following command:

kubectl patch secret/<cluster-name>-vsphere-csi-addon \
     -p '{"metadata":{"annotations":{"tkg.tanzu.vmware.com/addon-paused": ""}}}' \
     --type=merge

Next, get a copy of the Vsphere Config secret and save it to a YAML file:

kubectl get secrets vsphere-config-secret -o yaml > vsphere-config-secret.yaml

Then, remove the vsphere-csi package on the workload cluster:

kubectl delete packageinstalls -n tkg-system vsphere-csi

If the secret has been removed by the package uninstall, recreate it with:

kubectl -n kube-system apply -f vsphere-config-secret.yaml

Download the CSI driver YAML:

wget https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/v2.5.0-rc.5/manifests/vanilla/vsphere-csi-driver.yaml

If preferred, replace the namespace in this file from vmware-system-csi to kube-system to run the driver in the kube-system namespace.

Apply the driver on the workload cluster:

kubectl apply -f  vsphere-csi-driver.yaml

Enable the volume snapshot feature on the workload cluster:

kubectl patch configmap/internal-feature-states.csi.vsphere.vmware.com \
-n  kube-system \
--type merge \
-p '{"data":{"block-volume-snapshot":"true"}}'

Download the CSI Snapshot controller install script:

wget https://github.com/kubernetes-sigs/vsphere-csi-driver/blob/v2.5.0-rc.5/manifests/vanilla/deploy-csi-snapshot-components.sh

If preferred, replace the namespace in this file from vmware-system-csi to kube-system to run the driver in the kube-system namespace.

Run the script on the workload cluster:

bash  deploy-csi-snapshot-components.sh

After you have installed the CSI driver, a snapshot class must be created.

Install Trilio using Helm Charts

Start by creating a new namespace on the workload cluster for Trilio, and switch to that new namespace. Run the following command to add Trilio's helm repository and install the Trilio Vault Operator:

helm repo add trilio-vault-operator https://charts.k8strilio.net/trilio-stable/k8s-triliovault-operator
helm install tvm trilio-vault-operator/k8s-triliovault-operator

Upstream Kubernetes Installation (Method 2)

Install the VolumeSnapshot CRDs onto the TKG cluster before performing the T4K installation, by following the steps outlined in the prerequisite section for the Installation of VolumeSnapshot CRDs. VolumeSnapshot CRDs are not shipped with the TKG cluster by default, so users must install them onto the TKG cluster before performing the T4K installation. Failure to perform this step first may cause one or more pods not to run successfully.
The installation of Trilio for Kubernetes on the TKGm (Tanzu Kubernetes Grid for Multi-cloud) or TKGs (Tanzu Kubernetes Grid Service for vSphere) is the same as Upstream Kubernetes, so follow the upstream Helm Quickstart Installation guide.
Users must follow the instructions to install the VMware vSphere CSI Driver from the official documentation. T4K requires a CSI driver with volume snapshot capability to enable it to perform the backup operation. The VMware vSphere CSI Driver with v2.5 and above supports the volume snapshot capability.

Potential Issues and Workarounds

Operator Pod Not Starting

There may be an initial failure during the operator installation due to the Operator pod not starting as the ReplicaSet (RS) couldn't be created. This may be due to an issue with the Pod Security Policy (PSP).

Please define a ClusterRole as shown below to resolve this issue:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: psp:privileged
rules:
- apiGroups: ['policy']
  resources: ['podsecuritypolicies']
  verbs:     ['use']
  resourceNames:
  - vmware-system-privileged
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: all:psp:privileged
roleRef:
  kind: ClusterRole
  name: psp:privileged
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  name: system:serviceaccounts
  apiGroup: rbac.authorization.k8s.io

Antrea Feature Gates

Following the resolution of the PSP issue, it was necessary to enable NetworkPolicyStats from the antrea-agent configuration.

Here are the necessary steps:

Edit the configmap antrea-config-c2g88k9fbh in the kube-system namespace using the following command:

kubectl edit configmap antrea-config-c2g88k9fbh --namespace kube-system

Within the antrea-controller.conf, set antrea.config.featureGates.NetworkPolicyStats = true.
Also within antrea-controller.conf, set antrea.config.featureGates.AntreaPolicy = true.

Please refer to the attachments for more information.

Afterwards, run the following commands to restart deployment and daemonset:

kubectl rollout restart deployment antrea-controller -n kube-system
kubectl rollout restart daemonset antrea-agent -n kube-system

Trilio Manager Service Type

Depending upon the network architecture of your cluster, you may need to change the TVM service.type to LoadBalancer in order to obtain the EXTERNAL-IP for the ingress service, which will then allow you to access the TVK user interface.

Access Trilio UI

Independent of which install method you employ, when the installation is complete and all pods are up and running, you can access the Trilio web UI by pointing your browser to the configured Node Port.

Licensing Trilio for Kubernetes

To generate and apply the Trilio license, perform the following steps:

You must have your kube-system UUID available before generating a license. This can be achieved as follows:

kubectl get ns kube-system -o jsonpath='{.metadata.uid}'

Though a cluster license enables Trilio features across all namespaces in a cluster, the license should only be created in the Trilio install namespace.

2. A license file must be generated for your specific environment.

a) Navigate to your Trilio Welcome email.

b) Click on the License link.

c) On the Trilio for Kubernetes License page, choose the Clustered scope.

d) Provide the kube-system Namespace UUID obtained in Step 1.

e) Click Generate License.

f) On the details confirmation page, copy or download the highlighted text to a file.

You can use the download button to save the highlighted text as a local file or use the copy button to copy the text and create your file manually.

3. Once the license file has been created, apply it to a Trilio instance using the command line or UI:

Execute the following command:

kubectl apply -f <licensefile> -n trilio-system

2. If the previous step is successful, check that output generated is similar to the following:

NAMESPACE            NAME         STATUS   MESSAGE                                   CURRENT CPU COUNT   GRACE PERIOD END TIME   EDITION     CAPACITY   EXPIRATION TIME        MAX CPUS
trilio-system     license-sample   Active   Cluster License Activated successfully.   4                                           FreeTrial   10         2025-07-08T00:00:00Z   8

Additional license details can be obtained using the following:

kubectl get license -o json -m trilio-system

Upgrading a license

A license upgrade is required when moving from one license type to another.

Trilio maintains only one instance of a license for every installation of Trilio for Kubernetes.

To upgrade a license, run kubectl apply -f <licensefile> -n <install-namespace> against a new license file to activate it. The previous license will be replaced automatically.

Learn more about Licensing Trilio for Kubernetes

Create a Backup Target

The Target CR (Customer Resource) is defined from the Trilio Management Console or from your own self-prepared YAML.

The Target object references the NFS or S3 backup storage share you provide as a target for your backups. Trilio will create a validation pod in the corresponding namespace and attempt to validate the NFS or S3 settings you have defined in the Target CR.

Trilio makes it easy to automatically create your backup Target CRD from the Management Console.

Learn how to Create a Target from the Management Console

Take control of Trilio and define your own self-prepared YAML and apply it to the cluster using the kubectl tool.

Example S3 Target

kubectl apply -f sample-secret.yaml

kubectl apply -f demo-s3-target.yaml

apiVersion: v1
kind: Secret
metadata:
  name: sample-secret
type: Opaque
stringData:
  accessKey: AKIAS5B35DGFSTY7T55D
  secretKey: xWBupfGvkgkhaH8ansJU1wRhFoGoWFPmhXD6/vVDcode

apiVersion: triliovault.trilio.io/v1
kind: Target
metadata:
  name: demo-s3-target
spec:
  type: ObjectStore
  vendor: AWS
  objectStoreCredentials:
    region: us-east-1
    bucketName: trilio-browser-test
    credentialSecret:
      name: sample-secret
      namespace: TARGET_NAMESPACE
  thresholdCapacity: 5Gi

See more Example Target YAML

Testing Backup and Restore Operation

Trilio is a cloud-native application for Kubernetes, therefore all operations are managed with CRDs (Custom Resource Definitions). We will discuss the purpose of each Trilio CRD and provide examples of how to create these objects Automatically in the Trilio Management Console or from the kubectl tool.

About Backup Plans and Backups

The Backup Plan CR is defined from the Trilio Management Console or from your own self-prepared YAML.

The Backup Plan CR must reference the following:

Your Application Data (label/helm/operator)
Backup Target CR
Scheduling Policy CR
Retention Policy CR

A Target CR is defined from the Trilio Management Console or from your own self-prepared YAML. Trilio will test the backup target to insure it is reachable and writable. Look at Trilio validation pod logs to troubleshoot any backup target creation issues.
Retention and Schedule Policy CRs are defined from the Trilio Management Console or from your own self-prepared YAML.
- Scheduling Policies allow users to automate the backup of Kubernetes applications on a periodic basis. With this feature, users can create a scheduling policy that includes multiple cron strings to specify the frequency of backups.
- Retention Policies make it easy for users to define the number of backups they want to retain and the rate at which old backups should be deleted. With the retention policy CR, users can use a simple YAML specification to define the number of backups to retain in terms of days, weeks, months, years, or the latest backup. This provides a flexible and customizable way to manage your backup retention policy and ensure you meet your compliance requirements.
The Backup CR is defined from the Trilio Management Console or from your own self-prepared YAML.
The backup object references the actual backup Trilio creates on the Target. The backup is taken as either a Full or Incremental backup as defined by the user in the Backup CR.

Creating a Backup Plan and Backup

Trilio makes it easy to automatically create your backup plans and all required target and policy CRDs from the Management Console.

Learn more about Creating Backups from the Management Console

Take control of Trilio, define your self-prepared YAML, and apply it to the cluster using the kubectl tool.

Example Namespace Scope BackupPlan:

# kubectl apply -f ns-backupplan.yaml

apiVersion: triliovault.trilio.io/v1
kind: BackupPlan
metadata:
  name: ns-backupplan
spec:
  backupConfig:
    target:
      namespace: default
      name: demo-s3-target

kubectl apply -f sample-schedule.yaml

kind: "Policy"
apiVersion: "triliovault.trilio.io/v1"
metadata:
  name: "sample-schedule"
spec:
  type: "Schedule"
  scheduleConfig:
    schedule:
      - "0 0 * * *"
      - "0 */1 * * *"
      - "0 0 * * 0"
      - "0 0 1 * *"
      - "0 0 1 1 *"

kubectl apply -f sample-retention.yaml

apiVersion: triliovault.trilio.io/v1
kind: Policy
metadata:
  name: sample-retention
spec:
  type: Retention
  default: false
  retentionConfig:
    latest: 2
    weekly: 1
    dayOfWeek: Wednesday
    monthly: 1
    dateOfMonth: 15
    monthOfYear: March
    yearly: 1

kubectl apply -f sample-backup.yaml

apiVersion: triliovault.trilio.io/v1
kind: Backup
metadata:
  name: sample-backup
spec:
  type: Full
  backupPlan:
    name: sample-application
    namespace: default

See more Examples of Backup Plan YAML

About Restores

A Restore CR (Custom Resource) is defined from the Trilio Management Console or from your own self-prepared YAML. The Restore CR references a backup object which has been created previously from a Backup CR.

In a Migration scenario, the location of the backup should be specified within the desired target as there will be no Backup CR defining the location.

Trilio restores the backup into a specified namespace and upon completion of the restore operation, the application is ready to be used on the cluster.

Creating a Restore

Trilio makes it easy to automatically create your Restore CRDs from the Management Console.

Learn more about Creating Restores from the Management Console

Take control of Trilio, define your self-prepared YAML, and apply it to the cluster using the kubectl tool.

kubectl apply -f sample-restore.yaml

apiVersion: triliovault.trilio.io/v1
kind: Restore
metadata:
  name: sample-restore
spec:
  source:
    type: Backup
    backup:
      name: sample-backup
      namespace: default