The high-level process is the same for all Distributions.

1. Uninstall the Horizon Plugin or the Trilio Horizon container

2. Uninstall the datamover-api container

3. Uninstall the datamover container

4. Uninstall the workloadmgr container

Each T4O release includes a set of artifacts such as version-tagged containers, package repositories, and distribution packages.

To help users quickly identify the resources associated with each release, we have added dedicated sub-pages corresponding to a specific release version.

1] Uninstall trilio-openstack helm chart

Run following commands to uninstall trilio services from MOSK cloud.

2] Uninstall Dynamic Backup Targets (Optional)

If you have installed multiple backup targets other than the default one, you can uninstall them using:

3] Verify uninstallation

Verify if the trilio-openstack chart and other components got uninstalled. No objects should be listed in following command output:

4] Clean TrilioVault’s OpenStack resources

5] Unmount the targets from All Worker and Master Nodes

To unmount the backup target mount points from each node, run:

Repeat this on each worker and master node where the target was mounted.

Trilio Data, Inc

Trilio Data, Inc. is the leader in providing backup and recovery solutions for cloud-native applications. Established in 2013, its flagship products, Trilio for OpenStack(T4O) and Trilio for Kubernetes(T4K) are used by a wide number of large corporations around the world.

About Trilio for OpenStack

T4O, by Trilio Data, is a native OpenStack service-based solution that provides policy-based comprehensive backup and recovery for OpenStack workloads. It captures point-in-time workloads (Application, OS, Compute, Network, Configurations, Data, and Metadata of an environment) as full or incremental snapshots. These snapshots can be held in a variety of storage environments including NFS, AWS S3, and other S3-compatible storages. With Trilio and its one-click recovery, organizations can improve Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). With Trilio, IT departments are enabled to fully deploy OpenStack solutions and provide business assurance through enhanced data retention, protection, and integrity.

With the use of Trilio’s VAST (Virtual Snapshot Technology), Enterprise IT and Cloud Service Providers can now deploy backup and disaster recovery as a service to prevent data loss or data corruption through point-in-time snapshots and seamless one-click recovery. Trilio takes point-in-time backup of the entire workload consisting of compute resources, network configurations, and storage data as one unit. It also takes incremental backups that capture only the changes that were made since the last backup. Incremental snapshots save a considerable amount of storage space as the backup only includes changes since the last backup. The benefits of using VAST for backup and restore could be summarized below:

About this Documentation

This documentation serves as the end-user technical resource to accompany Trilio for OpenStack. You will learn about the architecture, installation, and vast number of operations of this product. The intended audience is anyone who wants to understand the value, operations, and nuances of protecting their cloud-native applications with Trilio for OpenStack.

Trilio seamlessly integrates with OpenStack, functioning exclusively through APIs utilizing the OpenStack Endpoints. Furthermore, Trilio establishes its own set of OpenStack endpoints. Additionally, both the Trilio appliance and compute nodes interact with the backup target, impacting the network strategy for a Trilio installation.

Existing endpoints in OpenStack

OpenStack comprises three endpoint groupings:

• Public Endpoints

Public endpoints are meant to be used by the OpenStack end-users to work with OpenStack.

• Internal Endpoints

Internal endpoints are intended to be used by the OpenStack services to communicate with each

• Admin Endpoints

Admin endpoints are meant to be used by OpenStack administrators.

Among these three endpoint categories, it's important to note that the admin endpoint occasionally hosts APIs not accessible through any other type of endpoint.

To learn more about OpenStack endpoints please visit the official OpenStack documentation.

OpenStack endpoints required by Trilio

Trilio communicates with all OpenStack services through a designated endpoint type, determined and configured during the deployment of Trilio's services.

The following network requirements can be identified this way:

• Trilio services need access to the Keystone admin endpoint on the admin endpoint network if it is available.

• Trilio services need access to all endpoints of the set endpoint type during deployment.

Recommendation: Provide access to all OpenStack Endpoint types

Trilio recommends granting comprehensive access to all OpenStack endpoints for all Trilio services, aligning with OpenStack's established standards and best practices.

Additionally, Trilio generates its own endpoints, which are integrated within the same network as other OpenStack API services.

To adhere to OpenStack's prescribed standards and best practices, it's advisable that Trilio containers operate on the same network as other OpenStack containers.

• The public endpoint to be used by OpenStack users when using Trilio CLI or API

• The internal endpoint to communicate with the OpenStack services

• The admin endpoint to use the required admin-only APIs of Keystone

Backup target access required by Trilio

The Trilio solution uses backup target storage to place the backup data securely. Trilio divides its backup data into two parts:

1. Metadata

2. Volume Disk Data

The first type of data is generated by the Trilio Workloadmgr services through communication with the OpenStack Endpoints. All metadata that is stored together with a backup is written by the Trilio Workloadmgr services to the backup target in the JSON format.

The second type of data is generated by the Trilio Datamover service running on the compute nodes. The Datamover service reads the Volume Data from the Cinder or Nova storage and transfers this data as a qcow2 image to the backup target. Each Datamover service is hereby responsible for the VMs running on its compute node.

The network requirements are therefor:

• Every Trilio Workloadmgr service containers need access to the backup target

• Every Trilio Datamover service containers need access to the backup target

Before embarking on the installation process for Trilio in your OpenStack environment, it is highly advisable to carefully consider several key elements. These considerations will not only streamline the installation procedure but also ensure the optimal setup and functionality of Trilio's solutions within your OpenStack infrastructure.

Tenant Quotas

Trilio leverages Cinder snapshots to facilitate the computation of both full and incremental backups.

When executing full backups, Trilio orchestrates the generation of Cinder snapshots for all volumes included in the backup job. These Cinder snapshots remain intact for subsequent incremental backup image calculations.

During incremental backup operations, Trilio generates fresh Cinder snapshots and computes the altered blocks between these new snapshots and the earlier retained snapshots from full or previous backups. The old snapshots are subsequently deleted, while the newly generated snapshots are preserved.

Consequently, it becomes imperative for each tenant benefiting from Trilio's backup functionality to possess adequate Cinder snapshot quotas capable of accommodating these supplementary snapshots. As a guiding principle, it is recommended to append 2 snapshots for each volume incorporated into the backup quotas for the respective tenant. Additionally, a commensurate increase in volume quotas for the tenant is advisable, as Trilio briefly materializes a volume from the snapshot to access data for backup purposes.

During the restoration process, Trilio generates supplementary instances and Cinder volumes. To facilitate seamless restore operations, tenants should maintain adequate quota levels for Nova instances and Cinder volumes. Failure to meet these quota requirements may lead to disruptions in restoration procedures.

AWS S3 Eventual Consistency

The AWS S3 object consistency model includes:

1. Read-after-write

2. Read-after-update

3. Read-after-delete

Each of these models explains how an object becomes consistent after being created, updated, or deleted. None of these methods ensures strong consistency, leading to a delay before an object becomes fully consistent.

Although Trilio has introduced measures to address AWS S3's eventual consistency limitations, the exact time an object achieves consistency cannot be predicted through deterministic means.

There is no official statement from AWS on how long it takes for an object to reach a consistent state. However, read-after-write has a shorter time to reach consistency compared to other IO patterns. Therefore, our solution is designed to maximize the read-after-write IO pattern.

AWS Region Considerations

The time in which an object reaches eventual consistency also depends on the AWS region.

For instance, the AWS-standard region doesn't offer the same level of strong consistency as regions like us-east or us-west. Opting for these regions when setting up S3 buckets for Trilio is advisable. While fully avoiding the read-after-update IO pattern is complex, we've introduced significant access delays for objects to achieve consistency over longer periods. On rare occasions when this does happen, it will cause a backup failure and require a retry.

Trilio Cluster

Trilio can be deployed as a single node or a three-node cluster. It is highly recommended that Trilio is deployed as a three-node cluster for fault tolerance and load balancing. Starting with the 3.0 release, Trilio requires additional IP or FQDN for the cluster and is required for both single-node and three-node deployments. Cluster IP a.k.a virtual IP is used for managing clusters and is used to register the Trilio service endpoint in the keystone service catalog.

After all Trilio components are installed, the license can be applied.

The license can be applied either through the Admin tab in Horizon or the CLI

Apply license through Horizon

To apply for the license through Horizon follow these steps:

1. Login to Horizon using admin user.

2. Click on the Admin Tab.

3. Navigate to Backups-Admin

4. Navigate to Trilio

5. Navigate to License

6. Click "Update License"

7. Read the license agreement

8. Click on "I accept the terms in the License Agreement"

9. Click on "Next"

10. Click "Choose File"

11. Choose license file on the client system

12. Click "Apply"

Apply license through CLI

workloadmgr license-create <license_file>

• <license_file> ➡️ path to the license file

Read and accept the End User License Agreement to complete the license application.

Users can preview the latest EULA at our main site: https://trilio.io/eula/

Ceph is the most common OpenSource solution to provide block storage through OpenStack Cinder.

Ceph is a very flexible solution. The possibilities of Ceph require additional steps to the Trilio solution.

The following versions can be upgraded to each other:

Cloud-Native and Containerized

T4O represents a comprehensively containerized deployment model, eliminating the necessity for any KVM-based appliances for service deployment. This marks a departure from earlier T4O releases, where such appliances were required.

Summary of Requirements

Trilio requires its containers to be deployed on the same plane as OpenStack, utilizing existing cluster resources.

As described in the architecture overview, Trilio requires sufficient cluster resources to deploy its components on both the Controller Plane and Compute Planes.

1. Valid Trilio License & Acceptance of the EULA

2. When deploying the Control Plane to OpenShift, ensure sufficient resources are available on Worker Nodes.

3. Sufficient storage capacity and connectivity on Cinder for snapshotting operations

4. Sufficient network capabilities for efficient data transfer of workloads

5. User and Role permissions for access to required cluster objects

6. Optional features may have specific requirements such as encryption, file search, snapshot mount, FRM instance, etc

7. Set hw_qemu_guest_agent=True property on the image and install qemu-guest-agent on the VM, in order to avoid any file system inconsistencies post restore.

8. For the VMware to OpenStack migration feature, please refer to the prerequisite and limitations pages.

1] Prerequisites

Please ensure the following requirements are met before starting the upgrade process:

• No Snapshot or Restore is running

• Global job scheduler is disabled

• Disable the wlm services on all controller nodes

• Disable the datamover service on all compute nodes

2] Install latest Trilio release

Take a backup of existing triliovault-cfg-scripts directory

Clone triliovault-cfg-scripts github repository of latest release.

Copy the passwords and triliovault wlm ids files from old cfg-scripts directory.

Please follow from section 1.4 to section 10. Please note that sections 5 and 6 needs to be skipped during this process.

While providing the backup target details in section 4.1, please ensure to provide the existing backup target (of 5.x release) as default.

3] Restart wlm-cron resource

From any one of the controller nodes, run the below command.

4] Fetch backup target details

From any one of the controller nodes, login to wlm api container and run the below commands to fetch the backup target details.

This command will return the backup_target_id and backup_target_type_id.

5] Import workloads

From any one of the controller ndoes, login to wlm api container and run the below commands.

6] Update backing file path

Please follow to update the backing file path of existing snapshots.

Trilio employs a base64 hash to establish the mount point for NFS Backup targets, ensuring compatibility across multiple NFS Shares within a single Trilio installation. This hash is an integral component of Trilio's incremental backups, functioning as an absolute path for backing files.

Consequently, during a disaster recovery or rapid migration situation, the utilization of a mount bind becomes necessary.

In scenarios that allow for a comprehensive migration period, an alternative approach comes into play. This involves modifying the backing file, thereby enabling the accessibility of Trilio backups from a different NFS Share. The backing file is updated to correspond with the mount point of the new NFS Share.

Backing file change script

Trilio provides a shell script for the purpose of changing the backing file. This script is used after the Trilio appliance has been reconfigured to use the new NFS share.

Downloading the shell script

Prerequisites

The following requirements need to be met before the change of the backing file can be attempted.

• The Trilio Appliance has been reconfigured with the new NFS Share

• The OpenStack environment has been reconfigured with the new NFS Share

  • Please check for Red Hat OpenStack Platform

• The workloads are available on the new NFS Share

• The workloads are owned by nova:nova user

Usage

The shell script is changing one workload at a time.

Run the following command:

with

• /var/triliovault-mounts/<base64>/ being the new NFS mount path

• workload_<workload_id> being the workload to rebase

Logging of the procedure

The shell script is generating the following log file at the following location:

The log file will not get overwritten when the script is run multiple times. Each run of the script will append the available log file.

1] Configuration change

If any config parameter changed in tvo-operator-inputs.yaml like db user password or service endpoints, you can apply the changes using following command.

Above command will output ‘configured' or 'unchanged’ depending upon changes happened in tvo-operator-inputs.yaml.

2] Upgrade to new build

Please follow below steps to upgrade to new build on RHOSO18 setup.

Take a backup of existing triliovault-cfg-scripts and clone latest triliovault-cfg-scripts github repository.

Manually copy the input values from triliovault-cfg-scripts-old to latest directory.

2.1] Upgrade Trilio for OpenStack Operator

Run operator deployment with new image tag as mentioned in

2.2] Upgrade Trilio OpenStack Control Plane Services

Update the image tags in tvo-operator-inputs.yaml as mentioned in and apply the changes using below command

Verify the deployment status and successful deployment.

2.3] Upgrade Trilio Data Plane Services

Update the image tags in cm-trilio-datamover.yaml and apply the changes as mentioned in

Update the ansible runner tag in trilio-datamover-service.yaml and apply the changes as mentioned in

Update the deployment name and trigger deployment as mentioned in

Verify the deployment as mentioned in

2.4] Upgrade Trilio Horizon Plugin

Follow and update the trilio horizon plugin image tag.

It is possible to configure Cinder and Ceph to use different Ceph users for different Ceph pools and Cinder volume types. Or to have the nova boot volumes and cinder block volumes controlled by different users.

In the case of multiple Ceph users, it is required to delete the keyring extension from the triliovault-datamover.conf inside the Ceph block by following below mentioned steps:

1. Deploy Trilio as per the documented steps.

2. Post successful deployment, please modify the triliovault-datamover.conf file present at following locations on all compute nodes.

   For RHOSP : /var/lib/config-data/puppet-generated/triliovaultdm/etc/triliovault-datamover/

   For Kolla : /etc/kolla/triliovault-datamover/

3. Modify keyring_ext value with valid keyring extension (eg. .keyring). This extension is expected to be same for all the keyring files. It will be present under [ceph] block in triliovault-datamover.conf file.

Sample conf entry below. This will try all files with the extension keyring that are located inside /etc/ceph to access the Ceph cluster for a Trilio related task.

1. Restart triliovault_datamover container on all compute nodes.

Backup-as-a-Service

Trilio is an add-on service to OpenStack cloud infrastructure and provides backup and disaster recovery solutions for tenant workloads. Trilio is very similar to other OpenStack services including Nova, Cinder, Glance, etc., and adheres to all tenets of OpenStack. It is a stateless service that scales with your cloud.

Main Components

Trilio has three main software components which are again segregated into multiple services.

WorkloadManager

This component is registered as a keystone service of type workloads which manages all the workloads being created for utilizing the snapshot and restore functionalities. It has 4 services responsible for managing these workloads, their snapshots, and restores.

1. workloadmgr-api

2. workloadmgr-scheduler

3. workloadmgr-workloads

4. workloadmgr-cron

DataMover

Similar to WorkloadManager, this component registers a keystone service of the type datamover which manages the transfer of extensive data to/from the backup targets. It has 2 services that are responsible for taking care of the data transfer and communication with WorkloadManager

1. datamover-api

2. datamover

Horizon Dashboard Plugin

For ease of access and better user experience, T4O provides an integrated UI with the OpenStack dashboard service Horizon,

1. Trilio API is a Python module that is installed on all OpenStack controller nodes where the nova-api service is running.

2. Trilio Datamover is a Python module that is installed on every OpenStack compute nodes

3. Trilio Horizon plugin is installed as an add-on to Horizon servers. This module is installed on every server that runs Horizon service.

Service Endpoints

Trilio is both a provider and consumer in the OpenStack ecosystem. It uses other OpenStack services such as Nova, Cinder, Glance, Neutron, and Keystone and provides its own services to OpenStack tenants. To accommodate all possible OpenStack deployments, Trilio can be configured to use either public or internal URLs of services. Likewise, Trilio provides its own public, internal, and admin URLs for two of its services WorkloadManager API and Datamover API.

Network Topology

Unlike the previous versions of Trilio for OpenStack, now it utilizes the existing network of the OpenStack deployed environment. The networks for the Trilio services can be configured as per the user's desire in the same way the user configures any other OpenStack service. Additionally, a dedicated network can be provided to Trilio services on both control and compute planes for storing and retrieving backup data from the backup target store.

Starting t4o-6.x, a new feature has been introduced to add new backup target without the need of redeploying complete t4o. Please follow mentioned steps to add a backup target

1. Update backup target specific yaml file

1.1] Navigate to the backup targets chart

cd triliovault-cfg-scripts/openstack-helm/trilio-backup-targets/values_overrides

1.2] Update the <backup-target-type>.yaml file (nfs.yaml, other_s3.yaml or amazon_s3.yaml).

NFS Backup Target Example

trilio_backup_target:
  backup_target_name: 'NFS_BackupTarget'
  backup_target_type: 'nfs'
  is_default: true
  nfs_server: '10.10.0.1'
  nfs_shares: /home/openstack-helm
  nfs_options: "nolock,soft,timeo=600,intr,lookupcache=none,nfsvers=3,retrans=10"
  storage_size: 20Gi
  storage_class_name: nfs

images:
  trilio_backup_targets: docker.io/trilio/trilio-wlm-helm:<image-tag>

S3 Backup Target Example

trilio_backup_target:
  backup_target_name: 'S3_BackupTarget'
  backup_target_type: 's3'
  is_default: true

  # S3 Configuration
  s3_type: 'amazon_s3'           # if not Amazon S3, use 'other_s3'
  s3_access_key: 'ACCESSKEY1'
  s3_secret_key: 'SECRETKEY1'
  s3_region_name: 'REGION1'
  s3_bucket: 'BUCKET1'
  s3_endpoint_url: ''             # required only for 'other_s3'
  s3_signature_version: 'default'
  s3_auth_version: 'DEFAULT'
  s3_ssl_enabled: true
  s3_ssl_verify: true
  s3_ssl_ca_cert: ''              # add CA cert for 'other_s3'

images:
  trilio_backup_targets: docker.io/trilio/trilio-wlm-helm:<image-tag>

2. Install/Upgrade Backup Target

2.1] Helm Install/Upgrade Command

Please ensure that values_overrides/<backup-target-type>.yaml file is updated with correct NFS/S3 backup target configuration.

helm upgrade --install <release-name> ./trilio-backup-targets \
  -n trilio-openstack \
  -f ./trilio-backup-targets/values_overrides/nfs.yaml \
  --wait \
  --timeout 5m

3. Verification

3.1] Verify Backup Target Pods

Check if the backup target pods are running

kubectl get pods -n trilio-openstack -l component=nfs-mount

3.2] Check Mounts (for NFS targets)

Exec into the nfs-mount pod and check if the mount is successful.

kubectl exec -n trilio-openstack -it <nfs-mount-pod-name> -- mount | grep trilio

Mount path like below should be visible

/var/lib/trilio/triliovault-mounts/<base64-nfs-path>

3.3] Verify Persistent Volumes (PV/PVCs)

Check if volumes are properly created:

kubectl get pv | grep trilio-openstack
kubectl get pvc -n trilio-openstack | grep trilio

Required installation/updates are done.

If everything looks good, the NFS/S3 backup target is successfully installed and ready to use.

By default, during a snapshot, Trilio will consider all the instances that are members of a workload and starts uploading the resources of each instance simulateously. In some cases, it may be required that the snapshots must be done in serial per VM such that backup of all the disks of one VM/instance must be finished before taking backup of the other VMs that are part of that workload.

Trilio has a provision to update the default behaviour of considering all the instances of a workload during backup process, by introducing a configrable parameter in the on of its services.

Follow the below mentioned steps to set the serial upload as the default behavior.

Finding the required configuration file

As per the OpenStack distribution, look for the configuration file of workloadmgr-api(wlm-api) and workloadmgr-workloads(wlm-workloads) services or containers on the controller plane.

The default location of this configration file in the container or VM where it is running is /etc/triliovault/triliovault-wlm.conf

Once the file is located, take a backup of the file in-case of future reference.

Updating the configuration file

Once the files are located, add the below mentioned parameter with its value to the [DEFAULT] section of the configuration file. serial_vm_backup=true

After successfully updating the file, restart the services or containers and these steps need to be followed on all the controller plane.

6.1.1

Release Date : 21st May'2025

What's New?

1. Support for T4O on OpenStack Helm.

Known Issues

1. [OpenStack Helm] During T4O deployment, if S3 Immutable bucket is set as the default target, the UI goes into hang state.

   • Workaround:

     • Deploy T4O on OpenStack helm by keeping s3 immutable target as default, then add s3 mutable/NFS as dynamic target. Later, change the s3 mutable(dynamic target) target to default.

2. workload-reassign-workloads with --source-btt-all cli option doesn't work as expected

   • Workaround:

     • User can use --source-btt option to reassign the workloads from provided backup target type.Sample command : workloadmgr workload-reassign-workloads --old_tenant_ids 71637e3b98434ceeb5158087074af4aa --new_tenant_id 8e325e1056c94625854b3dffb4d1e64e --user_id 3a446f94fb57448a8c99c08238df0f13 --migrate_cloud True --source-btt 979a459d-26f9-4aa9-8f0d-21ca7a1f6b4f

6.1.0

Release Date : 10th April'2025

What's New?

1. Support for T4O on RHOSO18.0.

2. Support for Dynamic Backup Target addition without the need of redeployment of T4O.

3. Support for T4O on OpenStack Helm.

Known Issues

1. trilio-openstack-operator-rhoso images are not available on Operator Hub

   • Workaround:

     • To install from cli using install script as mentioned in the deployment doc.

2. workload-reassign-workloads with --source-btt-all cli option doesn't work as expected

   • Workaround:

     • User can use --source-btt option to reassign the workloads from provided backup target type.Sample command : workloadmgr workload-reassign-workloads --old_tenant_ids 71637e3b98434ceeb5158087074af4aa --new_tenant_id 8e325e1056c94625854b3dffb4d1e64e --user_id 3a446f94fb57448a8c99c08238df0f13 --migrate_cloud True --source-btt 979a459d-26f9-4aa9-8f0d-21ca7a1f6b4f

6.0.0

Release Date : 18th Jan'2025

What's New?

1. Deployment of Trilio on RHOCP (with RHOSP17.1).

2. Support for multiple target backends.

3. Support for advanced scheduling of workloads' snapshots.

4. Support for Immutable Backups on S3 target backend.

5. Support for Canonical Yoga, 2023.1(Antelope) & 2023.2(Bobcat) on Jammy.

Known Issues

1. T4O upgrade on Canonical OpenStack is not supported

2. Restore operations fail for image-booted instance after changing the image from public to private

   • Workaround:

     • Make the image public or accessible to the project/user.

3. Select All option does not work on Backup Target Types page

4. Sorting the columns on the backup's admin page is not working

5. Workload Policy field is missing in the workload creation wizard until the scheduler is enabled

6. In the workload-reassign-workloads help text --new_tenant_ids is provided instead of --new_tenant_id

7. [Canonical multiple Backup Target] If NFS Backup Target getting deployed before S3 Backup Targets, then object store services against S3 Backup Target go into error state

   • If the overlay bundle is having multiple Backup Target Types defined such that at least one is NFS and one is S3 and if deployment of NFS happens first, then it is noticed that NFS mount happens successfully whereas object store services against S3 Backup Target Types start failing. Error An error occured (403) when calling HeadBucket operation: Forbidden is observed.

   • Workaround:

     • manually unmount the NFS mountpoint. All required S3 object store services will start successfully along with mounting of S3 sharepath. Unmounted NFS also gets remounted within 2-3 min (via mtab).

8. workload modify CLI command with workload_id at end is throwing an error

   • Workaround:

     • Place the workload_id before the optional arguments.

     • Sample command workloadmgr workload-modify 858da6e0-5b55-4c63-9a8e-d9f116130686 --jobschedule enabled=True --jobschedule start_time='07:00 AM' --jobschedule start_date='01/08/2025' --hourly interval='1' retention='2' snapshot_type='incremental'

Trilio services logs on RHOSP

On the Controller Node

Below Trilio containers gets deployed on the Contoller node.

• triliovault_datamover_api

• triliovault_wlm_api

• triliovault_wlm_scheduler

• triliovault_wlm_workloads

• triliovault-wlm-cron

The log files for above Trilio services can be found here:

/var/log/containers/triliovault-datamover-api/triliovault-datamover-api.log

/var/log/containers/triliovault-wlm-api/triliovault-wlm-api.log

/var/log/containers/triliovault-wlm-cron/triliovault-wlm-cron.log

/var/log/containers/triliovault-wlm-scheduler/triliovault-wlm-scheduler.log

/var/log/containers/triliovault-wlm-workloads/triliovault-wlm-workloads.log

In the case of using S3 as a backup target is there also a log file that keeps track of the S3-Fuse plugin used to connect with the S3 storage.

/var/log/containers/triliovault-wlm-workloads/triliovault-object-store.log

For file serach operation, logs can be found on Controller node at below location:

/var/log/containers/triliovault-wlm-workloads/workloadmgr-filesearch.log

On the Compute Node

Trilio Datamover container gets deployed on the Compute node.

The log file for the Trilio Datamover service can be found at:

/var/log/containers/triliovault-datamover/triliovault-datamover.log

Please follow the below steps to add new backup target on RHOSO18.

1] Mount Backup target on Trilio Control Plane

Navigate to trilio ctlplane-scripts directory

Plan which type of backup target you want to add. T4O supports two types of backup targets.

1. nfs

2. s3

If you want to add backup target of type 'nfs' then edit the following yaml file and create TVOBackupTarget resource.

If you want to add backup target of type 's3' then edit the following yaml file. In following secret user needs to mention base64 encoded s3 bucket access key and secret key.

To get base64 encoded string use linux command

If your S3 bucket is an Amazon S3 bucket.

If your S3 is of any other type edit following file and create TVOBackupTarget resource

Check deployment progress

2] Mount Backup target on Trilio Data Plane

Navigate to data plane scripts directory

Create templates needed for adding backup targets. Please note that use unique backup target name for parameter <BACKUP_TARGET_NAME>. You should not have used this backup target name earlier for any other trilio backup target. For parameter <BACKUP_TARGET_TYPE>, valid choices are ‘s3' and 'nfs’

A new directory gets created with backup target name having necessary templates. You can list these templates.

Backup target name gets set in all the yaml files created in this directory. BACKUP_TARGET_NAME gets converted to all small case and if any underscore ‘_' character is there in it, it gets replaced by hyphen character '-'. Add backup target details to template

Change to <BACKUP_TARGET_NAME> if not done already

Edit config map file

Create config map

If you are adding 's3' type backup target, then only you need to create following secret. We have already filled in all details in yaml file. We just need to apply it.

Set Trilio Ansible Runner container image url and tag in parameter 'openStackAnsibleEERunnerImage:' of trilio-add-backup-target-service.yaml You don’t need to change any other parameter.

Create custom data plane service for this backup target

Edit trilio-add-backup-target-deployment.yaml file and set parameter ‘nodeSets' with correct value from your environment. You don’t need to change any other parameter.

Get OpenStackDataPlaneNodeSet name

Set 'nodeSets' parameter

Trigger deployment of this backup target

Check logs Edit <DEPLOYMENT_NAME>, take it from above deployment yaml.

If this pod does not get created, it means you have not used unique backup target name or some other issue happend. Please verify that

3] Add Backup Target Records

Login to triliovault-wlm-api pod and run below CLI command to create backup target in Trilio DB.

For NFS Backup Target:

Sample command:

For Object lock enabled S3 Backup Target:

Sample command:

For non-object lock S3 Backup Target:

Sample command:

Trilio for OpenStack Compatibility Matrix

NFS & S3 Support:

All versions of T4O-6.x releases support NFSv3 and S3 as backup targets on all the compatible distributions.

Encryption Support:

All versions of T4O-6.x releases support encryption using Barbican service on all the compatible distributions.

QEMU Guest Agent

Installing the QEMU guest agent is an optional but recommended service to run inside the VM to ensure backup data consistency with Trilio. QEMU Guest Agent is the standard OpenSource QEMU/KVM hypervisor agent that runs inside a virtual machine (VM) and communicates with the host system (the hypervisor) to provide enhanced management and control of the VM. It is an essential component of any hypervisor platform that OpenStack can use to provide enhanced functionality when required.

Enabling QEMU Guest Agent

To enable this feature, you must set hw_qemu_guest_agent=yes as a metadata parameter on the image you wish to use to create the guest-agent-capable instances from. Reference: https://docs.openstack.org/nova/pike/admin/configuration/hypervisor-kvm.html

If the Virtual Machine is already created from the glance image which doesn't have the hw_qemu_guest_agent=yes property set, guest agent can be enabled using following process:

The process is stopping the VM, manually adding the lines below to the virsh definition using "virsh edit ", and starting it again:

The VM the needs to be running the qemu-guest agent software:

Ubuntu/Debian: sudo apt install -y qemu-guest-agent && sudo systemctl enable qemu-guest-agent && sudo systemctl restart qemu-guest-agent && sudo systemctl status qemu-guest-agent --no-pager

RHEL/CentOS VMs: sudo yum install -y qemu-guest-agent && sudo systemctl enable qemu-guest-agent && sudo systemctl restart qemu-guest-agent && sudo systemctl status qemu-guest-agent --no-pager

Windows VMs: Inside the guest OS, install the guest agent tools from: https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win-guest-tools.exe. Finally, make sure the service is running and set to start on boot automatically, in PowerShell, as Administrator: Set-Service QEMU-GA -StartupType Automatic; Restart-Service QEMU-GA; Get-Service QEMU-GA

You can confirm the qemu-guest-agent is running properly after installed, from the compute host, like this:

virsh qemu-agent-command $INSTANCE '{"execute":"guest-ping"}'

The output must be: {"return":{}}

Software Driven Migration: VMware to OpenStack

Software Driven Migration is supported in T4O 5.X release. Please refer

After the installation and configuration of Trilio for OpenStack did succeed the following steps can be done to verify that the Trilio installation is healthy.

On the Controller node

Make sure below containers are in a running state, triliovault-wlm-cron would be running on only one of the controllers in case of multi-controller setup.

• triliovault_datamover_api

• triliovault_wlm_api

• triliovault_wlm_scheduler

• triliovault_wlm_workloads

• triliovault-wlm-cron

If the containers are in a restarting state or not listed by the following command then your deployment is not done correctly. Please recheck if you followed the complete documentation.

After successful deployment, triliovault-wlm-cron service would get added in pcs cluster as a cluster resource, you can verify through pcs status command

Verify the HAproxy configuration under:

On Compute node

Make sure the Trilio Datamover container is in running state and no other Trilio container is deployed on compute nodes.

Check if provided backup target is mounted well on Compute host.

On the node with Horizon service

Make sure the horizon container is in a running state. Please note that the Horizon container is replaced with Trilio's Horizon container. This container will have the latest OpenStack horizon + Trilio horizon plugin.

About the workloadmgr CLI client

The workloadmgr CLI client is provided as rpm and deb packages.

The following operating systems have been verified with the installation:

• Rocky9, RHEL9

• CentOS8

• Ubuntu 18.04, Ubuntu 20.04

Installing the workloadmgr client will automatically install all required OpenStack clients as well.

Further, the installation of the workloadmgr client will integrate the client into the global OpenStack Python client, if available.

Install workloadmgr client rpm package on CentOS8

The Trilio workload manager CLI client has several requirements that need to be met before the client can be installed without dependency issues.

Preparing the workloadmgr client installation

The following steps need to be done to prepare the installation of the workloadmgr client:

1. Add required repositories

   1. epel-release yum -y install epel-release

   2. centos-release-openstack-train yum -y install centos-release-openstack-train

Installing the workloadmgr client

Installing from the Trilio online repository

To install the latest available workloadmgr package for a Trilio release from the Trilio repository the following steps need to be done:

Create the Trilio yum repository file /etc/yum.repos.d/trilio.repo Enter the following details into the repository file:

Install the workloadmgr client issuing the following command:

yum install python-3-workloadmgrclient-el8

Install workloadmgr client rpm package on Rocky9 and RHEL9

The Trilio workload manager CLI client has several requirements that need to be met before the client can be installed without dependency issues.

Preparing the workloadmgr client installation

The following steps need to be done to prepare the installation of the workloadmgr client:

1. Add required repositories

   1. epel-release yum -y install epel-release

Installing the workloadmgr client

Installing from the Trilio online repository

To install the latest available workloadmgr package for a Trilio release from the Trilio repository the following steps need to be done:

Create the Trilio yum repository file /etc/yum.repos.d/trilio.repo Enter the following details into the repository file:

Install the workloadmgr client issuing the following command:

yum install python-3-workloadmgrclient-el9

Install workloadmgr client deb packages on Ubuntu

The Trilio workloadmgr client packages for Ubuntu are only available from the online repository.

Preparing the workloadmgr client installation

There is no preparation required. All dependencies are automatically resolved by the standard repositories provided by Ubuntu.

Installing the Workloadmgr client

Installing from the Trilio online repository

To install the latest available workloadmgr package for a Trilio release from the Trilio repository the following steps need to be done:

Create the Trilio yum repository file /etc/apt/sources.list.d/fury.list Enter the following details into the repository file:

run apt update to make the new repository available.

The apt package manager is used to install the workloadmgr client package:

apt-get install python3-workloadmgrclient

Introduction

Filename and location:

This file exclusively comes into play when users aim to configure Trilio with a NFS backup target that employs multiple network endpoints. For all other scenarios, such as single IP NFS or S3, this file remains inactive, and in such instances, please consult the standard installation documentation.

When using an NFS backup target with multiple network endpoints, T4O will mount a single IP/endpoint on a designated compute node for a specific NFS share. This approach enables users to distribute NFS share IPs/endpoints across various compute nodes.

The 'triliovault_nfs_map_input.yml' file allows users to distribute/load balance NFS share endpoints across compute nodes in a given cloud.

Examples

Using hostnames

Here, the User has ‘one’ NFS share exposed with three IP addresses. 192.168.1.34, 192.168.1.35, 192.168.1.33 Share directory path is: /var/share1

So, this NFS share supports the following full paths that clients can mount:

There are 32 compute nodes in the OpenStack cloud. 30 node hostnames have the following naming pattern

The remaining 2 node hostnames do not follow any format/pattern.

Now the mapping file will look like this

Using IPs

Compute node IP range used here: 172.30.3.11-40 and 172.30.4.40, 172.30.4.50 Total of 32 compute nodes

Getting the correct compute hostnames/IPs

RHOSP

Use the following command to get compute hostnames. Check the ‘Name' column. Use these exact hostnames in 'triliovault_nfs_map_input.yml' file.

In the following command output, ‘overcloudtrain1-novacompute-0' and ‘overcloudtrain1-novacompute-1' are correct hostnames.

The Trilio solution is using the qcow2 backing file to provide full synthetic backups.

Especially when the NFS backup target is used, there are scenarios at which this backing file needs to be updated to a new mount path.

To make this process easier and streamlined Trilio is providing the following rebase tool.

Same cloud - different owner

Migration within the same cloud to a different owner Cloud A — Domain A — Project A — User A => Cloud A — Domain A — Project A — User B Cloud A — Domain A — Project A — User A => Cloud A — Domain A — Project B — User B Cloud A — Domain A — Project A — User A =>Cloud A — Domain B — Project B — User B

Steps used:

1. Create a secret for Project A in Domain A via User A.

2. Create encrypted workload in Project A in Domain A via User A. Take snapshot.

3. Reassign workload to new owner

4. Load rc file of User A & provide read only rights through acl to the new owner

   openstack acl user add --user <userB_id> <secret_href> --insecure

Different cloud

Migration between clouds Cloud A — Domain A — Project A — User A => Cloud B — Domain B — Project B — User B

Steps used:

1. Create a secret for Project A in Domain A via User A.

2. Create an encrypted workload in Project A in Domain A via User A. Trigger snapshot.

3. Reassign workload to Cloud B - Domain B — Project B — User B

4. Load RC file of User B.

5. Create a secret for Project B in Domain B via User B with the same payload used in Cloud A.

6. Create token via “openstack token issue --insecure”

7. Add migrated workload's metadata to the new secret (provide issued token to Auth-Token & workload id to matadata as below)

curl -i -X PUT \
   -H "X-Auth-Token:gAAAAABh0ttjiKRPpVNPBjRjZywzsgVton2HbMHUFrbTXDhVL1w2zCHF61erouo4ZUjGyHVoIQMG-NyGLdR7nexmgOmG7ed66LJ3IMVul1LC6CPzqmIaEIM48H0kc-BGvhV0pvX8VMZiozgFdiFnqYHPDvnLRdh7cK6_X5dw4FHx_XPmkhx7PsQ" \
   -H "Content-Type:application/json" \
   -d \
'{
  "metadata": {
      "workload_id": "c13243a3-74c8-4f23-b3ac-771460d76130",
      "workload_name": "workload-c13243a3-74c8-4f23-b3ac-771460d76130"
    }
}' \
 'https://kolla-victoria-ubuntu20-1.triliodata.demo:9311/v1/secrets/f3b2fce0-3c7b-4728-b178-7eb8b8ebc966/metadata'
 
 
curl -i -X GET \
   -H "X-Auth-Token:gAAAAABh0ttjiKRPpVNPBjRjZywzsgVton2HbMHUFrbTXDhVL1w2zCHF61erouo4ZUjGyHVoIQMG-NyGLdR7nexmgOmG7ed66LJ3IMVul1LC6CPzqmIaEIM48H0kc-BGvhV0pvX8VMZiozgFdiFnqYHPDvnLRdh7cK6_X5dw4FHx_XPmkhx7PsQ" \
 'https://kolla-victoria-ubuntu20-1.triliodata.demo:9311/v1/secrets/f3b2fce0-3c7b-4728-b178-7eb8b8ebc966/metadata'

Troubleshooting inside a complex environment like OpenStack can be very time-consuming.

The following tips will help to speed up the troubleshooting process to identify root causes.

What is happening where

OpenStack and Trilio are divided into multiple services. Each service has a very specific purpose that is called during a backup or recovery procedure. Knowing which service is doing what helps to understand where the error is happening, allowing more focused troubleshooting.

Trilio Workloadmgr

The Trilio Workloadmgr is the Controller of Trilio. It receives all Workload related requests from the users.

Every task of a backup or restore process is triggered and managed from here. This includes the creation of the directory structure and initial metadata files on the Backup Target.

During a backup process

During a backup process is the Trilio Workloadmgr also responsible for gathering the metadata about the backed-up VMs and networks from the OpenStack environment. It sends API calls to the OpenStack endpoints on the configured endpoint type to fetch this information. Once the metadata has been received, the Trilio Workloadmgr writes it as JSON files on the Backup Target.

The Trilio Workloadmgr is also sending the Cinder Snapshot command.

During a restore process

During the restore process, the Trilio Workloadmgr reads the VM metadata from its Database and uses the metadata to create the Shell for the restore. It sends API calls to the OpenStack environment to create the necessary resources.

Datamover API

The dmapi service is the connector between the Trilio cluster and the Datamover running on the compute nodes.

The purpose of the dmapi service is to identify which compute node is responsible for the current backup or restore task. To do so, the dmapi service connects to the nova API requesting the compute hose of a provided VM.

Once the compute host has been identified, the dmapi forwards the command from the Trilio Workloadmgr to the datamover running on the identified compute host.

Datamover

The datamover is the Trilio service running on the compute nodes.

Each datamover is responsible for the VMs running on top of its compute node. A datamover can not work with VMs running on a different compute node.

The datamover controls the freeze and thaw of VMs as well as the actual movement of the data.

Everything on the Backup Target is happening as user nova

Trilio is reading and writing on the Backup Target as nova:nova.

The POSIX user-id and group-id of nova:nova need to be aligned between the Trilio Cluster and all compute nodes. Otherwise, backup or restores may fail with permission or file not found issues.

Alternativ ways to achieve the goal are possible, as long as all required nodes can fully write and read as nova:nova on the Backup Target.

It is recommended to verify the required permissions on the Backup Target in case of any errors during the data transfer phase or in case of any file permission errors.

Input/Output Error with Cohesity NFS

On Cohesity NFS if an Input/Output error is observed, then increase the timeo and retrans parameter values in your NFS options.

Timeout receiving packet error in multipath ISCSi environment

Logging inside all datamover containers and add uxsock_timeout with value as 60000 which is equal to 60 sec inside /etc/multipath.conf.Restart datamover container

Trilio Trustee Role

Trilio is using RBAC to allow the usage of Trilio features to users.

This trustee role is absolutely required and can not be overwritten using the admin role.

It is recommended to verify the assignment of the Trilio Trustee Role in case of any permission errors from Trilio during the creation of Workloads, backups, or restores.

OpenStack Quotas

Trilio is creating Cinder Snapshots and temporary Cinder Volumes. The OpenStack Quotas need to allow that.

Every disk that is getting backed up requires one temporary Cinder Volumes.

Every Cinder Volume that is getting backup requires two Cinder Snapshots. The second Cinder Snapshot is temporary to calculate the incremental.

Definition

Trilio can notify users via E-Mail upon the completion of backup and restore jobs.

The E-Mail will be sent to the owner of the Workload.

Requirements to activate E-Mail Notifications

To use the E-mail notifications, two requirements need to be met.

User E-Mail assigned

As the E-Mail will be sent to the owner of the Workload does the OpenStack User, who created the workload, require to have an E-Mail address associated.

Trilio E-Mail Server configured

Trilio needs to know which E-Mail server to use, to send the E-mail notifications. Backup Administrators can do this in the "Backup Admin" area.

Activate/Deactivate the E-Mail Notifications

E-Mail notifications are activated tenant wide. To activate the E-Mail notification feature for a tenant follow these steps:

1. Login to Horizon

2. Navigate to the Backups

3. Navigate to Settings

4. Check/Uncheck the box for "Enable Email Alerts"

Example E-Mails

The following screenshots show example E-mails send by Trilio.

Clean Trilio Datamover API and Workloadmanager services

The following steps need to be run on all nodes, which have the Trilio Datamover API & Workloadmanager services running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the below entries

Once the role that runs the Trilio Datamover API & Workloadmanager services has been identified will the following commands clean the nodes from the service.

Remove triliovault_datamover_api container.

Clean Trilio Datamover API service conf directory.

Clean Trilio Datamover API service log directory.

Remove triliovault_wlm_api container.

Clean Trilio Workloadmanager API service conf directory.

Clean Trilio Workloadmanager API service log directory.

Remove triliovault_wlm_workloads container.

Clean Trilio Workloadmanager Workloads service conf directory.

Clean Trilio Workloadmanager Workloads service log directory.

Remove triliovault_wlm_scheduler container.

Clean Trilio Workloadmanager Scheduler service conf directory.

Clean Trilio Workloadmanager Scheduler service log directory.

Remove triliovault-wlm-cron-podman-0 container from controller.

Clean Trilio Workloadmanager Cron service conf directory.

Clean Trilio Workloadmanager Cron service log directory.

Clean Trilio Datamover Service

The following steps need to be run on all nodes, which have the Trilio Datamover service running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the entry OS::TripleO::Services::TrilioDatamover.

Once the role that runs the Trilio Datamover service has been identified will the following commands clean the nodes from the service.

Remove triliovault_datamover container.

Unmount the Trilio Backup Target on the compute host.

Clean Trilio Datamover service conf directory.

Clean log directory of Trilio Datamover service.

Clean wlm-cron resource from pcs cluster

Remove wlm cron resource from pcs cluster on the controller node.

Clean Trilio HAproxy resources

The following steps need to be run on all nodes, which have the HAproxy service running. Those nodes can be identified by checking the roles_data.yaml for the role that contains the entry OS::TripleO::Services::HAproxy.

Once the role that runs the HAproxy service has been identified will the following commands clean the nodes from all the Trilio resources.

Edit the following file inside the HAproxy container and remove all Trilio entries.

/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg

An example of these entries is given below.

Restart the HAproxy container once all edits have been done.

Clean Trilio Keystone resources

Trilio registers services and users in Keystone. Those need to be unregistered and deleted.

Clean Trilio database resources

Trilio creates databases for dmapi and workloadmgr services. These databases need to be cleaned.

Login into the database cluster

Run the following SQL statements to clean the database.

Revert overcloud deploy command

Remove the following entries from roles_data.yaml used in the overcloud deploy command.

• OS::TripleO::Services::TrilioDatamoverApi

• OS::TripleO::Services::TrilioWlmApi

• OS::TripleO::Services::TrilioWlmWorkloads

• OS::TripleO::Services::TrilioWlmScheduler

• OS::TripleO::Services::TrilioWlmCron

• OS::TripleO::Services::TrilioDatamover

Follow these steps to clean the overcloud deploy command from all Trilio entries.

1. Remove trilio_env.yaml entry

2. Remove trilio endpoint map file Replace with original map file if existing

Revert back to the original RHOSP Horizon container

Run the cleaned overcloud deploy command.

Definition

Every Workload has its own schedule. Those schedules can be activated, deactivated and modified.

A schedule is defined by:

• Status (Enabled/Disabled)

• Start Day/Time

• End Day

• Hrs between 2 snapshots

Disable a schedule

Using Horizon

To disable the scheduler of a single Workload in Horizon do the following steps:

1. Login to the Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload to be modified

5. Click the small arrow next to "Create Snapshot" to open the sub-menu

6. Click "Edit Workload"

7. Navigate to the tab "Schedule"

8. Uncheck "Enabled"

9. Click "Update"

Using CLI

• --workloadid <workloadid> ➡️ Requires at least one workloadid, Specify an ID of the workload whose scheduler disables. Specify option multiple times to include multiple workloads. --workloadids <workloadid> --workloadids <workloadid>

Enable a schedule

Using Horizon

To disable the scheduler of a single Workload in Horizon do the following steps:

1. Login to the Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload to be modified

5. Click the small arrow next to "Create Snapshot" to open the sub-menu

6. Click "Edit Workload"

7. Navigate to the tab "Schedule"

8. check "Enabled"

9. Click "Update"

Using CLI

• --workloadid <workloadid> ➡️ Requires at least one workloadid, Specify an ID of the workload whose scheduler disables. Specify option multiple times to include multiple workloads. --workloadids <workloadid> --workloadids <workloadid>

Modify a schedule

To modify a schedule the workload itself needs to be modified.

Please follow this procedure to .

Verify the scheduler trust is working

Trilio is using the which enables the Trilio service user to act in the name of another OpenStack user.

This system is used during all backup and restore features.

Using Horizon

As a trust is bound to a specific user for each Workload does the Trilio Horizon plugin show the status of the Scheduler on the Workload list page.

Using CLI

• <workload_id> ➡️ ID of the workload to validate

1. Can Trilio for OpenStack restore instance UUIDs?

Answer: NO

Trilio for OpenStack does not restore Instance UUIDs (also known as Instance IDs). The only scenario where we do not modify the Instance UUID is during an Inplace Restore, where we only recover the data without creating new instances.

When Trilio for OpenStack restores virtual machines (VMs), it effectively creates new instances. This means that new Virtual Machine Instance UUIDs are generated for the restored VMs. We achieve this by orchestrating a call to Nova, which creates new VMs with new UUIDs.

By following this approach, we maintain the principles of OpenStack and auditing. We do not update or modify existing database entries when objects are deleted and subsequently recovered. Instead, all deletions are marked as such, and new instances, including the recovered ones, are created as new objects in the Nova tables. This ensures compliance and preserves the integrity of the OpenStack environment.

2. Can Trilio for OpenStack restore MAC addresses?

Answer: YES

Trilio can restore the VMs MAC address, however, there is a caveat when restoring a virtual machine (VM) to a different IP address: a new MAC address will be assigned to the VM.

In the case of a One-Click Restore, the original MAC addresses and IP addresses will be recovered, but the VM will be created with a new UUID, as mentioned in question #1.

When performing a Selective Restore, you have the option to recover the original MAC address. To do so, you need to select the original IP address from the available dropdown menu during the recovery process.

By choosing the original IP address, Trilio for OpenStack will ensure that the VM is restored with its original MAC address, providing more flexibility and customization in the restoration process.

Example of Selective Restore with original MAC (and IP address):

1. In this example, we have taken a Trilio backup of a VM called prod-1.

1. The VM is deleted and we perform a Selective Restore of a VM called prod-1, selecting the IP address it was originally assigned from the drop-down menu:

1. Trilio then restores the VM with the original MAC address:

1. If you left the option as "Choose next available IP address", it will assign a new MAC to the VM instead as Neutron maps all MAC addresses to IP addresses on the Subnet - so logically a new IP will result in a new MAC address.

Trilio Workloads are designed to allow a Disaster Recovery without the need to backup the Trilio database.

As long as the Trilio Workloads are existing on the Backup Target Storage and a Trilio installation has access to them, it is possible to restore the Workloads.

Disaster Recovery Process

1. and Trilio for the target cloud

4. Notify users to of Workloads being available

Mount-paths

Trilio incremental Snapshots involve a backing file to the prior backup taken, which makes every Trilio incremental backup a synthetic full backup.

Trilio is using qcow2 backing files for this feature:

As can be seen in the example is the backing file an absolute path, which makes it necessary, that this path exists so the backing files can be accessed.

Trilio is using the base64 hashing algorithm for the NFS mount-paths, to allow the configuration of multiple NFS Volumes at the same time. The hash value is calculated using the provided NFS path.

When the path of the backing file is not available on the Trilio appliance and Compute nodes, will the restores of incremental backups fail.

The tested and recommended method to make the backing files available is creating the required directory path and using mount --bind to make the path available for the backups.

Running the mount --bind command will make the necessary path available until the next reboot. If it is required to have access to the path beyond a reboot is it necessary to edit the fstab.

Immutable Backups

Release version: 6.0

Trilio offers the capability to create immutable T4O backups/snapshots, ensuring they cannot be deleted until the specified retention period has ended. It leverages the target's locking and versioning features to achieve this immutability.

Purpose

In recent years, the Trilio S3 Fuse plugin introduced the ability to support backup immutability through the S3 object lock mechanism. This feature ensures that backups remain secure even if the OpenStack platform is compromised, acting as a crucial safeguard against cyber attacks. While traditional backup solutions prevent direct exposure of backup media to the platform, Trilio's scale-out architecture necessitates direct attachment of the

Key Highlights

• Pre-requisites:

  • Create a new S3 bucket to enable the object-locking feature, as it cannot be enabled on existing buckets.

  • Configure Object Locking in Compliance mode with a retention period of at least one day for enhanced protection.

  • Disable object lifecycle policies to prevent automated deletions.

  • Perform manual deletion of objects only after the retention period has expired.

• Retention

  With the introduction of the immutable backups feature, Trilio’s retention policy has been changed. It's important to note that Trilio can no longer support incremental/full synthetic backups forever, as this functionality requires modifying the backup images. Instead, the following retention policy is now in place:

  • If the user wants to retain n number of backups, then the user needs to take full backups every n backups.

  • To keep at least n backups available, Trilio may retain 2n backups, forming two backup chains. Once a chain is fully formed with one full backup and n-1 incremental backup images, the second chain will start with a new full backup. The first chain will be removed once the second chain is fully formed with a full backup and n-1 incremental backups.

How To Use It

Follow the steps mentioned here to know how to use this feature in the product.

Configuration

To enable this feature, you need to adjust certain configuration settings in the product. Refer to the deployment guides of the desired distribution for detailed instructions.

Limitations

• The feature may not work as expected if the object lifecycle policy is set for a backup target.

• The same backup target cannot be used for DR or migration purposes, additional steps will be required to perform it.

Advanced Scheduler & Retention

Release version: 6.0

The Advanced Scheduler is a newly introduced feature in Trilio that provides enhanced flexibility and granularity in scheduling and retaining snapshots. With this feature, users can now schedule both full and incremental snapshots across various intervals—hourly, daily, weekly, monthly, and yearly. Each scheduling type offers customizable retention policies, allowing users to manage snapshots as per their specific requirements effectively.

Purpose

The purpose of the Advanced Scheduler is to empower users with more control over their snapshot management. By addressing limitations in the previous scheduler, this feature ensures improved efficiency, optimized resource usage, and greater alignment with organizational backup policies. This feature is especially beneficial for organizations with diverse workloads that demand varying backup frequencies and retention policies.

Key Highlights

• Flexible Scheduling Intervals: Users can now schedule snapshots across the following intervals:

  • Hourly: For tasks requiring frequent backups.

  • Daily: Ideal for daily operations requiring end-of-day snapshots.

  • Weekly: Suitable for less critical workloads.

  • Monthly: For long-term periodic backups.

  • Yearly: Designed for archival purposes.

• Snapshot Type Options:

  • Full Snapshots: Each scheduling type allows the user to choose the full snapshot type. But, full snapshot type is mandatory for Weekly, Monthly, and Yearly scheduling types to avoid longer backup chains which may lead to data corruption.

  • Incremental Snapshots: Hourly and Daily scheduling types provide the options from either incremental or full snapshots.

• Advanced Retention Policies:

  • Users can define retention settings for each scheduling type.

  • Retention can be based on:

    • Number of Snapshots: Keep a specified number of snapshots for the chosen schedule.

    • Retention Duration: This is based on the scheduling type selected by the user.

• Enhanced Management Efficiency:

  • Simplifies backup strategies by tailoring frequency, type, and retention for specific workloads.

  • Supports compliance and long-term data retention policies.

How To Use It

Follow the steps mentioned here to know how to use this feature in the product.

Limitations

While the Advanced Scheduler introduces significant improvements, it has certain limitations:

1. Immutable Backups:

   • In case the user chooses the immutable backup storage, the scheduling policy will get limited options. The user can have only hourly scheduling and retention types and hourly will have only 24hrs interval option.

2. Resource Utilization:

   • Frequent full snapshots (e.g., on an hourly basis) may lead to higher storage and performance overhead.

3. Complexity in Configuration:

   • Advanced options may require more planning and understanding, increasing the learning curve for new users.

4. Dependency on Storage Quotas:

   • Users must ensure sufficient storage capacity to handle the retention policies and the frequency of snapshots.

5. Manual Monitoring:

   • In cases of excessive scheduling, manual intervention might be necessary to adjust retention policies or optimize schedules.

Multiple Backup Targets

Release version: 6.0

Previously, Trilio allowed a single backup target per cloud, either NFS or S3, for all backups. Now, Trilio supports multiple backup targets, enabling both NFS and S3 types to coexist as backup options. Admin users can configure multiple backup targets and define their accessibility for tenant users. Additionally, admin users have the ability to restrict backup targets to specific projects for enhanced control.

Purpose

The feature aims to enhance the flexibility, scalability, and manageability of backup configurations in Trilio. By supporting multiple backup targets:

• Organizations can align their backup strategies with diverse operational requirements.

• Admin users can provide tenant users with more options for workload backups.

• Access control over backup targets is improved, allowing organizations to secure sensitive data.

Key Highlights

• Support for Multiple Backup Targets: Trilio now supports both NFS and S3 simultaneously for backup storage.

• Admin Configuration: Admin users can configure multiple backup targets during deployment.

• Tenant User Flexibility: Tenant users can choose from available backup targets when creating workloads.

• Enhanced Access Control: Admin users can mark backup targets as private and restrict access to specific projects by creating backup target types and marking them as private.

How To Use It

Follow the steps mentioned here to know how to use this feature in the product.

Configuration

To take advantage of this feature, you need to adjust certain configuration settings in the product. Refer to the deployment guides of the desired distribution for detailed instructions.

Limitations

• Backup Targets are configured during deployment. Modifying them may require redeploying Trilio.

• While this feature is highly expandable, it currently limits the isolation of the backup targets to the projects only.

Definition

The file search functionality allows the user to search for files and folders located on a chosen VM in a workload in one or more Backups.

Navigating to the file search tab in Horizon

The file search tab is part of every workload overview. To reach it follow these steps:

1. Login to Horizon

2. Navigate to Backups

3. Navigate to Workloads

4. Identify the workload a file search shall be done in

5. Click the workload name to enter the Workload overview

6. Click File Search to enter the file search tab

Configuring and starting a file search Horizon

A file search runs against a single virtual machine for a chosen subset of backups using a provided search string.

To run a file search the following elements need to be decided and configured

Choose the VM the file search shall run against

Under VM Name/ID choose the VM that the search is done upon. The drop down menu provides a list of all VMs that are part of any Snapshot in the Workload.

Set the File Path

The File Path defines the search string that is run against the chosen VM and Snapshots. This search string does support basic RegEx.

Example File Path for all files inside /etc : /etc/*

Define the Snapshots to search in

"Filter Snapshots by" is the third and last component that needs to be set. This defines which Snapshots are going to be searched.

There are 3 possibilities for a pre-filtering:

1. All Snapshots - Lists all Snapshots that contain the chosen VM from all available Snapshots

2. Last Snapshots - Choose between the last 10, 25, 50, or custom Snapshots and click Apply to get the list of the available Snapshots for the chosen VM that match the criteria.

3. Date Range - Set a start and end date and click apply to get the list of all available Snapshots for the chosen VM within the set dates.

After the pre-filtering is done all matching Snapshots are automatic prechosen. Uncheck any Snapshot that shall not be searched.

Start the File Search and retrieve the results in Horizon

To start a File Search the following elements need to be set:

• A VM to search in has to be chosen

• A valid File Path provided

• At least one Snapshot to search in selected

Once those have been set click "Search" to start the file search.

After a short time the results will be presented. The results are presented in a tabular format grouped by Snapshots and Volumes inside the Snapshot.

For each found file or folder the following information are provided:

• POSIX permissions

• Amount of links pointing to the file or folder

• User ID who owns the file or folder

• Group ID assigned to the file or folder

• Actual size in Bytes of the file or folder

• Time of creation

• Time of last modification

• Time of last access

• Full path to the found file or folder

Once the Snapshot of interest has been identified it is possible to go directly to the Snapshot using the "View Snapshot" option at the top of the table. It is also possible to directly mount the Snapshot using the "Mound Snapshot" Button at the end of the table.

Doing a CLI File Search

workloadmgr filepath-search [--snapshotids <snapshotid>]
                            [--end_filter <end_filter>]
                            [--start_filter <start_filter>]
                            [--date_from <date_from>]
                            [--date_to <date_to>]
                            <vm_id> <file_path>

• <vm_id> ➡️ ID of the VM to be searched

• <file_path> ➡️ Path of the file to search for

• --snapshotids <snapshotid> ➡️ Search only in specified snapshot ids snapshot-id: include the instance with this UUID

• --end_filter <end_filter> ➡️Displays last snapshots, example , last 10 snapshots, default 0 means displays all snapshots

• --start_filter <start_filter> ➡️Displays snapshots starting from , example , snapshot starting from 5, default 0 means starts from first snapshot

• --date_from <date_from> ➡️ From date in format 'YYYY-MM-DDTHH:MM:SS' eg 2016-10-10T00:00:00, If time isn't specified then it takes 00:00 by default

• --date_to <date_to> ➡️ To date in format 'YYYY-MM-DDTHH:MM:SS'(defult is current day),Specify HH:MM:SS to get snapshots within same day inclusive/exclusive results for date_from and date_to

Trilio enables OpenStack administrators to set Project Quotas against the usage of Trilio.

The following Quotas can be set:

• Number of Workloads a Project is allowed to have

• Number of Snapshots a Project is allowed to have

• Number of VMs a Project is allowed to protect

• Amount of Storage a Project is allowed to use on the Backup Target

Work with Workload Quotas via Horizon

Workload Quotas are managed like any other Project Quotas.

1. Login into Horizon as user with admin role

2. Navigate to Identity

3. Navigate to Projects

4. Identify the Project to modify or show the quotas on

5. Use the small arrow next to "Manage Members" to open the submenu

6. Choose "Modify Quotas"

7. Navigate to "Workload Manager"

8. Edit Quotas as desired

9. Click "Save"

Work with Workload Quotas via CLI

List available Quota Types

Trilio is providing several different Quotas. The following command allows listing those.

workloadmgr project-quota-type-list

Show Quota Type Details

The following command will show the details of a provided Quota Type.

workloadmgr project-quota-type-show <quota_type_id>

• <quota_type_id> ➡️ID of the Quota Type to show

Create a Quota

The following command will create a Quota for a given project and set the provided value.

workloadmgr project-allowed-quota-create --quota-type-id quota_type_id
                                         --allowed-value allowed_value 
                                         --high-watermark high_watermark 
                                         --project-id project_id

• <quota_type_id> ➡️ID of the Quota Type to be created

• <allowed_value>➡️ Value to set for this Quota Type

• <high_watermark>➡️ Value to set for High Watermark warnings

• <project_id>➡️ Project to assign the quota to

List allowed Quotas

The following command lists all Trilio Quotas set for a given project.

workloadmgr project-allowed-quota-list <project_id>

• <project_id>➡️ Project to list the Quotas from

Show allowed Quota

The following command shows the details about a provided allowed Quota.

workloadmgr project-allowed-quota-show <allowed_quota_id>

• <allowed_quota_id> ➡️ID of the allowed Quota to show.

Update allowed Quota

The following command shows how to update the value of an already existing allowed Quota.

workloadmgr project-allowed-quota-update [--allowed-value <allowed_value>]
                                         [--high-watermark <high_watermark>]
                                         [--project-id <project_id>]
                                         <allowed_quota_id>

• <allowed_value>➡️ Value to set for this Quota Type

• <high_watermark>➡️ Value to set for High Watermark warnings

• <project_id>➡️ Project to assign the quota to

• <allowed_quota_id> ➡️ID of the allowed Quota to update

Delete allowed Quota

The following command will delete an allowed Quota and sets the value of the connected Quota Type back to unlimited for the affected project.

workloadmgr project-allowed-quota-delete <allowed_quota_id>

• <allowed_quota_id> ➡️ID of the allowed Quota to delete

Trilio Branch

Branch of the repository to be used for DevOps scripts.

Deployment Scripts

Container/Revision Information

Package Repositories & Versions

All packages are Python 3 (py3.6 - py3.10) compatible only.

Trilio Branch

Branch of the repository to be used for DevOps scripts.

Deployment Scripts

Container/Revision Information

Package Repositories & Versions

All packages are Python 3 (py3.6 - py3.10) compatible only.

Trilio and Canonical have started a partnership to ensure a native deployment of Trilio using JuJu Charms.

Those JuJu Charms are publicly available as Open Source Charms.

Prerequisite

A Canonical OpenStack base setup deployed for a required release. Refer

Steps to install the Trilio charms

1. Export the OpenStack base bundle

2. Create a Trilio overlay bundle as per the OpenStack setup release using the charms given above.

Sample Trilio overlay bundles (T5O release wise) available in triliovault-cfg-scripts repository at path : juju-charms/sample_overlay_bundles

Following table provides the details of various values to be updated in overlay bundle.

3. T4O Deployment

3.1] Do a dry run to check if the Trilio bundle is working

3.2] Trigger deployment

3.3] Wait until all the Trilio units are deployed successfully. Check the status via juju status command.

4. Post Deployment Steps

4.1] Once the deployment is complete, perform the below operations:

a. Create cloud admin trust & add licence

Note: Reach out to the Trilio support team for the license file.

5. Verify the T4O Deployment

5.1] After T4O deployment steps are over, it can take sometime for all units to get deployed successfully. Deployment is considered successful when all the units show Unit is Ready in the message column.

To verify the same, following command (& sample output) can be used to fetch the Trilio units/applications.

6. Troubleshooting T4O Deployment

6.1] To debug any specific unit : juju debug-log --include <UNIT_NAME_IN_ERROR>

Eg. If trilio-wlm/6 unit is in 'error' state, it's logs can be fetched using following command. Specific correct unit number from respective deployment using juju status command.

juju debug log --include trilio-wlm/6

For multipath enabled environments, perform the following actions

1. log into each nova compute node

2. add uxsock_timeout with value as 60000 (i.e. 60 sec) in /etc/multipath.conf

3. restart tvault-contego service

Trilio Branch

Branch of the repository to be used for DevOps scripts.

trilio_branch : 6.1.1

Deployment Scripts

git clone -b 6.1.1 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/rhosp18/

git clone -b 6.1.1 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/redhat-director-scripts/rhosp17/

git clone -b 6.1.1 https://github.com/trilioData/triliovault-cfg-scripts.git
cd triliovault-cfg-scripts/openstack-helm/

Container/Revision Information

Package Repositories & Versions

All packages are Python 3 (py3.6 - py3.10) compatible only.

https://pypi.fury.io/trilio-6-1/

deb [trusted=yes] https://apt.fury.io/trilio-6-1/ /

https://yum.fury.io/trilio-6-1/

[trilio-fury]
name=Trilio Gemfury Private Repo
baseurl=https://yum.fury.io/trilio-6-1/
enabled=1
gpgcheck=0

Start File Search

POST https://$(tvm_address):8780/v1/$(tenant_id)/search

Starts a File Search with the given parameters

Path Parameters

Headers

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Mon, 09 Nov 2020 13:23:25 GMT
Content-Type: application/json
Content-Length: 244
Connection: keep-alive
X-Compute-Request-Id: req-bdfd3fb8-5cbf-4108-885f-63160426b2fa

{
   "file_search":{
      "created_at":"2020-11-09T13:23:25.698534",
      "updated_at":null,
      "id":14,
      "deleted_at":null,
      "status":"executing",
      "error_msg":null,
      "filepath":"/etc/h*",
      "json_resp":null,
      "vm_id":"08dab61c-6efd-44d3-a9ed-8e789d338c1b"
   }
}

Body format

{
   "file_search":{
      "start":<Integer>,
      "end":<Integer>,
      "filepath":"<Reg-Ex String>",
      "date_from":<Date Format: YYYY-MM-DDTHH:MM:SS>,
      "date_to":<Date Format: YYYY-MM-DDTHH:MM:SS>,
      "snapshot_ids":[
         "<Snapshot-ID>"
      ],
      "vm_id":"<VM-ID>"
   }
}

Get File Search Results

POST https://$(tvm_address):8780/v1/$(tenant_id)/search/<search_id>

Starts a filesearch with the given parameters

Path Parameters

Headers

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Mon, 09 Nov 2020 13:24:28 GMT
Content-Type: application/json
Content-Length: 819
Connection: keep-alive
X-Compute-Request-Id: req-d57bea9a-9968-4357-8743-e0b906466063

{
   "file_search":{
      "created_at":"2020-11-09T13:23:25.000000",
      "updated_at":"2020-11-09T13:23:48.000000",
      "id":14,
      "deleted_at":null,
      "status":"completed",
      "error_msg":null,
      "filepath":"/etc/h*",
      "json_resp":"[
                      {
                         "ed4f29e8-7544-4e1c-af8a-a76031211926":[
                            {
                               "/dev/vda1":[
                                  "/etc/hostname",
                                  "/etc/hosts"
                               ],
                               "/etc/hostname":{
                                  "dev":"2049",
                                  "ino":"32",
                                  "mode":"33204",
                                  "nlink":"1",
                                  "uid":"0",
                                  "gid":"0",
                                  "rdev":"0",
                                  "size":"1",
                                  "blksize":"1024",
                                  "blocks":"2",
                                  "atime":"1603455255",
                                  "mtime":"1603455255",
                                  "ctime":"1603455255"
                               },
                               "/etc/hosts":{
                                  "dev":"2049",
                                  "ino":"127",
                                  "mode":"33204",
                                  "nlink":"1",
                                  "uid":"0",
                                  "gid":"0",
                                  "rdev":"0",
                                  "size":"37",
                                  "blksize":"1024",
                                  "blocks":"2",
                                  "atime":"1603455257",
                                  "mtime":"1431011050",
                                  "ctime":"1431017172"
                               }
                            }
                         ]
                      }
                  ]",
      "vm_id":"08dab61c-6efd-44d3-a9ed-8e789d338c1b"
   }
}